Oct 31 12:26:34.978802 ip-10-0-130-190 systemd[1]: Starting Kubernetes Kubelet...
Oct 31 12:26:35.431299 ip-10-0-130-190 kubenswrapper[2576]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 12:26:35.431299 ip-10-0-130-190 kubenswrapper[2576]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 31 12:26:35.431299 ip-10-0-130-190 kubenswrapper[2576]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 12:26:35.431299 ip-10-0-130-190 kubenswrapper[2576]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 12:26:35.437420 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.437332    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 12:26:35.437420 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.437415    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 12:26:35.437420 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.437420    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 12:26:35.437420 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.437424    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 12:26:35.437420 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.437429    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 12:26:35.437737 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.437434    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 12:26:35.437737 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.437438    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 12:26:35.437737 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.437442    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 12:26:35.437737 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.437446    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 12:26:35.437737 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.437450    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 12:26:35.437737 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.437454    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 12:26:35.437737 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.437458    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 12:26:35.437737 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.437461    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 12:26:35.437737 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.437476    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 12:26:35.437737 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.437481    2576 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 12:26:35.437737 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.437503    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 12:26:35.437737 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.437510    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 12:26:35.437737 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.437514    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 12:26:35.437737 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.437520    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 12:26:35.437737 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.437526    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 12:26:35.437737 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.437531    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 12:26:35.437737 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.437535    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 12:26:35.437737 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.437540    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 12:26:35.437737 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.437544    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 12:26:35.438529 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.437548    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 12:26:35.438529 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.437553    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 12:26:35.438529 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.437557    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 12:26:35.438529 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.437562    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 12:26:35.438529 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.437566    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 12:26:35.438529 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.437570    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 12:26:35.438529 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.437575    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 12:26:35.438529 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.437580    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 12:26:35.438529 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.437585    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 12:26:35.438529 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.437590    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 12:26:35.438529 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.437594    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 12:26:35.438529 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.437598    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 12:26:35.438529 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.437602    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 12:26:35.438529 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.437607    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 12:26:35.438529 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.437612    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 12:26:35.438529 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.437616    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 12:26:35.438529 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.437620    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 12:26:35.438529 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.437625    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 12:26:35.438529 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.437630    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 12:26:35.438529 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.437634    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 12:26:35.439346 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.437640    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 12:26:35.439346 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.437644    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 12:26:35.439346 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.437649    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 12:26:35.439346 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.437654    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 12:26:35.439346 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.437658    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 12:26:35.439346 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.437663    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 12:26:35.439346 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.437667    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 12:26:35.439346 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.437671    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 12:26:35.439346 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.437676    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 12:26:35.439346 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.437680    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 12:26:35.439346 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.437684    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 12:26:35.439346 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.437689    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 12:26:35.439346 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.437693    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 12:26:35.439346 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.437697    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 12:26:35.439346 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.437701    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 12:26:35.439346 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.437706    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 12:26:35.439346 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.437710    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 12:26:35.439346 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.437714    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 12:26:35.439346 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.437718    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 12:26:35.439346 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.437722    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 12:26:35.440240 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.437727    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 12:26:35.440240 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.437731    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 12:26:35.440240 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.437735    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 12:26:35.440240 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.437739    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 12:26:35.440240 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.437743    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 12:26:35.440240 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.437748    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 12:26:35.440240 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.437754    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 12:26:35.440240 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.437758    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 12:26:35.440240 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.437762    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 12:26:35.440240 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.437767    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 12:26:35.440240 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.437771    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 12:26:35.440240 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.437776    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 12:26:35.440240 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.437780    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 12:26:35.440240 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.437784    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 12:26:35.440240 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.437789    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 12:26:35.440240 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.437794    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 12:26:35.440240 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.437798    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 12:26:35.440240 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.437802    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 12:26:35.440240 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.437806    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 12:26:35.440240 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.437810    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 12:26:35.441139 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.437815    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 12:26:35.441139 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.437820    2576 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 12:26:35.441139 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.437824    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 12:26:35.441139 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.437828    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 12:26:35.441139 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.437832    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 12:26:35.441139 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.437836    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 12:26:35.441139 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.437840    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 12:26:35.441139 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.437844    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 12:26:35.441139 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.437848    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 12:26:35.441139 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.437855    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 12:26:35.441139 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.437859    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 12:26:35.441139 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.437863    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 12:26:35.441139 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.437868    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 12:26:35.441139 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.437872    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 12:26:35.441139 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.437876    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 12:26:35.441139 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.437880    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 12:26:35.441139 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.437884    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 12:26:35.441139 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.437888    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 12:26:35.441139 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.437893    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 12:26:35.441139 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.437898    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 12:26:35.442094 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.437902    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 12:26:35.442094 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.437907    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 12:26:35.442094 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.439336    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 12:26:35.442094 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.439347    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 12:26:35.442094 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.439351    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 12:26:35.442094 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.439355    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 12:26:35.442094 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.439359    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 12:26:35.442094 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.439363    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 12:26:35.442094 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.439375    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 12:26:35.442094 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.439380    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 12:26:35.442094 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.439385    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 12:26:35.442094 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.439390    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 12:26:35.442094 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.439394    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 12:26:35.442094 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.439398    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 12:26:35.442094 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.439402    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 12:26:35.442094 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.439406    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 12:26:35.442094 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.439411    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 12:26:35.442094 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.439415    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 12:26:35.442094 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.439419    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 12:26:35.442094 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.439424    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 12:26:35.442979 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.439428    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 12:26:35.442979 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.439432    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 12:26:35.442979 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.439436    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 12:26:35.442979 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.439440    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 12:26:35.442979 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.439445    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 12:26:35.442979 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.439449    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 12:26:35.442979 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.439453    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 12:26:35.442979 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.439457    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 12:26:35.442979 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.439462    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 12:26:35.442979 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.439467    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 12:26:35.442979 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.439472    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 12:26:35.442979 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.439476    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 12:26:35.442979 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.439480    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 12:26:35.442979 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.439485    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 12:26:35.442979 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.439506    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 12:26:35.442979 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.439511    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 12:26:35.442979 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.439515    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 12:26:35.442979 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.439519    2576 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 12:26:35.442979 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.439523    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 12:26:35.443442 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.439528    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 12:26:35.443442 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.439531    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 12:26:35.443442 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.439544    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 12:26:35.443442 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.439548    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 12:26:35.443442 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.439552    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 12:26:35.443442 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.439557    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 12:26:35.443442 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.439565    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 12:26:35.443442 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.439571    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 12:26:35.443442 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.439576    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 12:26:35.443442 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.439580    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 12:26:35.443442 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.439584    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 12:26:35.443442 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.439588    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 12:26:35.443442 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.439593    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 12:26:35.443442 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.439597    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 12:26:35.443442 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.439601    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 12:26:35.443442 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.439605    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 12:26:35.443442 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.439609    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 12:26:35.443442 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.439613    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 12:26:35.443442 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.439617    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 12:26:35.443953 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.439622    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 12:26:35.443953 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.439626    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 12:26:35.443953 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.439630    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 12:26:35.443953 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.439635    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 12:26:35.443953 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.439639    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 12:26:35.443953 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.439644    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 12:26:35.443953 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.439648    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 12:26:35.443953 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.439652    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 12:26:35.443953 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.439656    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 12:26:35.443953 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.439660    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 12:26:35.443953 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.439665    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 12:26:35.443953 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.439669    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 12:26:35.443953 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.439673    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 12:26:35.443953 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.439678    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 12:26:35.443953 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.439682    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 12:26:35.443953 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.439686    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 12:26:35.443953 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.439701    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 12:26:35.443953 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.439706    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 12:26:35.443953 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.439710    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 12:26:35.443953 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.439714    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 12:26:35.444433 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.439719    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 12:26:35.444433 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.439723    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 12:26:35.444433 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.439727    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 12:26:35.444433 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.439731    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 12:26:35.444433 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.439735    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 12:26:35.444433 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.439739    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 12:26:35.444433 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.439743    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 12:26:35.444433 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.439748    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 12:26:35.444433 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.439752    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 12:26:35.444433 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.439756    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 12:26:35.444433 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.439760    2576 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 12:26:35.444433 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.439765    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 12:26:35.444433 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.439769    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 12:26:35.444433 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.439773    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 12:26:35.444433 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.439777    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 12:26:35.444433 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.439783    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 12:26:35.444433 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.439788    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 12:26:35.444433 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.439800    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 12:26:35.444433 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.439805    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 12:26:35.444433 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.439809    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 12:26:35.444930 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.439814    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 12:26:35.444930 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.439819    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 12:26:35.444930 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.439823    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 12:26:35.444930 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.439828    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 12:26:35.444930 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.439832    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 12:26:35.444930 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.439836    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 12:26:35.444930 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.439840    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 12:26:35.444930 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.439844    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 12:26:35.444930 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.439852    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 12:26:35.444930 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.439865    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 12:26:35.444930 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.440817    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 12:26:35.444930 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.440826    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 12:26:35.444930 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.440831    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 12:26:35.444930 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.440836    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 12:26:35.444930 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.440841    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 12:26:35.444930 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.440845    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 12:26:35.444930 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.440849    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 12:26:35.444930 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.440853    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 12:26:35.444930 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.440858    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 12:26:35.445384 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.440863    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 12:26:35.445384 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.440867    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 12:26:35.445384 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.440871    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 12:26:35.445384 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.440876    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 12:26:35.445384 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.440880    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 12:26:35.445384 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.440884    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 12:26:35.445384 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.440888    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 12:26:35.445384 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.440892    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 12:26:35.445384 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.440897    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 12:26:35.445384 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.440901    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 12:26:35.445384 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.440905    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 12:26:35.445384 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.440910    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 12:26:35.445384 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.440915    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 12:26:35.445384 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.440919    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 12:26:35.445384 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.440923    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 12:26:35.445384 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.440928    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 12:26:35.445384 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.440932    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 12:26:35.445384 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.440936    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 12:26:35.445384 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.440940    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 12:26:35.445863 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.440944    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 12:26:35.445863 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.440949    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 12:26:35.445863 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.440953    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 12:26:35.445863 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.440965    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 12:26:35.445863 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.440970    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 12:26:35.445863 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.440974    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 12:26:35.445863 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.440978    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 12:26:35.445863 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.440982    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 12:26:35.445863 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.440986    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 12:26:35.445863 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.440990    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 12:26:35.445863 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.440994    2576 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 12:26:35.445863 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.440999    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 12:26:35.445863 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.441003    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 12:26:35.445863 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.441007    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 12:26:35.445863 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.441012    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 12:26:35.445863 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.441016    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 12:26:35.445863 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.441020    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 12:26:35.445863 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.441024    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 12:26:35.445863 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.441028    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 12:26:35.445863 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.441032    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 12:26:35.446345 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.441036    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 12:26:35.446345 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.441041    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 12:26:35.446345 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.441046    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 12:26:35.446345 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.441050    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 12:26:35.446345 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.441054    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 12:26:35.446345 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.441058    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 12:26:35.446345 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.441063    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 12:26:35.446345 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.441067    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 12:26:35.446345 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.441071    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 12:26:35.446345 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.441075    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 12:26:35.446345 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.441080    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 12:26:35.446345 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.441084    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 12:26:35.446345 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.441088    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 12:26:35.446345 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.441093    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 12:26:35.446345 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.441097    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 12:26:35.446345 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.441101    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 12:26:35.446345 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.441115    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 12:26:35.446345 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.441120    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 12:26:35.446345 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.441124    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 12:26:35.446345 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.441128    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 12:26:35.446838 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.441132    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 12:26:35.446838 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.441136    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 12:26:35.446838 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.441140    2576 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 12:26:35.446838 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.441144    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 12:26:35.446838 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.441148    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 12:26:35.446838 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.441152    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 12:26:35.446838 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.441159    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 12:26:35.446838 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.441165    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 12:26:35.446838 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.441170    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 12:26:35.446838 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.441175    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 12:26:35.446838 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.441179    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 12:26:35.446838 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.441184    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 12:26:35.446838 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.441188    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 12:26:35.446838 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.441195    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 12:26:35.446838 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.441199    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 12:26:35.446838 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.441203    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 12:26:35.446838 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.441207    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 12:26:35.446838 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.441212    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 12:26:35.446838 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.441220    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 12:26:35.447278 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.441224    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 12:26:35.447278 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.441229    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 12:26:35.447278 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.441233    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 12:26:35.447278 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.441237    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 12:26:35.447278 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.441242    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 12:26:35.447278 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.441246    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 12:26:35.447278 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.441250    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 12:26:35.447278 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.441254    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 12:26:35.447278 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.441259    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 12:26:35.447278 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.441263    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 12:26:35.447278 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.441275    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 12:26:35.447278 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.441279    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 12:26:35.447278 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.441283    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 12:26:35.447278 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.441287    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 12:26:35.447278 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.441294    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 12:26:35.447278 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.441299    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 12:26:35.447278 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.441304    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 12:26:35.447278 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.441309    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 12:26:35.447278 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.441313    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 12:26:35.447278 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.441451    2576 flags.go:64] FLAG: --address="0.0.0.0"
Oct 31 12:26:35.447759 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.441462    2576 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 31 12:26:35.447759 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.441476    2576 flags.go:64] FLAG: --anonymous-auth="true"
Oct 31 12:26:35.447759 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.441483    2576 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 31 12:26:35.447759 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.441506    2576 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 31 12:26:35.447759 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.441512    2576 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 31 12:26:35.447759 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.441519    2576 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 31 12:26:35.447759 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.441531    2576 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 31 12:26:35.447759 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.441536    2576 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 31 12:26:35.447759 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.441541    2576 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 31 12:26:35.447759 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.441547    2576 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 31 12:26:35.447759 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.441553    2576 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 31 12:26:35.447759 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.441558    2576 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 31 12:26:35.447759 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.441562    2576 flags.go:64] FLAG: --cgroup-root=""
Oct 31 12:26:35.447759 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.441567    2576 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 31 12:26:35.447759 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.441572    2576 flags.go:64] FLAG: --client-ca-file=""
Oct 31 12:26:35.447759 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.441576    2576 flags.go:64] FLAG: --cloud-provider="external"
Oct 31 12:26:35.447759 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.441583    2576 flags.go:64] FLAG: --cluster-dns="[]"
Oct 31 12:26:35.447759 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.441593    2576 flags.go:64] FLAG: --cluster-domain=""
Oct 31 12:26:35.447759 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.441598    2576 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 31 12:26:35.447759 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.441603    2576 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 31 12:26:35.447759 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.441608    2576 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 31 12:26:35.447759 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.441614    2576 flags.go:64] FLAG: --container-log-max-files="5"
Oct 31 12:26:35.447759 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.441620    2576 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 31 12:26:35.447759 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.441633    2576 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 31 12:26:35.448322 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.441639    2576 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 31 12:26:35.448322 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.441652    2576 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 31 12:26:35.448322 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.441656    2576 flags.go:64] FLAG: --contention-profiling="false"
Oct 31 12:26:35.448322 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.441661    2576 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 31 12:26:35.448322 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.441666    2576 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 31 12:26:35.448322 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.441670    2576 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 31 12:26:35.448322 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.441675    2576 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 31 12:26:35.448322 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.441682    2576 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 31 12:26:35.448322 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.441687    2576 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 31 12:26:35.448322 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.441692    2576 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 31 12:26:35.448322 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.441697    2576 flags.go:64] FLAG: --enable-load-reader="false"
Oct 31 12:26:35.448322 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.441702    2576 flags.go:64] FLAG: --enable-server="true"
Oct 31 12:26:35.448322 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.441706    2576 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 31 12:26:35.448322 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.441717    2576 flags.go:64] FLAG: --event-burst="100"
Oct 31 12:26:35.448322 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.441725    2576 flags.go:64] FLAG: --event-qps="50"
Oct 31 12:26:35.448322 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.441729    2576 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 31 12:26:35.448322 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.441733    2576 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 31 12:26:35.448322 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.441738    2576 flags.go:64] FLAG: --eviction-hard=""
Oct 31 12:26:35.448322 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.441744    2576 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 31 12:26:35.448322 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.441748    2576 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 31 12:26:35.448322 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.441752    2576 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 31 12:26:35.448322 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.441757    2576 flags.go:64] FLAG: --eviction-soft=""
Oct 31 12:26:35.448322 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.441761    2576 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 31 12:26:35.448322 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.441766    2576 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 31 12:26:35.448322 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.441772    2576 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 31 12:26:35.448977 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.441777    2576 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 31 12:26:35.448977 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.441782    2576 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 31 12:26:35.448977 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.441786    2576 flags.go:64] FLAG: --fail-swap-on="true"
Oct 31 12:26:35.448977 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.441790    2576 flags.go:64] FLAG: --feature-gates=""
Oct 31 12:26:35.448977 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.441797    2576 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 31 12:26:35.448977 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.441802    2576 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 31 12:26:35.448977 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.441806    2576 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 31 12:26:35.448977 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.441819    2576 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 31 12:26:35.448977 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.441824    2576 flags.go:64] FLAG: --healthz-port="10248"
Oct 31 12:26:35.448977 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.441829    2576 flags.go:64] FLAG: --help="false"
Oct 31 12:26:35.448977 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.441834    2576 flags.go:64] FLAG: --hostname-override="ip-10-0-130-190.us-east-2.compute.internal"
Oct 31 12:26:35.448977 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.441839    2576 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 31 12:26:35.448977 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.441844    2576 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 31 12:26:35.448977 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.441849    2576 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 31 12:26:35.448977 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.441854    2576 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 31 12:26:35.448977 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.441860    2576 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 31 12:26:35.448977 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.441865    2576 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 31 12:26:35.448977 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.441869    2576 flags.go:64] FLAG: --image-service-endpoint=""
Oct 31 12:26:35.448977 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.441874    2576 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 31 12:26:35.448977 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.441879    2576 flags.go:64] FLAG: --kube-api-burst="100"
Oct 31 12:26:35.448977 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.441884    2576 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 31 12:26:35.448977 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.441891    2576 flags.go:64] FLAG: --kube-api-qps="50"
Oct 31 12:26:35.448977 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.441896    2576 flags.go:64] FLAG: --kube-reserved=""
Oct 31 12:26:35.448977 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.441901    2576 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 31 12:26:35.449814 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.441905    2576 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 31 12:26:35.449814 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.441911    2576 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 31 12:26:35.449814 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.441915    2576 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 31 12:26:35.449814 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.441920    2576 flags.go:64] FLAG: --lock-file=""
Oct 31 12:26:35.449814 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.441924    2576 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 31 12:26:35.449814 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.441929    2576 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 31 12:26:35.449814 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.441934    2576 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 31 12:26:35.449814 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.441945    2576 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 31 12:26:35.449814 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.441950    2576 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 31 12:26:35.449814 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.441955    2576 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 31 12:26:35.449814 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.441960    2576 flags.go:64] FLAG: --logging-format="text"
Oct 31 12:26:35.449814 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.441965    2576 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 31 12:26:35.449814 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.441970    2576 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 31 12:26:35.449814 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.441975    2576 flags.go:64] FLAG: --manifest-url=""
Oct 31 12:26:35.449814 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.441979    2576 flags.go:64] FLAG: --manifest-url-header=""
Oct 31 12:26:35.449814 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.441987    2576 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 31 12:26:35.449814 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.442002    2576 flags.go:64] FLAG: --max-open-files="1000000"
Oct 31 12:26:35.449814 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.442008    2576 flags.go:64] FLAG: --max-pods="110"
Oct 31 12:26:35.449814 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.442013    2576 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 31 12:26:35.449814 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.442018    2576 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 31 12:26:35.449814 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.442023    2576 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 31 12:26:35.449814 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.442028    2576 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 31 12:26:35.449814 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.442033    2576 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 31 12:26:35.449814 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.442037    2576 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 31 12:26:35.449814 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.442042    2576 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 31 12:26:35.450828 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.442053    2576 flags.go:64] FLAG: --node-status-max-images="50"
Oct 31 12:26:35.450828 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.442059    2576 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 31 12:26:35.450828 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.442064    2576 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 31 12:26:35.450828 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.442069    2576 flags.go:64] FLAG: --pod-cidr=""
Oct 31 12:26:35.450828 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.442075    2576 flags.go:64] FLAG: --pod-manifest-path=""
Oct 31 12:26:35.450828 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.442079    2576 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 31 12:26:35.450828 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.442084    2576 flags.go:64] FLAG: --pods-per-core="0"
Oct 31 12:26:35.450828 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.442089    2576 flags.go:64] FLAG: --port="10250"
Oct 31 12:26:35.450828 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.442093    2576 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 31 12:26:35.450828 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.442097    2576 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-0c4d4bff4324dd77c"
Oct 31 12:26:35.450828 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.442102    2576 flags.go:64] FLAG: --qos-reserved=""
Oct 31 12:26:35.450828 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.442107    2576 flags.go:64] FLAG: --read-only-port="10255"
Oct 31 12:26:35.450828 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.442111    2576 flags.go:64] FLAG: --register-node="true"
Oct 31 12:26:35.450828 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.442116    2576 flags.go:64] FLAG: --register-with-taints=""
Oct 31 12:26:35.450828 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.442124    2576 flags.go:64] FLAG: --registry-burst="10"
Oct 31 12:26:35.450828 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.442129    2576 flags.go:64] FLAG: --registry-qps="5"
Oct 31 12:26:35.450828 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.442134    2576 flags.go:64] FLAG: --reserved-cpus=""
Oct 31 12:26:35.450828 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.442139    2576 flags.go:64] FLAG: --reserved-memory=""
Oct 31 12:26:35.450828 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.442145    2576 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 31 12:26:35.450828 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.442150    2576 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 31 12:26:35.450828 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.442154    2576 flags.go:64] FLAG: --rotate-certificates="false"
Oct 31 12:26:35.450828 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.442159    2576 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 31 12:26:35.450828 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.442163    2576 flags.go:64] FLAG: --runonce="false"
Oct 31 12:26:35.450828 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.442168    2576 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 31 12:26:35.450828 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.442175    2576 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 31 12:26:35.450828 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.442180    2576 flags.go:64] FLAG: --seccomp-default="false"
Oct 31 12:26:35.450828 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.442185    2576 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 31 12:26:35.451958 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.442190    2576 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 31 12:26:35.451958 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.442202    2576 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 31 12:26:35.451958 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.442207    2576 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 31 12:26:35.451958 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.442212    2576 flags.go:64] FLAG: --storage-driver-password="root"
Oct 31 12:26:35.451958 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.442217    2576 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 31 12:26:35.451958 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.442221    2576 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 31 12:26:35.451958 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.442226    2576 flags.go:64] FLAG: --storage-driver-user="root"
Oct 31 12:26:35.451958 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.442231    2576 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 31 12:26:35.451958 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.442236    2576 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 31 12:26:35.451958 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.442242    2576 flags.go:64] FLAG: --system-cgroups=""
Oct 31 12:26:35.451958 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.442247    2576 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 31 12:26:35.451958 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.442255    2576 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 31 12:26:35.451958 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.442260    2576 flags.go:64] FLAG: --tls-cert-file=""
Oct 31 12:26:35.451958 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.442265    2576 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 31 12:26:35.451958 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.442275    2576 flags.go:64] FLAG: --tls-min-version=""
Oct 31 12:26:35.451958 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.442280    2576 flags.go:64] FLAG: --tls-private-key-file=""
Oct 31 12:26:35.451958 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.442285    2576 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 31 12:26:35.451958 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.442289    2576 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 31 12:26:35.451958 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.442294    2576 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 31 12:26:35.451958 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.442300    2576 flags.go:64] FLAG: --v="2"
Oct 31 12:26:35.451958 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.442307    2576 flags.go:64] FLAG: --version="false"
Oct 31 12:26:35.451958 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.442314    2576 flags.go:64] FLAG: --vmodule=""
Oct 31 12:26:35.451958 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.442320    2576 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 31 12:26:35.451958 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.442325    2576 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 31 12:26:35.451958 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.442596    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 12:26:35.453045 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.442605    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 12:26:35.453045 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.442611    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 12:26:35.453045 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.442616    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 12:26:35.453045 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.442620    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 12:26:35.453045 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.442624    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 12:26:35.453045 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.442630    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 12:26:35.453045 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.442635    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 12:26:35.453045 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.442640    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 12:26:35.453045 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.442645    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 12:26:35.453045 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.442649    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 12:26:35.453045 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.442653    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 12:26:35.453045 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.442658    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 12:26:35.453045 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.442662    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 12:26:35.453045 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.442666    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 12:26:35.453045 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.442670    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 12:26:35.453045 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.442674    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 12:26:35.453045 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.442681    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 12:26:35.453045 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.442685    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 12:26:35.453045 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.442690    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 12:26:35.453882 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.442696    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 12:26:35.453882 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.442703    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 12:26:35.453882 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.442708    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 12:26:35.453882 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.442712    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 12:26:35.453882 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.442716    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 12:26:35.453882 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.442721    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 12:26:35.453882 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.442725    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 12:26:35.453882 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.442731    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 12:26:35.453882 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.442736    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 12:26:35.453882 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.442740    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 12:26:35.453882 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.442744    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 12:26:35.453882 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.442748    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 12:26:35.453882 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.442753    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 12:26:35.453882 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.442756    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 12:26:35.453882 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.442761    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 12:26:35.453882 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.442765    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 12:26:35.453882 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.442769    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 12:26:35.453882 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.442774    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 12:26:35.453882 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.442778    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 12:26:35.453882 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.442783    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 12:26:35.454833 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.442787    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 12:26:35.454833 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.442791    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 12:26:35.454833 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.442795    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 12:26:35.454833 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.442800    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 12:26:35.454833 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.442805    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 12:26:35.454833 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.442809    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 12:26:35.454833 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.442813    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 12:26:35.454833 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.442818    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 12:26:35.454833 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.442822    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 12:26:35.454833 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.442828    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 12:26:35.454833 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.442833    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 12:26:35.454833 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.442837    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 12:26:35.454833 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.442841    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 12:26:35.454833 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.442845    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 12:26:35.454833 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.442850    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 12:26:35.454833 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.442854    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 12:26:35.454833 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.442858    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 12:26:35.454833 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.442863    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 12:26:35.454833 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.442867    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 12:26:35.455359 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.442872    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 12:26:35.455359 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.442876    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 12:26:35.455359 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.442915    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 12:26:35.455359 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.442920    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 12:26:35.455359 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.442925    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 12:26:35.455359 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.442929    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 12:26:35.455359 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.442932    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 12:26:35.455359 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.442936    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 12:26:35.455359 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.442939    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 12:26:35.455359 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.442942    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 12:26:35.455359 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.442945    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 12:26:35.455359 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.442948    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 12:26:35.455359 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.442951    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 12:26:35.455359 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.442954    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 12:26:35.455359 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.442956    2576 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 12:26:35.455359 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.442959    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 12:26:35.455359 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.442962    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 12:26:35.455359 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.442966    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 12:26:35.455359 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.442970    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 12:26:35.455359 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.442974    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 12:26:35.456198 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.442977    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 12:26:35.456198 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.442980    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 12:26:35.456198 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.442983    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 12:26:35.456198 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.442986    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 12:26:35.456198 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.442989    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 12:26:35.456198 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.442993    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 12:26:35.456198 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.442996    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 12:26:35.456198 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.442999    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 12:26:35.456198 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.443001    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 12:26:35.456198 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.443006    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 12:26:35.456198 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.443009    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 12:26:35.456198 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.443011    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 12:26:35.456198 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.443014    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 12:26:35.456198 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.443017    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 12:26:35.456198 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.443020    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 12:26:35.456198 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.443023    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 12:26:35.456198 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.443025    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 12:26:35.456198 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.443028    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 12:26:35.456198 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.443031    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 12:26:35.456922 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.443034    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 12:26:35.456922 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.443036    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 12:26:35.456922 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.443039    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 12:26:35.456922 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.443042    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 12:26:35.456922 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.443045    2576 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 12:26:35.456922 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.443047    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 12:26:35.456922 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.443050    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 12:26:35.456922 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.443053    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 12:26:35.456922 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.443091    2576 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 12:26:35.456922 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.443149    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 12:26:35.456922 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.443153    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 12:26:35.456922 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.443156    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 12:26:35.456922 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.443159    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 12:26:35.456922 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.443162    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 12:26:35.456922 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.443165    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 12:26:35.456922 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.443168    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 12:26:35.457302 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.443170    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 12:26:35.457302 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.443173    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 12:26:35.457302 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.443176    2576 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 12:26:35.457302 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.443179    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 12:26:35.457302 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.443182    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 12:26:35.457302 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.443185    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 12:26:35.457302 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.443188    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 12:26:35.457302 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.443191    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 12:26:35.457302 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.443194    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 12:26:35.457302 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.443197    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 12:26:35.457302 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.443200    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 12:26:35.457302 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.443203    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 12:26:35.457302 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.443206    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 12:26:35.457302 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.443209    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 12:26:35.457302 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.443211    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 12:26:35.457302 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.443214    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 12:26:35.457302 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.443217    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 12:26:35.457302 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.443219    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 12:26:35.457302 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.443222    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 12:26:35.457960 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.443224    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 12:26:35.457960 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.443227    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 12:26:35.457960 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.443229    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 12:26:35.457960 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.443232    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 12:26:35.457960 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.443234    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 12:26:35.457960 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.443237    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 12:26:35.457960 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.443239    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 12:26:35.457960 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.443243    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 12:26:35.457960 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.443247    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 12:26:35.457960 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.443249    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 12:26:35.457960 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.443252    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 12:26:35.457960 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.443254    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 12:26:35.457960 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.443257    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 12:26:35.457960 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.443259    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 12:26:35.457960 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.443262    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 12:26:35.457960 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.443264    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 12:26:35.457960 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.443270    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 12:26:35.457960 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.443272    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 12:26:35.457960 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.443274    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 12:26:35.458413 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.443277    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 12:26:35.458413 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.443280    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 12:26:35.458413 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.443283    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 12:26:35.458413 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.443286    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 12:26:35.458413 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.443288    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 12:26:35.458413 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.443293    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 12:26:35.458413 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.443296    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 12:26:35.458413 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.443298    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 12:26:35.458413 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.443301    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 12:26:35.458413 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.443303    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 12:26:35.458413 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.443306    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 12:26:35.458413 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.443309    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 12:26:35.458413 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.443311    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 12:26:35.458413 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.443314    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 12:26:35.458413 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.443317    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 12:26:35.458413 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.443320    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 12:26:35.458413 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.443322    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 12:26:35.458413 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.443325    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 12:26:35.458413 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.443327    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 12:26:35.458933 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.443330    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 12:26:35.458933 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.443333    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 12:26:35.458933 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.443335    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 12:26:35.458933 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.443338    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 12:26:35.458933 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.443340    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 12:26:35.458933 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.443342    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 12:26:35.458933 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.443345    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 12:26:35.458933 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.443347    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 12:26:35.458933 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.443350    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 12:26:35.458933 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.443352    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 12:26:35.458933 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.443359    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 12:26:35.458933 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.443362    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 12:26:35.458933 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.443365    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 12:26:35.458933 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.443368    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 12:26:35.458933 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.443370    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 12:26:35.458933 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.443374    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 12:26:35.458933 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.443378    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 12:26:35.458933 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.443381    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 12:26:35.458933 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.443383    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 12:26:35.458933 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.443386    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 12:26:35.459404 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.443389    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 12:26:35.459404 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.443391    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 12:26:35.459404 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.443394    2576 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 12:26:35.459404 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.443396    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 12:26:35.459404 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.443399    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 12:26:35.459404 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.443403    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 12:26:35.459404 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.443408    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 12:26:35.459404 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.443411    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 12:26:35.459404 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.443413    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 12:26:35.459404 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.443416    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 12:26:35.459404 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.443419    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 12:26:35.459404 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.443421    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 12:26:35.459404 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.443424    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 12:26:35.459404 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.443426    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 12:26:35.459404 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.443429    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 12:26:35.459404 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.443432    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 12:26:35.459404 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.443434    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 12:26:35.459404 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.443436    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 12:26:35.459404 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.443439    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 12:26:35.459932 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.443441    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 12:26:35.459932 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.443444    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 12:26:35.459932 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.443447    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 12:26:35.459932 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.452460    2576 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 31 12:26:35.459932 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.452478    2576 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 31 12:26:35.459932 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.452520    2576 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 31 12:26:35.459932 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.452529    2576 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 31 12:26:35.459932 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.452650    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 12:26:35.459932 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.452658    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 12:26:35.459932 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.452664    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 12:26:35.459932 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.452669    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 12:26:35.459932 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.452673    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 12:26:35.459932 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.452677    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 12:26:35.459932 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.452682    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 12:26:35.459932 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.452687    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 12:26:35.459932 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.452691    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 12:26:35.459932 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.452695    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 12:26:35.459932 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.452699    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 12:26:35.459932 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.452704    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 12:26:35.460383 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.452708    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 12:26:35.460383 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.452712    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 12:26:35.460383 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.452717    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 12:26:35.460383 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.452721    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 12:26:35.460383 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.452725    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 12:26:35.460383 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.452730    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 12:26:35.460383 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.452734    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 12:26:35.460383 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.452739    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 12:26:35.460383 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.452743    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 12:26:35.460383 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.452753    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 12:26:35.460383 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.452758    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 12:26:35.460383 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.452763    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 12:26:35.460383 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.452767    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 12:26:35.460383 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.452771    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 12:26:35.460383 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.452776    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 12:26:35.460383 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.452781    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 12:26:35.460383 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.452785    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 12:26:35.460383 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.452789    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 12:26:35.460383 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.452794    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 12:26:35.460383 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.452799    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 12:26:35.461051 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.452803    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 12:26:35.461051 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.452808    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 12:26:35.461051 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.452812    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 12:26:35.461051 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.452816    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 12:26:35.461051 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.452820    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 12:26:35.461051 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.452825    2576 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 12:26:35.461051 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.452829    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 12:26:35.461051 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.452833    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 12:26:35.461051 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.452837    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 12:26:35.461051 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.452841    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 12:26:35.461051 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.452846    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 12:26:35.461051 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.452850    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 12:26:35.461051 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.452854    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 12:26:35.461051 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.452859    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 12:26:35.461051 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.452864    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 12:26:35.461051 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.452868    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 12:26:35.461051 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.452872    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 12:26:35.461051 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.452877    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 12:26:35.461051 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.452881    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 12:26:35.461051 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.452885    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 12:26:35.461905 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.452890    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 12:26:35.461905 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.452894    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 12:26:35.461905 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.452898    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 12:26:35.461905 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.452902    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 12:26:35.461905 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.452906    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 12:26:35.461905 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.452911    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 12:26:35.461905 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.452915    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 12:26:35.461905 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.452919    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 12:26:35.461905 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.452923    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 12:26:35.461905 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.452928    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 12:26:35.461905 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.452933    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 12:26:35.461905 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.452937    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 12:26:35.461905 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.452941    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 12:26:35.461905 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.452946    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 12:26:35.461905 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.452950    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 12:26:35.461905 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.452957    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 12:26:35.461905 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.452964    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 12:26:35.461905 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.452968    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 12:26:35.461905 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.452973    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 12:26:35.462744 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.452977    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 12:26:35.462744 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.452982    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 12:26:35.462744 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.452986    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 12:26:35.462744 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.452990    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 12:26:35.462744 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.452995    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 12:26:35.462744 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.452999    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 12:26:35.462744 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.453003    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 12:26:35.462744 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.453007    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 12:26:35.462744 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.453013    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 12:26:35.462744 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.453018    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 12:26:35.462744 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.453022    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 12:26:35.462744 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.453027    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 12:26:35.462744 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.453031    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 12:26:35.462744 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.453035    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 12:26:35.462744 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.453040    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 12:26:35.462744 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.453044    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 12:26:35.462744 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.453048    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 12:26:35.462744 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.453051    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 12:26:35.462744 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.453055    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 12:26:35.463365 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.453059    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 12:26:35.463365 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.453063    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 12:26:35.463365 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.453067    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 12:26:35.463365 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.453071    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 12:26:35.463365 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.453075    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 12:26:35.463365 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.453079    2576 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 12:26:35.463365 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.453084    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 12:26:35.463365 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.453088    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 12:26:35.463365 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.453092    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 12:26:35.463365 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.453096    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 12:26:35.463365 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.453101    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 12:26:35.463365 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.453105    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 12:26:35.463365 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.453110    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 12:26:35.463365 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.453116    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 12:26:35.463365 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.453122    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 12:26:35.463365 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.453127    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 12:26:35.463773 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.453167    2576 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 12:26:35.463773 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.453404    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 12:26:35.463773 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.453413    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 12:26:35.463773 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.453417    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 12:26:35.463773 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.453421    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 12:26:35.463773 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.453426    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 12:26:35.463773 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.453431    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 12:26:35.463773 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.453436    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 12:26:35.463773 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.453440    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 12:26:35.463773 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.453444    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 12:26:35.463773 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.453451    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 12:26:35.463773 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.453458    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 12:26:35.463773 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.453463    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 12:26:35.463773 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.453468    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 12:26:35.463773 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.453473    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 12:26:35.463773 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.453478    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 12:26:35.464168 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.453483    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 12:26:35.464168 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.453505    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 12:26:35.464168 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.453510    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 12:26:35.464168 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.453514    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 12:26:35.464168 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.453518    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 12:26:35.464168 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.453522    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 12:26:35.464168 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.453526    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 12:26:35.464168 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.453530    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 12:26:35.464168 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.453534    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 12:26:35.464168 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.453539    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 12:26:35.464168 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.453543    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 12:26:35.464168 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.453547    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 12:26:35.464168 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.453551    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 12:26:35.464168 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.453555    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 12:26:35.464168 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.453559    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 12:26:35.464168 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.453563    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 12:26:35.464168 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.453567    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 12:26:35.464168 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.453571    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 12:26:35.464168 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.453576    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 12:26:35.464168 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.453580    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 12:26:35.464648 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.453584    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 12:26:35.464648 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.453588    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 12:26:35.464648 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.453592    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 12:26:35.464648 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.453598    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 12:26:35.464648 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.453602    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 12:26:35.464648 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.453606    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 12:26:35.464648 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.453611    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 12:26:35.464648 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.453615    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 12:26:35.464648 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.453619    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 12:26:35.464648 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.453623    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 12:26:35.464648 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.453627    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 12:26:35.464648 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.453632    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 12:26:35.464648 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.453636    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 12:26:35.464648 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.453641    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 12:26:35.464648 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.453645    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 12:26:35.464648 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.453649    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 12:26:35.464648 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.453653    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 12:26:35.464648 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.453657    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 12:26:35.464648 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.453661    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 12:26:35.465122 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.453665    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 12:26:35.465122 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.453669    2576 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 12:26:35.465122 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.453674    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 12:26:35.465122 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.453678    2576 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 12:26:35.465122 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.453682    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 12:26:35.465122 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.453686    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 12:26:35.465122 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.453691    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 12:26:35.465122 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.453696    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 12:26:35.465122 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.453700    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 12:26:35.465122 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.453704    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 12:26:35.465122 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.453708    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 12:26:35.465122 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.453713    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 12:26:35.465122 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.453717    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 12:26:35.465122 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.453721    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 12:26:35.465122 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.453725    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 12:26:35.465122 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.453730    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 12:26:35.465122 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.453734    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 12:26:35.465122 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.453741    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 12:26:35.465122 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.453745    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 12:26:35.465122 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.453750    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 12:26:35.465675 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.453754    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 12:26:35.465675 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.453758    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 12:26:35.465675 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.453762    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 12:26:35.465675 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.453767    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 12:26:35.465675 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.453773    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 12:26:35.465675 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.453780    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 12:26:35.465675 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.453785    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 12:26:35.465675 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.453790    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 12:26:35.465675 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.453794    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 12:26:35.465675 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.453799    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 12:26:35.465675 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.453803    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 12:26:35.465675 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.453807    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 12:26:35.465675 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.453811    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 12:26:35.465675 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.453816    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 12:26:35.465675 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.453820    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 12:26:35.465675 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.453824    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 12:26:35.465675 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.453828    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 12:26:35.465675 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.453832    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 12:26:35.465675 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.453837    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 12:26:35.466233 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.453841    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 12:26:35.466233 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.453845    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 12:26:35.466233 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.453850    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 12:26:35.466233 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.453854    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 12:26:35.466233 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.453858    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 12:26:35.466233 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.453863    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 12:26:35.466233 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.453867    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 12:26:35.466233 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.453871    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 12:26:35.466233 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.453876    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 12:26:35.466233 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.453879    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 12:26:35.466233 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.453886    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 12:26:35.466233 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.453891    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 12:26:35.466233 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.453895    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 12:26:35.466233 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.453936    2576 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 12:26:35.466233 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.453991    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 12:26:35.466233 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.453999    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 12:26:35.466628 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.454004    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 12:26:35.466628 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.454008    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 12:26:35.466628 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.454012    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 12:26:35.466628 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.454017    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 12:26:35.466628 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.454021    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 12:26:35.466628 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.454025    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 12:26:35.466628 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.454029    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 12:26:35.466628 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.454034    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 12:26:35.466628 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.454038    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 12:26:35.466628 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.454042    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 12:26:35.466628 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.454046    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 12:26:35.466628 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.454051    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 12:26:35.466628 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.454057    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 12:26:35.466628 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.454061    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 12:26:35.466628 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.454065    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 12:26:35.466628 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.454070    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 12:26:35.466628 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.454074    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 12:26:35.466628 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.454079    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 12:26:35.466628 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.454083    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 12:26:35.466628 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.454088    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 12:26:35.467140 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.454092    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 12:26:35.467140 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.454096    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 12:26:35.467140 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.454100    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 12:26:35.467140 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.454104    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 12:26:35.467140 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.454109    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 12:26:35.467140 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.454113    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 12:26:35.467140 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.454117    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 12:26:35.467140 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.454123    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 12:26:35.467140 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.454129    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 12:26:35.467140 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.454133    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 12:26:35.467140 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.454138    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 12:26:35.467140 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.454142    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 12:26:35.467140 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.454146    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 12:26:35.467140 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.454150    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 12:26:35.467140 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.454154    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 12:26:35.467140 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.454159    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 12:26:35.467140 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.454163    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 12:26:35.467140 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.454168    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 12:26:35.467140 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.454172    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 12:26:35.467140 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.454176    2576 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 12:26:35.467676 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.454181    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 12:26:35.467676 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.454186    2576 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 12:26:35.467676 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.454190    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 12:26:35.467676 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.454201    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 12:26:35.467676 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.454207    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 12:26:35.467676 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.454211    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 12:26:35.467676 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.454216    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 12:26:35.467676 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.454220    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 12:26:35.467676 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.454224    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 12:26:35.467676 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.454228    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 12:26:35.467676 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.454232    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 12:26:35.467676 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.454237    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 12:26:35.467676 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.454241    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 12:26:35.467676 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.454245    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 12:26:35.467676 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.454249    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 12:26:35.467676 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.454254    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 12:26:35.467676 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.454258    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 12:26:35.467676 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.454262    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 12:26:35.467676 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.454267    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 12:26:35.467676 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.454272    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 12:26:35.468299 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.454277    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 12:26:35.468299 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.454282    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 12:26:35.468299 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.454288    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 12:26:35.468299 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.454294    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 12:26:35.468299 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.454299    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 12:26:35.468299 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.454304    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 12:26:35.468299 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.454309    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 12:26:35.468299 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.454314    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 12:26:35.468299 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.454318    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 12:26:35.468299 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.454322    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 12:26:35.468299 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.454326    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 12:26:35.468299 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.454331    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 12:26:35.468299 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.454335    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 12:26:35.468299 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.454340    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 12:26:35.468299 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.454344    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 12:26:35.468299 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.454348    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 12:26:35.468299 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.454354    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 12:26:35.468299 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.454359    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 12:26:35.468299 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.454363    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 12:26:35.468815 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.454367    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 12:26:35.468815 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.454372    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 12:26:35.468815 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.454376    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 12:26:35.468815 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.454380    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 12:26:35.468815 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.454384    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 12:26:35.468815 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.454389    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 12:26:35.468815 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.454393    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 12:26:35.468815 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.454397    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 12:26:35.468815 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.454401    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 12:26:35.468815 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.454406    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 12:26:35.468815 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.454410    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 12:26:35.468815 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.454415    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 12:26:35.468815 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.454420    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 12:26:35.468815 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.454425    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 12:26:35.468815 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.454429    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 12:26:35.468815 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.454435    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 12:26:35.468815 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.454439    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 12:26:35.468815 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.454443    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 12:26:35.468815 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.454448    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 12:26:35.468815 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.454452    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 12:26:35.469292 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.454460    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 12:26:35.469292 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.454466    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 12:26:35.469292 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.454472    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 12:26:35.469292 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.454476    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 12:26:35.469292 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:35.454481    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 12:26:35.469292 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.454571    2576 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 31 12:26:35.469292 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.457581    2576 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Oct 31 12:26:35.469292 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.458769    2576 server.go:1014] "Starting client certificate rotation"
Oct 31 12:26:35.469292 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.458876    2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 31 12:26:35.469292 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.459732    2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 31 12:26:35.469292 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.461288    2576 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 31 12:26:35.469292 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.463082    2576 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 31 12:26:35.474476 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.474460    2576 log.go:25] "Validated CRI v1 runtime API"
Oct 31 12:26:35.478746 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.478731    2576 log.go:25] "Validated CRI v1 image API"
Oct 31 12:26:35.479583 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.479560    2576 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 31 12:26:35.482423 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.482398    2576 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 ea297aca-9120-4875-9123-81f10ff3a7a9:/dev/nvme0n1p3 eaa7c2b6-cdc6-44ca-9bdd-fdaa17fd7979:/dev/nvme0n1p4]
Oct 31 12:26:35.482469 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.482424    2576 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Oct 31 12:26:35.487982 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.487874    2576 manager.go:217] Machine: {Timestamp:2026-10-31 12:26:35.487367979 +0000 UTC m=+0.380015121 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100022 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec21afcc06851de4c41dc10297111780 SystemUUID:ec21afcc-0685-1de4-c41d-c10297111780 BootID:2f054b22-1f8d-49a7-a430-cfe17c0f04f1 Filesystems:[{Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:f2:f3:ef:db Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:f2:f3:ef:db Speed:0 Mtu:9001} {Name:ovs-system MacAddress:7a:5e:81:5a:f1:15 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 31 12:26:35.488907 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.488895    2576 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 31 12:26:35.489039 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.488978    2576 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 31 12:26:35.493486 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.493460    2576 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 31 12:26:35.493675 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.493500    2576 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-190.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 31 12:26:35.493724 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.493681    2576 topology_manager.go:143] "Creating topology manager with none policy"
Oct 31 12:26:35.493724 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.493694    2576 container_manager_linux.go:311] "Creating device plugin manager"
Oct 31 12:26:35.493724 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.493708    2576 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 31 12:26:35.493799 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.493724    2576 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 31 12:26:35.495057 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.495045    2576 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 31 12:26:35.499626 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.499614    2576 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 31 12:26:35.499759 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.499749    2576 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 31 12:26:35.499793 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.499784    2576 kubelet.go:493] "Attempting to sync node with API server"
Oct 31 12:26:35.499820 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.499802    2576 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 31 12:26:35.499820 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.499819    2576 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 31 12:26:35.499874 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.499832    2576 kubelet.go:405] "Adding apiserver pod source"
Oct 31 12:26:35.499874 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.499843    2576 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 31 12:26:35.500276 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.500230    2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 31 12:26:35.500911 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.500895    2576 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 31 12:26:35.502822 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.502804    2576 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 31 12:26:35.509499 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.509480    2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 31 12:26:35.510513 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.510500    2576 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 31 12:26:35.510548 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.510531    2576 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 31 12:26:35.510679 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.510669    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 31 12:26:35.510711 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.510685    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 31 12:26:35.510711 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.510692    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 31 12:26:35.510711 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.510698    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 31 12:26:35.510711 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.510703    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 31 12:26:35.510711 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.510710    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 31 12:26:35.511509 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.511500    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 31 12:26:35.511547 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.511509    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 31 12:26:35.511547 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.511526    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 31 12:26:35.511547 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.511533    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 31 12:26:35.511547 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.511540    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 31 12:26:35.511731 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.511555    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 31 12:26:35.513367 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.513358    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 31 12:26:35.513403 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.513371    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 31 12:26:35.517543 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.517527    2576 server.go:1263] "Started kubelet"
Oct 31 12:26:35.518311 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.517619    2576 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 31 12:26:35.518311 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.517725    2576 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 31 12:26:35.518311 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.518037    2576 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 31 12:26:35.518316 ip-10-0-130-190 systemd[1]: Started Kubernetes Kubelet.
Oct 31 12:26:35.520241 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.520228    2576 server.go:317] "Adding debug handlers to kubelet server"
Oct 31 12:26:35.520987 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.520889    2576 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 31 12:26:35.526123 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:26:35.526099    2576 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Oct 31 12:26:35.526728 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.526713    2576 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 31 12:26:35.526768 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.526736    2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 31 12:26:35.527587 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:26:35.527005    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-190.us-east-2.compute.internal\" not found"
Oct 31 12:26:35.527587 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.527439    2576 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 31 12:26:35.527587 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.527453    2576 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 31 12:26:35.527587 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.527438    2576 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 31 12:26:35.527587 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.527581    2576 reconstruct.go:97] "Volume reconstruction finished"
Oct 31 12:26:35.527587 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.527590    2576 reconciler.go:29] "Reconciler: start to sync state"
Oct 31 12:26:35.527832 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.527767    2576 factory.go:55] Registering systemd factory
Oct 31 12:26:35.527832 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.527784    2576 factory.go:223] Registration of the systemd container factory successfully
Oct 31 12:26:35.528017 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.528001    2576 factory.go:153] Registering CRI-O factory
Oct 31 12:26:35.528017 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.528017    2576 factory.go:223] Registration of the crio container factory successfully
Oct 31 12:26:35.528175 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.528081    2576 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 31 12:26:35.528175 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.528100    2576 factory.go:103] Registering Raw factory
Oct 31 12:26:35.528175 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.528113    2576 manager.go:1199] Started watching for new ooms in manager
Oct 31 12:26:35.529414 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.529398    2576 manager.go:319] Starting recovery of all containers
Oct 31 12:26:35.531256 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.531221    2576 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-130-190.us-east-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Oct 31 12:26:35.531371 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:26:35.531317    2576 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-130-190.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Oct 31 12:26:35.532047 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:26:35.532024    2576 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-130-190.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Oct 31 12:26:35.532047 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:26:35.532021    2576 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Oct 31 12:26:35.535877 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:26:35.531143    2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-190.us-east-2.compute.internal.18e39ce7af63327e  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-190.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-130-190.us-east-2.compute.internal,},FirstTimestamp:2026-10-31 12:26:35.517481598 +0000 UTC m=+0.410128739,LastTimestamp:2026-10-31 12:26:35.517481598 +0000 UTC m=+0.410128739,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-190.us-east-2.compute.internal,}"
Oct 31 12:26:35.539592 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:26:35.539567    2576 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Oct 31 12:26:35.544419 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.544406    2576 manager.go:324] Recovery completed
Oct 31 12:26:35.546820 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:26:35.546792    2576 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory
Oct 31 12:26:35.548855 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.548834    2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-ts6s8"
Oct 31 12:26:35.549758 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.549747    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 12:26:35.551474 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.551457    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-190.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 12:26:35.551552 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.551503    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-190.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 12:26:35.551552 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.551515    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-190.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 12:26:35.552035 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.552024    2576 cpu_manager.go:226] "Starting" policy="none"
Oct 31 12:26:35.552035 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.552033    2576 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 31 12:26:35.552120 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.552048    2576 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 31 12:26:35.552120 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.552088    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 12:26:35.552822 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.552804    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-190.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 12:26:35.552895 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.552837    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-190.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 12:26:35.552895 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.552851    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-190.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 12:26:35.553658 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:26:35.553574    2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-190.us-east-2.compute.internal.18e39ce7b169de12  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-190.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-130-190.us-east-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-130-190.us-east-2.compute.internal,},FirstTimestamp:2026-10-31 12:26:35.55147317 +0000 UTC m=+0.444120315,LastTimestamp:2026-10-31 12:26:35.55147317 +0000 UTC m=+0.444120315,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-190.us-east-2.compute.internal,}"
Oct 31 12:26:35.554251 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.554235    2576 policy_none.go:50] "Start"
Oct 31 12:26:35.554310 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.554255    2576 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 31 12:26:35.554310 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.554267    2576 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 31 12:26:35.556314 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.556301    2576 policy_none.go:44] "Start"
Oct 31 12:26:35.564174 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.564156    2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-ts6s8"
Oct 31 12:26:35.566003 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:26:35.565918    2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-190.us-east-2.compute.internal.18e39ce7b16a6a30  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-190.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-130-190.us-east-2.compute.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-130-190.us-east-2.compute.internal,},FirstTimestamp:2026-10-31 12:26:35.55150904 +0000 UTC m=+0.444156182,LastTimestamp:2026-10-31 12:26:35.55150904 +0000 UTC m=+0.444156182,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-190.us-east-2.compute.internal,}"
Oct 31 12:26:35.586066 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.586043    2576 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 31 12:26:35.588394 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.588380    2576 manager.go:341] "Starting Device Plugin manager"
Oct 31 12:26:35.588475 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:26:35.588414    2576 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 31 12:26:35.588475 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.588422    2576 server.go:87] "Starting device plugin registration server"
Oct 31 12:26:35.588695 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.588681    2576 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 31 12:26:35.588748 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.588697    2576 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 31 12:26:35.588853 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.588794    2576 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 31 12:26:35.588974 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.588886    2576 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 31 12:26:35.588974 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.588894    2576 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 31 12:26:35.589295 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:26:35.589275    2576 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 31 12:26:35.589361 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:26:35.589325    2576 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-130-190.us-east-2.compute.internal\" not found"
Oct 31 12:26:35.654774 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.654742    2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 31 12:26:35.656756 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.655945    2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 31 12:26:35.656756 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.655964    2576 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 31 12:26:35.656756 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.656001    2576 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 31 12:26:35.656756 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:26:35.656043    2576 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Oct 31 12:26:35.660046 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.660026    2576 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 12:26:35.689138 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.689083    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 12:26:35.692720 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.692704    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-190.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 12:26:35.692793 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.692739    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-190.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 12:26:35.692793 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.692770    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-190.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 12:26:35.692855 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.692806    2576 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-130-190.us-east-2.compute.internal"
Oct 31 12:26:35.702568 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.702554    2576 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-130-190.us-east-2.compute.internal"
Oct 31 12:26:35.702611 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:26:35.702576    2576 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-130-190.us-east-2.compute.internal\": node \"ip-10-0-130-190.us-east-2.compute.internal\" not found"
Oct 31 12:26:35.740769 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:26:35.740739    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-190.us-east-2.compute.internal\" not found"
Oct 31 12:26:35.756420 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.756398    2576 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-190.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-130-190.us-east-2.compute.internal"]
Oct 31 12:26:35.756504 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.756463    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 12:26:35.757392 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.757378    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-190.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 12:26:35.757444 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.757415    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-190.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 12:26:35.757444 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.757425    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-190.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 12:26:35.759700 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.759689    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 12:26:35.759897 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.759883    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-190.us-east-2.compute.internal"
Oct 31 12:26:35.759930 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.759921    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 12:26:35.760499 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.760463    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-190.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 12:26:35.760582 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.760509    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-190.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 12:26:35.760582 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.760531    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-190.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 12:26:35.760582 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.760537    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-190.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 12:26:35.760582 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.760553    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-190.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 12:26:35.760582 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.760563    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-190.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 12:26:35.762735 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.762718    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-190.us-east-2.compute.internal"
Oct 31 12:26:35.762815 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.762758    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 12:26:35.763532 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.763510    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-190.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 12:26:35.763598 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.763541    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-190.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 12:26:35.763598 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.763556    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-190.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 12:26:35.798228 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:26:35.798208    2576 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-190.us-east-2.compute.internal\" not found" node="ip-10-0-130-190.us-east-2.compute.internal"
Oct 31 12:26:35.800826 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:26:35.800811    2576 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-190.us-east-2.compute.internal\" not found" node="ip-10-0-130-190.us-east-2.compute.internal"
Oct 31 12:26:35.829451 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.829430    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/928a0065cb02eec7c5308f7ecf1e7e6c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-190.us-east-2.compute.internal\" (UID: \"928a0065cb02eec7c5308f7ecf1e7e6c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-190.us-east-2.compute.internal"
Oct 31 12:26:35.829538 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.829455    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/928a0065cb02eec7c5308f7ecf1e7e6c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-190.us-east-2.compute.internal\" (UID: \"928a0065cb02eec7c5308f7ecf1e7e6c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-190.us-east-2.compute.internal"
Oct 31 12:26:35.829538 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.829486    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7fb532212935ad861267bbe741c1366b-config\") pod \"kube-apiserver-proxy-ip-10-0-130-190.us-east-2.compute.internal\" (UID: \"7fb532212935ad861267bbe741c1366b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-190.us-east-2.compute.internal"
Oct 31 12:26:35.841556 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:26:35.841536    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-190.us-east-2.compute.internal\" not found"
Oct 31 12:26:35.929977 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.929932    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/928a0065cb02eec7c5308f7ecf1e7e6c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-190.us-east-2.compute.internal\" (UID: \"928a0065cb02eec7c5308f7ecf1e7e6c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-190.us-east-2.compute.internal"
Oct 31 12:26:35.929977 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.929980    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/928a0065cb02eec7c5308f7ecf1e7e6c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-190.us-east-2.compute.internal\" (UID: \"928a0065cb02eec7c5308f7ecf1e7e6c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-190.us-east-2.compute.internal"
Oct 31 12:26:35.930123 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.930027    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7fb532212935ad861267bbe741c1366b-config\") pod \"kube-apiserver-proxy-ip-10-0-130-190.us-east-2.compute.internal\" (UID: \"7fb532212935ad861267bbe741c1366b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-190.us-east-2.compute.internal"
Oct 31 12:26:35.930123 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.930038    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/928a0065cb02eec7c5308f7ecf1e7e6c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-190.us-east-2.compute.internal\" (UID: \"928a0065cb02eec7c5308f7ecf1e7e6c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-190.us-east-2.compute.internal"
Oct 31 12:26:35.930123 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.930058    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7fb532212935ad861267bbe741c1366b-config\") pod \"kube-apiserver-proxy-ip-10-0-130-190.us-east-2.compute.internal\" (UID: \"7fb532212935ad861267bbe741c1366b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-190.us-east-2.compute.internal"
Oct 31 12:26:35.930123 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:35.930115    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/928a0065cb02eec7c5308f7ecf1e7e6c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-190.us-east-2.compute.internal\" (UID: \"928a0065cb02eec7c5308f7ecf1e7e6c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-190.us-east-2.compute.internal"
Oct 31 12:26:35.942078 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:26:35.942032    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-190.us-east-2.compute.internal\" not found"
Oct 31 12:26:36.042823 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:26:36.042792    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-190.us-east-2.compute.internal\" not found"
Oct 31 12:26:36.101003 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:36.100976    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-190.us-east-2.compute.internal"
Oct 31 12:26:36.103949 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:36.103934    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-190.us-east-2.compute.internal"
Oct 31 12:26:36.143715 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:26:36.143510    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-190.us-east-2.compute.internal\" not found"
Oct 31 12:26:36.244038 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:26:36.244009    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-190.us-east-2.compute.internal\" not found"
Oct 31 12:26:36.345093 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:26:36.345050    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-190.us-east-2.compute.internal\" not found"
Oct 31 12:26:36.445554 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:26:36.445521    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-190.us-east-2.compute.internal\" not found"
Oct 31 12:26:36.458628 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:36.458596    2576 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Oct 31 12:26:36.503114 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:36.503053    2576 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 12:26:36.545747 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:26:36.545720    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-190.us-east-2.compute.internal\" not found"
Oct 31 12:26:36.570227 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:36.570175    2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-30 12:21:35 +0000 UTC" deadline="2028-03-28 05:42:16.498745549 +0000 UTC"
Oct 31 12:26:36.570349 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:36.570226    2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12329h15m39.928526396s"
Oct 31 12:26:36.622669 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:36.622641    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 12:26:36.623860 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:36.623839    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-190.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 12:26:36.623965 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:36.623879    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-190.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 12:26:36.623965 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:36.623895    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-190.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 12:26:36.646260 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:26:36.646234    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-190.us-east-2.compute.internal\" not found"
Oct 31 12:26:36.747398 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:26:36.747364    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-190.us-east-2.compute.internal\" not found"
Oct 31 12:26:36.848482 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:26:36.848450    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-190.us-east-2.compute.internal\" not found"
Oct 31 12:26:36.865619 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:36.865578    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod928a0065cb02eec7c5308f7ecf1e7e6c.slice/crio-b61159ca029793099370ff0779dda563c7d5600e24e1415f4ca6047a637d4ed7 WatchSource:0}: Error finding container b61159ca029793099370ff0779dda563c7d5600e24e1415f4ca6047a637d4ed7: Status 404 returned error can't find the container with id b61159ca029793099370ff0779dda563c7d5600e24e1415f4ca6047a637d4ed7
Oct 31 12:26:36.872373 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:36.872356    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 12:26:36.877280 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:36.877257    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fb532212935ad861267bbe741c1366b.slice/crio-b539c38438f2cfd785f2a2a6e023750667ea93e764840b11ad42f359c3687b08 WatchSource:0}: Error finding container b539c38438f2cfd785f2a2a6e023750667ea93e764840b11ad42f359c3687b08: Status 404 returned error can't find the container with id b539c38438f2cfd785f2a2a6e023750667ea93e764840b11ad42f359c3687b08
Oct 31 12:26:36.949510 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:26:36.949477    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-190.us-east-2.compute.internal\" not found"
Oct 31 12:26:36.988620 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:36.988597    2576 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 12:26:37.008309 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.008289    2576 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 12:26:37.027341 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.027310    2576 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-190.us-east-2.compute.internal"
Oct 31 12:26:37.044247 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.044226    2576 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 12:26:37.045245 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.045229    2576 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-190.us-east-2.compute.internal"
Oct 31 12:26:37.055359 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.055343    2576 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 12:26:37.502150 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.502125    2576 apiserver.go:52] "Watching apiserver"
Oct 31 12:26:37.511830 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.511807    2576 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 31 12:26:37.512122 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.512104    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jmlvc","openshift-multus/network-metrics-daemon-94kh9","openshift-ovn-kubernetes/ovnkube-node-86758","openshift-image-registry/node-ca-wdkzp","openshift-multus/multus-additional-cni-plugins-44zj2","openshift-network-diagnostics/network-check-target-tdrmt","openshift-network-operator/iptables-alerter-plbnz","kube-system/kube-apiserver-proxy-ip-10-0-130-190.us-east-2.compute.internal","openshift-dns/node-resolver-v9ks4","openshift-multus/multus-ccps4","kube-system/global-pull-secret-syncer-2bmzr","kube-system/konnectivity-agent-8jlvq","openshift-cluster-node-tuning-operator/tuned-qkt2l","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-190.us-east-2.compute.internal"]
Oct 31 12:26:37.514701 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.514680    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-plbnz"
Oct 31 12:26:37.517083 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.517062    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-qkt2l"
Oct 31 12:26:37.518933 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.518911    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-ng46x\""
Oct 31 12:26:37.519046 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.518919    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 31 12:26:37.519118 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.519047    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 31 12:26:37.519118 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.519050    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 31 12:26:37.519528 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.519512    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-86758"
Oct 31 12:26:37.520765 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.520747    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 31 12:26:37.520856 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.520786    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 31 12:26:37.520856 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.520802    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-8ljk6\""
Oct 31 12:26:37.521819 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.521798    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-wdkzp"
Oct 31 12:26:37.523330 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.523311    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 31 12:26:37.523438 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.523388    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 31 12:26:37.523523 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.523445    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 31 12:26:37.523523 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.523451    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 31 12:26:37.523523 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.523461    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 31 12:26:37.523663 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.523584    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-hzb2l\""
Oct 31 12:26:37.524233 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.524215    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-44zj2"
Oct 31 12:26:37.525194 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.525175    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 31 12:26:37.525468 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.525447    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 31 12:26:37.525595 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.525520    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 31 12:26:37.525595 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.525538    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-kldqp\""
Oct 31 12:26:37.525676 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.525597    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 31 12:26:37.526487 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.526469    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tdrmt"
Oct 31 12:26:37.526599 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:26:37.526578    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tdrmt" podUID="7a402472-ca33-44ae-b598-96fc8e7a5eb2"
Oct 31 12:26:37.527826 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.527801    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 31 12:26:37.532106 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.528060    2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Oct 31 12:26:37.532106 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.528279    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-m5kv7\""
Oct 31 12:26:37.532106 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.528291    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 31 12:26:37.532106 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.528296    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 31 12:26:37.532106 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.528390    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 31 12:26:37.532106 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.528412    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 31 12:26:37.533099 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.532452    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-v9ks4"
Oct 31 12:26:37.534473 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.534450    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jmlvc"
Oct 31 12:26:37.537041 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.537020    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-94kh9"
Oct 31 12:26:37.537141 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:26:37.537122    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-94kh9" podUID="e64bd69d-946f-42ad-94ae-b87b7de3a581"
Oct 31 12:26:37.538681 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.538638    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d43a4e16-3834-4d31-bed4-42fc0aaaaf17-host-run-ovn-kubernetes\") pod \"ovnkube-node-86758\" (UID: \"d43a4e16-3834-4d31-bed4-42fc0aaaaf17\") " pod="openshift-ovn-kubernetes/ovnkube-node-86758"
Oct 31 12:26:37.538681 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.538671    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d43a4e16-3834-4d31-bed4-42fc0aaaaf17-host-cni-netd\") pod \"ovnkube-node-86758\" (UID: \"d43a4e16-3834-4d31-bed4-42fc0aaaaf17\") " pod="openshift-ovn-kubernetes/ovnkube-node-86758"
Oct 31 12:26:37.538681 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.538693    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tf6n\" (UniqueName: \"kubernetes.io/projected/7a402472-ca33-44ae-b598-96fc8e7a5eb2-kube-api-access-2tf6n\") pod \"network-check-target-tdrmt\" (UID: \"7a402472-ca33-44ae-b598-96fc8e7a5eb2\") " pod="openshift-network-diagnostics/network-check-target-tdrmt"
Oct 31 12:26:37.538922 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.538712    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nx5hk\" (UniqueName: \"kubernetes.io/projected/6c661ce8-9f7b-4509-bd17-2380bcd1c425-kube-api-access-nx5hk\") pod \"node-ca-wdkzp\" (UID: \"6c661ce8-9f7b-4509-bd17-2380bcd1c425\") " pod="openshift-image-registry/node-ca-wdkzp"
Oct 31 12:26:37.538922 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.538768    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7j2rp\" (UniqueName: \"kubernetes.io/projected/6bbe13aa-df4b-40b0-8595-68fe1166a082-kube-api-access-7j2rp\") pod \"node-resolver-v9ks4\" (UID: \"6bbe13aa-df4b-40b0-8595-68fe1166a082\") " pod="openshift-dns/node-resolver-v9ks4"
Oct 31 12:26:37.538922 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.538834    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3bb2c04e-55a2-4206-9530-7298f4469379-iptables-alerter-script\") pod \"iptables-alerter-plbnz\" (UID: \"3bb2c04e-55a2-4206-9530-7298f4469379\") " pod="openshift-network-operator/iptables-alerter-plbnz"
Oct 31 12:26:37.538922 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.538856    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xv9qt\" (UniqueName: \"kubernetes.io/projected/3bb2c04e-55a2-4206-9530-7298f4469379-kube-api-access-xv9qt\") pod \"iptables-alerter-plbnz\" (UID: \"3bb2c04e-55a2-4206-9530-7298f4469379\") " pod="openshift-network-operator/iptables-alerter-plbnz"
Oct 31 12:26:37.538922 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.538884    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee-etc-sysctl-conf\") pod \"tuned-qkt2l\" (UID: \"51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee\") " pod="openshift-cluster-node-tuning-operator/tuned-qkt2l"
Oct 31 12:26:37.539208 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.538935    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee-lib-modules\") pod \"tuned-qkt2l\" (UID: \"51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee\") " pod="openshift-cluster-node-tuning-operator/tuned-qkt2l"
Oct 31 12:26:37.539208 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.539016    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a1c3175b-e32e-420e-877f-71bfe64d9c6c-cnibin\") pod \"multus-additional-cni-plugins-44zj2\" (UID: \"a1c3175b-e32e-420e-877f-71bfe64d9c6c\") " pod="openshift-multus/multus-additional-cni-plugins-44zj2"
Oct 31 12:26:37.539208 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.539051    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee-host\") pod \"tuned-qkt2l\" (UID: \"51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee\") " pod="openshift-cluster-node-tuning-operator/tuned-qkt2l"
Oct 31 12:26:37.539208 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.539118    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee-etc-tuned\") pod \"tuned-qkt2l\" (UID: \"51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee\") " pod="openshift-cluster-node-tuning-operator/tuned-qkt2l"
Oct 31 12:26:37.539208 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.539151    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5nx4\" (UniqueName: \"kubernetes.io/projected/51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee-kube-api-access-t5nx4\") pod \"tuned-qkt2l\" (UID: \"51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee\") " pod="openshift-cluster-node-tuning-operator/tuned-qkt2l"
Oct 31 12:26:37.539208 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.539184    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d43a4e16-3834-4d31-bed4-42fc0aaaaf17-host-kubelet\") pod \"ovnkube-node-86758\" (UID: \"d43a4e16-3834-4d31-bed4-42fc0aaaaf17\") " pod="openshift-ovn-kubernetes/ovnkube-node-86758"
Oct 31 12:26:37.539208 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.539200    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d43a4e16-3834-4d31-bed4-42fc0aaaaf17-host-run-netns\") pod \"ovnkube-node-86758\" (UID: \"d43a4e16-3834-4d31-bed4-42fc0aaaaf17\") " pod="openshift-ovn-kubernetes/ovnkube-node-86758"
Oct 31 12:26:37.539558 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.539223    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzcgz\" (UniqueName: \"kubernetes.io/projected/d43a4e16-3834-4d31-bed4-42fc0aaaaf17-kube-api-access-hzcgz\") pod \"ovnkube-node-86758\" (UID: \"d43a4e16-3834-4d31-bed4-42fc0aaaaf17\") " pod="openshift-ovn-kubernetes/ovnkube-node-86758"
Oct 31 12:26:37.539558 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.539250    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a1c3175b-e32e-420e-877f-71bfe64d9c6c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-44zj2\" (UID: \"a1c3175b-e32e-420e-877f-71bfe64d9c6c\") " pod="openshift-multus/multus-additional-cni-plugins-44zj2"
Oct 31 12:26:37.539558 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.539276    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a1c3175b-e32e-420e-877f-71bfe64d9c6c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-44zj2\" (UID: \"a1c3175b-e32e-420e-877f-71bfe64d9c6c\") " pod="openshift-multus/multus-additional-cni-plugins-44zj2"
Oct 31 12:26:37.539558 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.539326    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee-run\") pod \"tuned-qkt2l\" (UID: \"51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee\") " pod="openshift-cluster-node-tuning-operator/tuned-qkt2l"
Oct 31 12:26:37.539558 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.539368    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee-var-lib-kubelet\") pod \"tuned-qkt2l\" (UID: \"51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee\") " pod="openshift-cluster-node-tuning-operator/tuned-qkt2l"
Oct 31 12:26:37.539558 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.539391    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d43a4e16-3834-4d31-bed4-42fc0aaaaf17-systemd-units\") pod \"ovnkube-node-86758\" (UID: \"d43a4e16-3834-4d31-bed4-42fc0aaaaf17\") " pod="openshift-ovn-kubernetes/ovnkube-node-86758"
Oct 31 12:26:37.539558 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.539434    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d43a4e16-3834-4d31-bed4-42fc0aaaaf17-ovn-node-metrics-cert\") pod \"ovnkube-node-86758\" (UID: \"d43a4e16-3834-4d31-bed4-42fc0aaaaf17\") " pod="openshift-ovn-kubernetes/ovnkube-node-86758"
Oct 31 12:26:37.539558 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.539538    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a1c3175b-e32e-420e-877f-71bfe64d9c6c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-44zj2\" (UID: \"a1c3175b-e32e-420e-877f-71bfe64d9c6c\") " pod="openshift-multus/multus-additional-cni-plugins-44zj2"
Oct 31 12:26:37.539558 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.539569    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6c661ce8-9f7b-4509-bd17-2380bcd1c425-serviceca\") pod \"node-ca-wdkzp\" (UID: \"6c661ce8-9f7b-4509-bd17-2380bcd1c425\") " pod="openshift-image-registry/node-ca-wdkzp"
Oct 31 12:26:37.540091 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.539593    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee-etc-sysconfig\") pod \"tuned-qkt2l\" (UID: \"51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee\") " pod="openshift-cluster-node-tuning-operator/tuned-qkt2l"
Oct 31 12:26:37.540091 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.539655    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-ccps4"
Oct 31 12:26:37.540091 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.539657    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee-etc-kubernetes\") pod \"tuned-qkt2l\" (UID: \"51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee\") " pod="openshift-cluster-node-tuning-operator/tuned-qkt2l"
Oct 31 12:26:37.540091 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.539738    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee-sys\") pod \"tuned-qkt2l\" (UID: \"51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee\") " pod="openshift-cluster-node-tuning-operator/tuned-qkt2l"
Oct 31 12:26:37.540091 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.539758    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d43a4e16-3834-4d31-bed4-42fc0aaaaf17-etc-openvswitch\") pod \"ovnkube-node-86758\" (UID: \"d43a4e16-3834-4d31-bed4-42fc0aaaaf17\") " pod="openshift-ovn-kubernetes/ovnkube-node-86758"
Oct 31 12:26:37.540091 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.539781    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d43a4e16-3834-4d31-bed4-42fc0aaaaf17-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-86758\" (UID: \"d43a4e16-3834-4d31-bed4-42fc0aaaaf17\") " pod="openshift-ovn-kubernetes/ovnkube-node-86758"
Oct 31 12:26:37.540091 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.539810    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6c661ce8-9f7b-4509-bd17-2380bcd1c425-host\") pod \"node-ca-wdkzp\" (UID: \"6c661ce8-9f7b-4509-bd17-2380bcd1c425\") " pod="openshift-image-registry/node-ca-wdkzp"
Oct 31 12:26:37.540091 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.539833    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d43a4e16-3834-4d31-bed4-42fc0aaaaf17-run-ovn\") pod \"ovnkube-node-86758\" (UID: \"d43a4e16-3834-4d31-bed4-42fc0aaaaf17\") " pod="openshift-ovn-kubernetes/ovnkube-node-86758"
Oct 31 12:26:37.540091 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.539858    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d43a4e16-3834-4d31-bed4-42fc0aaaaf17-env-overrides\") pod \"ovnkube-node-86758\" (UID: \"d43a4e16-3834-4d31-bed4-42fc0aaaaf17\") " pod="openshift-ovn-kubernetes/ovnkube-node-86758"
Oct 31 12:26:37.540091 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.539889    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6bbe13aa-df4b-40b0-8595-68fe1166a082-hosts-file\") pod \"node-resolver-v9ks4\" (UID: \"6bbe13aa-df4b-40b0-8595-68fe1166a082\") " pod="openshift-dns/node-resolver-v9ks4"
Oct 31 12:26:37.540091 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.539944    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3bb2c04e-55a2-4206-9530-7298f4469379-host-slash\") pod \"iptables-alerter-plbnz\" (UID: \"3bb2c04e-55a2-4206-9530-7298f4469379\") " pod="openshift-network-operator/iptables-alerter-plbnz"
Oct 31 12:26:37.540091 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.539984    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee-etc-systemd\") pod \"tuned-qkt2l\" (UID: \"51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee\") " pod="openshift-cluster-node-tuning-operator/tuned-qkt2l"
Oct 31 12:26:37.540091 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.540027    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d43a4e16-3834-4d31-bed4-42fc0aaaaf17-host-slash\") pod \"ovnkube-node-86758\" (UID: \"d43a4e16-3834-4d31-bed4-42fc0aaaaf17\") " pod="openshift-ovn-kubernetes/ovnkube-node-86758"
Oct 31 12:26:37.540091 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.540054    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d43a4e16-3834-4d31-bed4-42fc0aaaaf17-log-socket\") pod \"ovnkube-node-86758\" (UID: \"d43a4e16-3834-4d31-bed4-42fc0aaaaf17\") " pod="openshift-ovn-kubernetes/ovnkube-node-86758"
Oct 31 12:26:37.540091 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.540070    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d43a4e16-3834-4d31-bed4-42fc0aaaaf17-host-cni-bin\") pod \"ovnkube-node-86758\" (UID: \"d43a4e16-3834-4d31-bed4-42fc0aaaaf17\") " pod="openshift-ovn-kubernetes/ovnkube-node-86758"
Oct 31 12:26:37.541383 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.540107    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d43a4e16-3834-4d31-bed4-42fc0aaaaf17-ovnkube-config\") pod \"ovnkube-node-86758\" (UID: \"d43a4e16-3834-4d31-bed4-42fc0aaaaf17\") " pod="openshift-ovn-kubernetes/ovnkube-node-86758"
Oct 31 12:26:37.541383 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.540135    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee-tmp\") pod \"tuned-qkt2l\" (UID: \"51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee\") " pod="openshift-cluster-node-tuning-operator/tuned-qkt2l"
Oct 31 12:26:37.541383 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.540151    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d43a4e16-3834-4d31-bed4-42fc0aaaaf17-run-systemd\") pod \"ovnkube-node-86758\" (UID: \"d43a4e16-3834-4d31-bed4-42fc0aaaaf17\") " pod="openshift-ovn-kubernetes/ovnkube-node-86758"
Oct 31 12:26:37.541383 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.540165    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d43a4e16-3834-4d31-bed4-42fc0aaaaf17-run-openvswitch\") pod \"ovnkube-node-86758\" (UID: \"d43a4e16-3834-4d31-bed4-42fc0aaaaf17\") " pod="openshift-ovn-kubernetes/ovnkube-node-86758"
Oct 31 12:26:37.541383 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.540188    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d43a4e16-3834-4d31-bed4-42fc0aaaaf17-node-log\") pod \"ovnkube-node-86758\" (UID: \"d43a4e16-3834-4d31-bed4-42fc0aaaaf17\") " pod="openshift-ovn-kubernetes/ovnkube-node-86758"
Oct 31 12:26:37.541383 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.540203    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d43a4e16-3834-4d31-bed4-42fc0aaaaf17-ovnkube-script-lib\") pod \"ovnkube-node-86758\" (UID: \"d43a4e16-3834-4d31-bed4-42fc0aaaaf17\") " pod="openshift-ovn-kubernetes/ovnkube-node-86758"
Oct 31 12:26:37.541383 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.540245    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a1c3175b-e32e-420e-877f-71bfe64d9c6c-system-cni-dir\") pod \"multus-additional-cni-plugins-44zj2\" (UID: \"a1c3175b-e32e-420e-877f-71bfe64d9c6c\") " pod="openshift-multus/multus-additional-cni-plugins-44zj2"
Oct 31 12:26:37.541383 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.540263    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a1c3175b-e32e-420e-877f-71bfe64d9c6c-cni-binary-copy\") pod \"multus-additional-cni-plugins-44zj2\" (UID: \"a1c3175b-e32e-420e-877f-71bfe64d9c6c\") " pod="openshift-multus/multus-additional-cni-plugins-44zj2"
Oct 31 12:26:37.541383 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.540286    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwj8z\" (UniqueName: \"kubernetes.io/projected/a1c3175b-e32e-420e-877f-71bfe64d9c6c-kube-api-access-nwj8z\") pod \"multus-additional-cni-plugins-44zj2\" (UID: \"a1c3175b-e32e-420e-877f-71bfe64d9c6c\") " pod="openshift-multus/multus-additional-cni-plugins-44zj2"
Oct 31 12:26:37.541383 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.540311    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6bbe13aa-df4b-40b0-8595-68fe1166a082-tmp-dir\") pod \"node-resolver-v9ks4\" (UID: \"6bbe13aa-df4b-40b0-8595-68fe1166a082\") " pod="openshift-dns/node-resolver-v9ks4"
Oct 31 12:26:37.541383 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.540335    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee-etc-modprobe-d\") pod \"tuned-qkt2l\" (UID: \"51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee\") " pod="openshift-cluster-node-tuning-operator/tuned-qkt2l"
Oct 31 12:26:37.541383 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.540350    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee-etc-sysctl-d\") pod \"tuned-qkt2l\" (UID: \"51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee\") " pod="openshift-cluster-node-tuning-operator/tuned-qkt2l"
Oct 31 12:26:37.541383 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.540366    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d43a4e16-3834-4d31-bed4-42fc0aaaaf17-var-lib-openvswitch\") pod \"ovnkube-node-86758\" (UID: \"d43a4e16-3834-4d31-bed4-42fc0aaaaf17\") " pod="openshift-ovn-kubernetes/ovnkube-node-86758"
Oct 31 12:26:37.542406 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.542382    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2bmzr"
Oct 31 12:26:37.542554 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:26:37.542524    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2bmzr" podUID="a2b450be-ca79-4edf-a1ea-377531a49519"
Oct 31 12:26:37.545936 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.545918    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-8jlvq"
Oct 31 12:26:37.565226 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.565197    2576 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 31 12:26:37.565333 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.565275    2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-h78pz"
Oct 31 12:26:37.576730 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.576709    2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-h78pz"
Oct 31 12:26:37.628018 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.627989    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 31 12:26:37.628180 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.628155    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 31 12:26:37.628180 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.628174    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 31 12:26:37.628382 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.628352    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 31 12:26:37.628456 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.628406    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 31 12:26:37.628456 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.628420    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 31 12:26:37.628597 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.628491    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 31 12:26:37.628597 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.628576    2576 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 31 12:26:37.629657 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.629634    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 31 12:26:37.629657 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.629636    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-df646\""
Oct 31 12:26:37.629824 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.629675    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-9d2j7\""
Oct 31 12:26:37.629824 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.629684    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-bbb6f\""
Oct 31 12:26:37.629824 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.629750    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-x5rvd\""
Oct 31 12:26:37.641447 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.641419    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a2b450be-ca79-4edf-a1ea-377531a49519-dbus\") pod \"global-pull-secret-syncer-2bmzr\" (UID: \"a2b450be-ca79-4edf-a1ea-377531a49519\") " pod="kube-system/global-pull-secret-syncer-2bmzr"
Oct 31 12:26:37.641615 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.641464    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6c661ce8-9f7b-4509-bd17-2380bcd1c425-host\") pod \"node-ca-wdkzp\" (UID: \"6c661ce8-9f7b-4509-bd17-2380bcd1c425\") " pod="openshift-image-registry/node-ca-wdkzp"
Oct 31 12:26:37.641615 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.641595    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d43a4e16-3834-4d31-bed4-42fc0aaaaf17-env-overrides\") pod \"ovnkube-node-86758\" (UID: \"d43a4e16-3834-4d31-bed4-42fc0aaaaf17\") " pod="openshift-ovn-kubernetes/ovnkube-node-86758"
Oct 31 12:26:37.641748 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.641692    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6c661ce8-9f7b-4509-bd17-2380bcd1c425-host\") pod \"node-ca-wdkzp\" (UID: \"6c661ce8-9f7b-4509-bd17-2380bcd1c425\") " pod="openshift-image-registry/node-ca-wdkzp"
Oct 31 12:26:37.641748 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.641731    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6b3e77ff-4445-42fa-a7c3-f35d442c276d-host-var-lib-kubelet\") pod \"multus-ccps4\" (UID: \"6b3e77ff-4445-42fa-a7c3-f35d442c276d\") " pod="openshift-multus/multus-ccps4"
Oct 31 12:26:37.641897 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.641760    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6bbe13aa-df4b-40b0-8595-68fe1166a082-hosts-file\") pod \"node-resolver-v9ks4\" (UID: \"6bbe13aa-df4b-40b0-8595-68fe1166a082\") " pod="openshift-dns/node-resolver-v9ks4"
Oct 31 12:26:37.641897 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.641809    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d43a4e16-3834-4d31-bed4-42fc0aaaaf17-host-slash\") pod \"ovnkube-node-86758\" (UID: \"d43a4e16-3834-4d31-bed4-42fc0aaaaf17\") " pod="openshift-ovn-kubernetes/ovnkube-node-86758"
Oct 31 12:26:37.641897 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.641835    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d43a4e16-3834-4d31-bed4-42fc0aaaaf17-host-cni-bin\") pod \"ovnkube-node-86758\" (UID: \"d43a4e16-3834-4d31-bed4-42fc0aaaaf17\") " pod="openshift-ovn-kubernetes/ovnkube-node-86758"
Oct 31 12:26:37.641897 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.641872    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6b3e77ff-4445-42fa-a7c3-f35d442c276d-host-run-multus-certs\") pod \"multus-ccps4\" (UID: \"6b3e77ff-4445-42fa-a7c3-f35d442c276d\") " pod="openshift-multus/multus-ccps4"
Oct 31 12:26:37.642101 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.641943    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d43a4e16-3834-4d31-bed4-42fc0aaaaf17-host-cni-bin\") pod \"ovnkube-node-86758\" (UID: \"d43a4e16-3834-4d31-bed4-42fc0aaaaf17\") " pod="openshift-ovn-kubernetes/ovnkube-node-86758"
Oct 31 12:26:37.642101 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.641977    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a1c3175b-e32e-420e-877f-71bfe64d9c6c-system-cni-dir\") pod \"multus-additional-cni-plugins-44zj2\" (UID: \"a1c3175b-e32e-420e-877f-71bfe64d9c6c\") " pod="openshift-multus/multus-additional-cni-plugins-44zj2"
Oct 31 12:26:37.642101 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.642004    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a1c3175b-e32e-420e-877f-71bfe64d9c6c-cni-binary-copy\") pod \"multus-additional-cni-plugins-44zj2\" (UID: \"a1c3175b-e32e-420e-877f-71bfe64d9c6c\") " pod="openshift-multus/multus-additional-cni-plugins-44zj2"
Oct 31 12:26:37.642101 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.642039    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwj8z\" (UniqueName: \"kubernetes.io/projected/a1c3175b-e32e-420e-877f-71bfe64d9c6c-kube-api-access-nwj8z\") pod \"multus-additional-cni-plugins-44zj2\" (UID: \"a1c3175b-e32e-420e-877f-71bfe64d9c6c\") " pod="openshift-multus/multus-additional-cni-plugins-44zj2"
Oct 31 12:26:37.642101 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.642068    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e64bd69d-946f-42ad-94ae-b87b7de3a581-metrics-certs\") pod \"network-metrics-daemon-94kh9\" (UID: \"e64bd69d-946f-42ad-94ae-b87b7de3a581\") " pod="openshift-multus/network-metrics-daemon-94kh9"
Oct 31 12:26:37.642346 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.642242    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqk48\" (UniqueName: \"kubernetes.io/projected/e64bd69d-946f-42ad-94ae-b87b7de3a581-kube-api-access-gqk48\") pod \"network-metrics-daemon-94kh9\" (UID: \"e64bd69d-946f-42ad-94ae-b87b7de3a581\") " pod="openshift-multus/network-metrics-daemon-94kh9"
Oct 31 12:26:37.642346 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.642272    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a1e578b0-2e77-4f52-8a44-379b6059c3d9-device-dir\") pod \"aws-ebs-csi-driver-node-jmlvc\" (UID: \"a1e578b0-2e77-4f52-8a44-379b6059c3d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jmlvc"
Oct 31 12:26:37.642346 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.642299    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6b3e77ff-4445-42fa-a7c3-f35d442c276d-host-run-netns\") pod \"multus-ccps4\" (UID: \"6b3e77ff-4445-42fa-a7c3-f35d442c276d\") " pod="openshift-multus/multus-ccps4"
Oct 31 12:26:37.642509 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.642353    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b5014b57-9b83-4fc9-b0ed-55793b2b3a98-agent-certs\") pod \"konnectivity-agent-8jlvq\" (UID: \"b5014b57-9b83-4fc9-b0ed-55793b2b3a98\") " pod="kube-system/konnectivity-agent-8jlvq"
Oct 31 12:26:37.642509 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.642381    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6bbe13aa-df4b-40b0-8595-68fe1166a082-tmp-dir\") pod \"node-resolver-v9ks4\" (UID: \"6bbe13aa-df4b-40b0-8595-68fe1166a082\") " pod="openshift-dns/node-resolver-v9ks4"
Oct 31 12:26:37.642509 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.642410    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee-etc-modprobe-d\") pod \"tuned-qkt2l\" (UID: \"51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee\") " pod="openshift-cluster-node-tuning-operator/tuned-qkt2l"
Oct 31 12:26:37.642509 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.642437    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d43a4e16-3834-4d31-bed4-42fc0aaaaf17-host-run-ovn-kubernetes\") pod \"ovnkube-node-86758\" (UID: \"d43a4e16-3834-4d31-bed4-42fc0aaaaf17\") " pod="openshift-ovn-kubernetes/ovnkube-node-86758"
Oct 31 12:26:37.642509 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.642464    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6b3e77ff-4445-42fa-a7c3-f35d442c276d-system-cni-dir\") pod \"multus-ccps4\" (UID: \"6b3e77ff-4445-42fa-a7c3-f35d442c276d\") " pod="openshift-multus/multus-ccps4"
Oct 31 12:26:37.642819 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.642503    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6b3e77ff-4445-42fa-a7c3-f35d442c276d-cni-binary-copy\") pod \"multus-ccps4\" (UID: \"6b3e77ff-4445-42fa-a7c3-f35d442c276d\") " pod="openshift-multus/multus-ccps4"
Oct 31 12:26:37.642819 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.642613    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6bbe13aa-df4b-40b0-8595-68fe1166a082-hosts-file\") pod \"node-resolver-v9ks4\" (UID: \"6bbe13aa-df4b-40b0-8595-68fe1166a082\") " pod="openshift-dns/node-resolver-v9ks4"
Oct 31 12:26:37.642819 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.642624    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d43a4e16-3834-4d31-bed4-42fc0aaaaf17-env-overrides\") pod \"ovnkube-node-86758\" (UID: \"d43a4e16-3834-4d31-bed4-42fc0aaaaf17\") " pod="openshift-ovn-kubernetes/ovnkube-node-86758"
Oct 31 12:26:37.642819 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.642657    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d43a4e16-3834-4d31-bed4-42fc0aaaaf17-host-slash\") pod \"ovnkube-node-86758\" (UID: \"d43a4e16-3834-4d31-bed4-42fc0aaaaf17\") " pod="openshift-ovn-kubernetes/ovnkube-node-86758"
Oct 31 12:26:37.642819 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.642696    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6b3e77ff-4445-42fa-a7c3-f35d442c276d-host-var-lib-cni-multus\") pod \"multus-ccps4\" (UID: \"6b3e77ff-4445-42fa-a7c3-f35d442c276d\") " pod="openshift-multus/multus-ccps4"
Oct 31 12:26:37.642819 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.642698    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee-etc-modprobe-d\") pod \"tuned-qkt2l\" (UID: \"51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee\") " pod="openshift-cluster-node-tuning-operator/tuned-qkt2l"
Oct 31 12:26:37.642819 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.642719    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d43a4e16-3834-4d31-bed4-42fc0aaaaf17-host-run-ovn-kubernetes\") pod \"ovnkube-node-86758\" (UID: \"d43a4e16-3834-4d31-bed4-42fc0aaaaf17\") " pod="openshift-ovn-kubernetes/ovnkube-node-86758"
Oct 31 12:26:37.642819 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.642728    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7j2rp\" (UniqueName: \"kubernetes.io/projected/6bbe13aa-df4b-40b0-8595-68fe1166a082-kube-api-access-7j2rp\") pod \"node-resolver-v9ks4\" (UID: \"6bbe13aa-df4b-40b0-8595-68fe1166a082\") " pod="openshift-dns/node-resolver-v9ks4"
Oct 31 12:26:37.642819 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.642770    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a1c3175b-e32e-420e-877f-71bfe64d9c6c-system-cni-dir\") pod \"multus-additional-cni-plugins-44zj2\" (UID: \"a1c3175b-e32e-420e-877f-71bfe64d9c6c\") " pod="openshift-multus/multus-additional-cni-plugins-44zj2"
Oct 31 12:26:37.642819 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.642818    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xv9qt\" (UniqueName: \"kubernetes.io/projected/3bb2c04e-55a2-4206-9530-7298f4469379-kube-api-access-xv9qt\") pod \"iptables-alerter-plbnz\" (UID: \"3bb2c04e-55a2-4206-9530-7298f4469379\") " pod="openshift-network-operator/iptables-alerter-plbnz"
Oct 31 12:26:37.643342 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.642856    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a1c3175b-e32e-420e-877f-71bfe64d9c6c-cni-binary-copy\") pod \"multus-additional-cni-plugins-44zj2\" (UID: \"a1c3175b-e32e-420e-877f-71bfe64d9c6c\") " pod="openshift-multus/multus-additional-cni-plugins-44zj2"
Oct 31 12:26:37.643342 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.642948    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee-lib-modules\") pod \"tuned-qkt2l\" (UID: \"51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee\") " pod="openshift-cluster-node-tuning-operator/tuned-qkt2l"
Oct 31 12:26:37.643342 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.642984    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6bbe13aa-df4b-40b0-8595-68fe1166a082-tmp-dir\") pod \"node-resolver-v9ks4\" (UID: \"6bbe13aa-df4b-40b0-8595-68fe1166a082\") " pod="openshift-dns/node-resolver-v9ks4"
Oct 31 12:26:37.643342 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.643069    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a1c3175b-e32e-420e-877f-71bfe64d9c6c-cnibin\") pod \"multus-additional-cni-plugins-44zj2\" (UID: \"a1c3175b-e32e-420e-877f-71bfe64d9c6c\") " pod="openshift-multus/multus-additional-cni-plugins-44zj2"
Oct 31 12:26:37.643342 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.643102    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6b3e77ff-4445-42fa-a7c3-f35d442c276d-multus-daemon-config\") pod \"multus-ccps4\" (UID: \"6b3e77ff-4445-42fa-a7c3-f35d442c276d\") " pod="openshift-multus/multus-ccps4"
Oct 31 12:26:37.643342 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.643133    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vknw2\" (UniqueName: \"kubernetes.io/projected/6b3e77ff-4445-42fa-a7c3-f35d442c276d-kube-api-access-vknw2\") pod \"multus-ccps4\" (UID: \"6b3e77ff-4445-42fa-a7c3-f35d442c276d\") " pod="openshift-multus/multus-ccps4"
Oct 31 12:26:37.643342 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.643147    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a1c3175b-e32e-420e-877f-71bfe64d9c6c-cnibin\") pod \"multus-additional-cni-plugins-44zj2\" (UID: \"a1c3175b-e32e-420e-877f-71bfe64d9c6c\") " pod="openshift-multus/multus-additional-cni-plugins-44zj2"
Oct 31 12:26:37.643342 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.643161    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee-etc-tuned\") pod \"tuned-qkt2l\" (UID: \"51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee\") " pod="openshift-cluster-node-tuning-operator/tuned-qkt2l"
Oct 31 12:26:37.643342 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.643231    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5nx4\" (UniqueName: \"kubernetes.io/projected/51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee-kube-api-access-t5nx4\") pod \"tuned-qkt2l\" (UID: \"51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee\") " pod="openshift-cluster-node-tuning-operator/tuned-qkt2l"
Oct 31 12:26:37.643342 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.643289    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee-lib-modules\") pod \"tuned-qkt2l\" (UID: \"51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee\") " pod="openshift-cluster-node-tuning-operator/tuned-qkt2l"
Oct 31 12:26:37.643342 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.643299    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d43a4e16-3834-4d31-bed4-42fc0aaaaf17-host-kubelet\") pod \"ovnkube-node-86758\" (UID: \"d43a4e16-3834-4d31-bed4-42fc0aaaaf17\") " pod="openshift-ovn-kubernetes/ovnkube-node-86758"
Oct 31 12:26:37.643882 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.643403    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d43a4e16-3834-4d31-bed4-42fc0aaaaf17-host-run-netns\") pod \"ovnkube-node-86758\" (UID: \"d43a4e16-3834-4d31-bed4-42fc0aaaaf17\") " pod="openshift-ovn-kubernetes/ovnkube-node-86758"
Oct 31 12:26:37.643882 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.643443    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzcgz\" (UniqueName: \"kubernetes.io/projected/d43a4e16-3834-4d31-bed4-42fc0aaaaf17-kube-api-access-hzcgz\") pod \"ovnkube-node-86758\" (UID: \"d43a4e16-3834-4d31-bed4-42fc0aaaaf17\") " pod="openshift-ovn-kubernetes/ovnkube-node-86758"
Oct 31 12:26:37.643882 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.643513    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6b3e77ff-4445-42fa-a7c3-f35d442c276d-hostroot\") pod \"multus-ccps4\" (UID: \"6b3e77ff-4445-42fa-a7c3-f35d442c276d\") " pod="openshift-multus/multus-ccps4"
Oct 31 12:26:37.643882 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.643541    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee-run\") pod \"tuned-qkt2l\" (UID: \"51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee\") " pod="openshift-cluster-node-tuning-operator/tuned-qkt2l"
Oct 31 12:26:37.643882 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.643536    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d43a4e16-3834-4d31-bed4-42fc0aaaaf17-host-kubelet\") pod \"ovnkube-node-86758\" (UID: \"d43a4e16-3834-4d31-bed4-42fc0aaaaf17\") " pod="openshift-ovn-kubernetes/ovnkube-node-86758"
Oct 31 12:26:37.643882 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.643556    2576 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 31 12:26:37.643882 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.643604    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee-var-lib-kubelet\") pod \"tuned-qkt2l\" (UID: \"51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee\") " pod="openshift-cluster-node-tuning-operator/tuned-qkt2l"
Oct 31 12:26:37.643882 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.643659    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d43a4e16-3834-4d31-bed4-42fc0aaaaf17-ovn-node-metrics-cert\") pod \"ovnkube-node-86758\" (UID: \"d43a4e16-3834-4d31-bed4-42fc0aaaaf17\") " pod="openshift-ovn-kubernetes/ovnkube-node-86758"
Oct 31 12:26:37.643882 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.643684    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee-run\") pod \"tuned-qkt2l\" (UID: \"51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee\") " pod="openshift-cluster-node-tuning-operator/tuned-qkt2l"
Oct 31 12:26:37.643882 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.643709    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee-var-lib-kubelet\") pod \"tuned-qkt2l\" (UID: \"51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee\") " pod="openshift-cluster-node-tuning-operator/tuned-qkt2l"
Oct 31 12:26:37.643882 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.643767    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a2b450be-ca79-4edf-a1ea-377531a49519-kubelet-config\") pod \"global-pull-secret-syncer-2bmzr\" (UID: \"a2b450be-ca79-4edf-a1ea-377531a49519\") " pod="kube-system/global-pull-secret-syncer-2bmzr"
Oct 31 12:26:37.643882 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.643845    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6c661ce8-9f7b-4509-bd17-2380bcd1c425-serviceca\") pod \"node-ca-wdkzp\" (UID: \"6c661ce8-9f7b-4509-bd17-2380bcd1c425\") " pod="openshift-image-registry/node-ca-wdkzp"
Oct 31 12:26:37.644392 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.643898    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee-etc-kubernetes\") pod \"tuned-qkt2l\" (UID: \"51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee\") " pod="openshift-cluster-node-tuning-operator/tuned-qkt2l"
Oct 31 12:26:37.644392 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.643954    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee-sys\") pod \"tuned-qkt2l\" (UID: \"51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee\") " pod="openshift-cluster-node-tuning-operator/tuned-qkt2l"
Oct 31 12:26:37.644392 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.643981    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d43a4e16-3834-4d31-bed4-42fc0aaaaf17-etc-openvswitch\") pod \"ovnkube-node-86758\" (UID: \"d43a4e16-3834-4d31-bed4-42fc0aaaaf17\") " pod="openshift-ovn-kubernetes/ovnkube-node-86758"
Oct 31 12:26:37.644392 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.644030    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d43a4e16-3834-4d31-bed4-42fc0aaaaf17-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-86758\" (UID: \"d43a4e16-3834-4d31-bed4-42fc0aaaaf17\") " pod="openshift-ovn-kubernetes/ovnkube-node-86758"
Oct 31 12:26:37.644392 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.644061    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a1e578b0-2e77-4f52-8a44-379b6059c3d9-registration-dir\") pod \"aws-ebs-csi-driver-node-jmlvc\" (UID: \"a1e578b0-2e77-4f52-8a44-379b6059c3d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jmlvc"
Oct 31 12:26:37.644392 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.644088    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d43a4e16-3834-4d31-bed4-42fc0aaaaf17-run-ovn\") pod \"ovnkube-node-86758\" (UID: \"d43a4e16-3834-4d31-bed4-42fc0aaaaf17\") " pod="openshift-ovn-kubernetes/ovnkube-node-86758"
Oct 31 12:26:37.644392 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.644113    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6b3e77ff-4445-42fa-a7c3-f35d442c276d-host-run-k8s-cni-cncf-io\") pod \"multus-ccps4\" (UID: \"6b3e77ff-4445-42fa-a7c3-f35d442c276d\") " pod="openshift-multus/multus-ccps4"
Oct 31 12:26:37.644392 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.643564    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d43a4e16-3834-4d31-bed4-42fc0aaaaf17-host-run-netns\") pod \"ovnkube-node-86758\" (UID: \"d43a4e16-3834-4d31-bed4-42fc0aaaaf17\") " pod="openshift-ovn-kubernetes/ovnkube-node-86758"
Oct 31 12:26:37.644392 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.644202    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d43a4e16-3834-4d31-bed4-42fc0aaaaf17-ovnkube-script-lib\") pod \"ovnkube-node-86758\" (UID: \"d43a4e16-3834-4d31-bed4-42fc0aaaaf17\") " pod="openshift-ovn-kubernetes/ovnkube-node-86758"
Oct 31 12:26:37.644392 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.644232    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3bb2c04e-55a2-4206-9530-7298f4469379-host-slash\") pod \"iptables-alerter-plbnz\" (UID: \"3bb2c04e-55a2-4206-9530-7298f4469379\") " pod="openshift-network-operator/iptables-alerter-plbnz"
Oct 31 12:26:37.644392 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.644241    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee-sys\") pod \"tuned-qkt2l\" (UID: \"51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee\") " pod="openshift-cluster-node-tuning-operator/tuned-qkt2l"
Oct 31 12:26:37.644392 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.644288    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d43a4e16-3834-4d31-bed4-42fc0aaaaf17-etc-openvswitch\") pod \"ovnkube-node-86758\" (UID: \"d43a4e16-3834-4d31-bed4-42fc0aaaaf17\") " pod="openshift-ovn-kubernetes/ovnkube-node-86758"
Oct 31 12:26:37.644392 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.644296    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee-etc-systemd\") pod \"tuned-qkt2l\" (UID: \"51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee\") " pod="openshift-cluster-node-tuning-operator/tuned-qkt2l"
Oct 31 12:26:37.644392 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.644293    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6c661ce8-9f7b-4509-bd17-2380bcd1c425-serviceca\") pod \"node-ca-wdkzp\" (UID: \"6c661ce8-9f7b-4509-bd17-2380bcd1c425\") " pod="openshift-image-registry/node-ca-wdkzp"
Oct 31 12:26:37.644392 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.644355    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee-etc-kubernetes\") pod \"tuned-qkt2l\" (UID: \"51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee\") " pod="openshift-cluster-node-tuning-operator/tuned-qkt2l"
Oct 31 12:26:37.644392 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.644397    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d43a4e16-3834-4d31-bed4-42fc0aaaaf17-log-socket\") pod \"ovnkube-node-86758\" (UID: \"d43a4e16-3834-4d31-bed4-42fc0aaaaf17\") " pod="openshift-ovn-kubernetes/ovnkube-node-86758"
Oct 31 12:26:37.645050 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.644411    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d43a4e16-3834-4d31-bed4-42fc0aaaaf17-run-ovn\") pod \"ovnkube-node-86758\" (UID: \"d43a4e16-3834-4d31-bed4-42fc0aaaaf17\") " pod="openshift-ovn-kubernetes/ovnkube-node-86758"
Oct 31 12:26:37.645050 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.644426    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d43a4e16-3834-4d31-bed4-42fc0aaaaf17-ovnkube-config\") pod \"ovnkube-node-86758\" (UID: \"d43a4e16-3834-4d31-bed4-42fc0aaaaf17\") " pod="openshift-ovn-kubernetes/ovnkube-node-86758"
Oct 31 12:26:37.645050 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.644455    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3bb2c04e-55a2-4206-9530-7298f4469379-host-slash\") pod \"iptables-alerter-plbnz\" (UID: \"3bb2c04e-55a2-4206-9530-7298f4469379\") " pod="openshift-network-operator/iptables-alerter-plbnz"
Oct 31 12:26:37.645720 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.645689    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee-etc-systemd\") pod \"tuned-qkt2l\" (UID: \"51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee\") " pod="openshift-cluster-node-tuning-operator/tuned-qkt2l"
Oct 31 12:26:37.645805 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.645767    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d43a4e16-3834-4d31-bed4-42fc0aaaaf17-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-86758\" (UID: \"d43a4e16-3834-4d31-bed4-42fc0aaaaf17\") " pod="openshift-ovn-kubernetes/ovnkube-node-86758"
Oct 31 12:26:37.645867 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.645810    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d43a4e16-3834-4d31-bed4-42fc0aaaaf17-log-socket\") pod \"ovnkube-node-86758\" (UID: \"d43a4e16-3834-4d31-bed4-42fc0aaaaf17\") " pod="openshift-ovn-kubernetes/ovnkube-node-86758"
Oct 31 12:26:37.645867 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.644473    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a2b450be-ca79-4edf-a1ea-377531a49519-original-pull-secret\") pod \"global-pull-secret-syncer-2bmzr\" (UID: \"a2b450be-ca79-4edf-a1ea-377531a49519\") " pod="kube-system/global-pull-secret-syncer-2bmzr"
Oct 31 12:26:37.645974 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.645866    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee-tmp\") pod \"tuned-qkt2l\" (UID: \"51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee\") " pod="openshift-cluster-node-tuning-operator/tuned-qkt2l"
Oct 31 12:26:37.645974 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.645916    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d43a4e16-3834-4d31-bed4-42fc0aaaaf17-run-systemd\") pod \"ovnkube-node-86758\" (UID: \"d43a4e16-3834-4d31-bed4-42fc0aaaaf17\") " pod="openshift-ovn-kubernetes/ovnkube-node-86758"
Oct 31 12:26:37.645974 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.645950    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d43a4e16-3834-4d31-bed4-42fc0aaaaf17-run-openvswitch\") pod \"ovnkube-node-86758\" (UID: \"d43a4e16-3834-4d31-bed4-42fc0aaaaf17\") " pod="openshift-ovn-kubernetes/ovnkube-node-86758"
Oct 31 12:26:37.646165 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.645980    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d43a4e16-3834-4d31-bed4-42fc0aaaaf17-node-log\") pod \"ovnkube-node-86758\" (UID: \"d43a4e16-3834-4d31-bed4-42fc0aaaaf17\") " pod="openshift-ovn-kubernetes/ovnkube-node-86758"
Oct 31 12:26:37.646165 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.646017    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a1e578b0-2e77-4f52-8a44-379b6059c3d9-sys-fs\") pod \"aws-ebs-csi-driver-node-jmlvc\" (UID: \"a1e578b0-2e77-4f52-8a44-379b6059c3d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jmlvc"
Oct 31 12:26:37.646165 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.646050    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6b3e77ff-4445-42fa-a7c3-f35d442c276d-etc-kubernetes\") pod \"multus-ccps4\" (UID: \"6b3e77ff-4445-42fa-a7c3-f35d442c276d\") " pod="openshift-multus/multus-ccps4"
Oct 31 12:26:37.646165 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.646084    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee-etc-sysctl-d\") pod \"tuned-qkt2l\" (UID: \"51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee\") " pod="openshift-cluster-node-tuning-operator/tuned-qkt2l"
Oct 31 12:26:37.646165 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.646131    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d43a4e16-3834-4d31-bed4-42fc0aaaaf17-var-lib-openvswitch\") pod \"ovnkube-node-86758\" (UID: \"d43a4e16-3834-4d31-bed4-42fc0aaaaf17\") " pod="openshift-ovn-kubernetes/ovnkube-node-86758"
Oct 31 12:26:37.646404 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.646181    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d43a4e16-3834-4d31-bed4-42fc0aaaaf17-host-cni-netd\") pod \"ovnkube-node-86758\" (UID: \"d43a4e16-3834-4d31-bed4-42fc0aaaaf17\") " pod="openshift-ovn-kubernetes/ovnkube-node-86758"
Oct 31 12:26:37.646404 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.646219    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tf6n\" (UniqueName: \"kubernetes.io/projected/7a402472-ca33-44ae-b598-96fc8e7a5eb2-kube-api-access-2tf6n\") pod \"network-check-target-tdrmt\" (UID: \"7a402472-ca33-44ae-b598-96fc8e7a5eb2\") " pod="openshift-network-diagnostics/network-check-target-tdrmt"
Oct 31 12:26:37.646404 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.646257    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a1e578b0-2e77-4f52-8a44-379b6059c3d9-socket-dir\") pod \"aws-ebs-csi-driver-node-jmlvc\" (UID: \"a1e578b0-2e77-4f52-8a44-379b6059c3d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jmlvc"
Oct 31 12:26:37.646404 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.646294    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nx5hk\" (UniqueName: \"kubernetes.io/projected/6c661ce8-9f7b-4509-bd17-2380bcd1c425-kube-api-access-nx5hk\") pod \"node-ca-wdkzp\" (UID: \"6c661ce8-9f7b-4509-bd17-2380bcd1c425\") " pod="openshift-image-registry/node-ca-wdkzp"
Oct 31 12:26:37.646404 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.646325    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3bb2c04e-55a2-4206-9530-7298f4469379-iptables-alerter-script\") pod \"iptables-alerter-plbnz\" (UID: \"3bb2c04e-55a2-4206-9530-7298f4469379\") " pod="openshift-network-operator/iptables-alerter-plbnz"
Oct 31 12:26:37.646404 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.646356    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee-etc-sysctl-conf\") pod \"tuned-qkt2l\" (UID: \"51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee\") " pod="openshift-cluster-node-tuning-operator/tuned-qkt2l"
Oct 31 12:26:37.646404 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.646389    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a1c3175b-e32e-420e-877f-71bfe64d9c6c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-44zj2\" (UID: \"a1c3175b-e32e-420e-877f-71bfe64d9c6c\") " pod="openshift-multus/multus-additional-cni-plugins-44zj2"
Oct 31 12:26:37.646759 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.646421    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dsg9j\" (UniqueName: \"kubernetes.io/projected/a1e578b0-2e77-4f52-8a44-379b6059c3d9-kube-api-access-dsg9j\") pod \"aws-ebs-csi-driver-node-jmlvc\" (UID: \"a1e578b0-2e77-4f52-8a44-379b6059c3d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jmlvc"
Oct 31 12:26:37.646759 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.646451    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6b3e77ff-4445-42fa-a7c3-f35d442c276d-cnibin\") pod \"multus-ccps4\" (UID: \"6b3e77ff-4445-42fa-a7c3-f35d442c276d\") " pod="openshift-multus/multus-ccps4"
Oct 31 12:26:37.646759 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.646481    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee-host\") pod \"tuned-qkt2l\" (UID: \"51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee\") " pod="openshift-cluster-node-tuning-operator/tuned-qkt2l"
Oct 31 12:26:37.646759 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.646514    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a1c3175b-e32e-420e-877f-71bfe64d9c6c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-44zj2\" (UID: \"a1c3175b-e32e-420e-877f-71bfe64d9c6c\") " pod="openshift-multus/multus-additional-cni-plugins-44zj2"
Oct 31 12:26:37.646759 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.646579    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a1c3175b-e32e-420e-877f-71bfe64d9c6c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-44zj2\" (UID: \"a1c3175b-e32e-420e-877f-71bfe64d9c6c\") " pod="openshift-multus/multus-additional-cni-plugins-44zj2"
Oct 31 12:26:37.646759 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.646614    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6b3e77ff-4445-42fa-a7c3-f35d442c276d-multus-socket-dir-parent\") pod \"multus-ccps4\" (UID: \"6b3e77ff-4445-42fa-a7c3-f35d442c276d\") " pod="openshift-multus/multus-ccps4"
Oct 31 12:26:37.646759 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.646647    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6b3e77ff-4445-42fa-a7c3-f35d442c276d-multus-conf-dir\") pod \"multus-ccps4\" (UID: \"6b3e77ff-4445-42fa-a7c3-f35d442c276d\") " pod="openshift-multus/multus-ccps4"
Oct 31 12:26:37.646759 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.646679    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b5014b57-9b83-4fc9-b0ed-55793b2b3a98-konnectivity-ca\") pod \"konnectivity-agent-8jlvq\" (UID: \"b5014b57-9b83-4fc9-b0ed-55793b2b3a98\") " pod="kube-system/konnectivity-agent-8jlvq"
Oct 31 12:26:37.646759 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.646723    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d43a4e16-3834-4d31-bed4-42fc0aaaaf17-systemd-units\") pod \"ovnkube-node-86758\" (UID: \"d43a4e16-3834-4d31-bed4-42fc0aaaaf17\") " pod="openshift-ovn-kubernetes/ovnkube-node-86758"
Oct 31 12:26:37.647214 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.646788    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee-etc-sysconfig\") pod \"tuned-qkt2l\" (UID: \"51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee\") " pod="openshift-cluster-node-tuning-operator/tuned-qkt2l"
Oct 31 12:26:37.647214 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.646843    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a1e578b0-2e77-4f52-8a44-379b6059c3d9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jmlvc\" (UID: \"a1e578b0-2e77-4f52-8a44-379b6059c3d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jmlvc"
Oct 31 12:26:37.647214 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.646877    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a1e578b0-2e77-4f52-8a44-379b6059c3d9-etc-selinux\") pod \"aws-ebs-csi-driver-node-jmlvc\" (UID: \"a1e578b0-2e77-4f52-8a44-379b6059c3d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jmlvc"
Oct 31 12:26:37.647214 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.646915    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6b3e77ff-4445-42fa-a7c3-f35d442c276d-multus-cni-dir\") pod \"multus-ccps4\" (UID: \"6b3e77ff-4445-42fa-a7c3-f35d442c276d\") " pod="openshift-multus/multus-ccps4"
Oct 31 12:26:37.647214 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.646985    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6b3e77ff-4445-42fa-a7c3-f35d442c276d-host-var-lib-cni-bin\") pod \"multus-ccps4\" (UID: \"6b3e77ff-4445-42fa-a7c3-f35d442c276d\") " pod="openshift-multus/multus-ccps4"
Oct 31 12:26:37.647788 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.647764    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d43a4e16-3834-4d31-bed4-42fc0aaaaf17-run-systemd\") pod \"ovnkube-node-86758\" (UID: \"d43a4e16-3834-4d31-bed4-42fc0aaaaf17\") " pod="openshift-ovn-kubernetes/ovnkube-node-86758"
Oct 31 12:26:37.647913 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.647832    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d43a4e16-3834-4d31-bed4-42fc0aaaaf17-run-openvswitch\") pod \"ovnkube-node-86758\" (UID: \"d43a4e16-3834-4d31-bed4-42fc0aaaaf17\") " pod="openshift-ovn-kubernetes/ovnkube-node-86758"
Oct 31 12:26:37.647913 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.647889    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d43a4e16-3834-4d31-bed4-42fc0aaaaf17-node-log\") pod \"ovnkube-node-86758\" (UID: \"d43a4e16-3834-4d31-bed4-42fc0aaaaf17\") " pod="openshift-ovn-kubernetes/ovnkube-node-86758"
Oct 31 12:26:37.648116 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.647957    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee-host\") pod \"tuned-qkt2l\" (UID: \"51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee\") " pod="openshift-cluster-node-tuning-operator/tuned-qkt2l"
Oct 31 12:26:37.648116 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.648025    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee-etc-sysctl-d\") pod \"tuned-qkt2l\" (UID: \"51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee\") " pod="openshift-cluster-node-tuning-operator/tuned-qkt2l"
Oct 31 12:26:37.648116 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.648087    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d43a4e16-3834-4d31-bed4-42fc0aaaaf17-var-lib-openvswitch\") pod \"ovnkube-node-86758\" (UID: \"d43a4e16-3834-4d31-bed4-42fc0aaaaf17\") " pod="openshift-ovn-kubernetes/ovnkube-node-86758"
Oct 31 12:26:37.648116 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.648111    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a1c3175b-e32e-420e-877f-71bfe64d9c6c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-44zj2\" (UID: \"a1c3175b-e32e-420e-877f-71bfe64d9c6c\") " pod="openshift-multus/multus-additional-cni-plugins-44zj2"
Oct 31 12:26:37.648441 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.648142    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d43a4e16-3834-4d31-bed4-42fc0aaaaf17-host-cni-netd\") pod \"ovnkube-node-86758\" (UID: \"d43a4e16-3834-4d31-bed4-42fc0aaaaf17\") " pod="openshift-ovn-kubernetes/ovnkube-node-86758"
Oct 31 12:26:37.648441 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.648152    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d43a4e16-3834-4d31-bed4-42fc0aaaaf17-ovn-node-metrics-cert\") pod \"ovnkube-node-86758\" (UID: \"d43a4e16-3834-4d31-bed4-42fc0aaaaf17\") " pod="openshift-ovn-kubernetes/ovnkube-node-86758"
Oct 31 12:26:37.648688 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.648654    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d43a4e16-3834-4d31-bed4-42fc0aaaaf17-systemd-units\") pod \"ovnkube-node-86758\" (UID: \"d43a4e16-3834-4d31-bed4-42fc0aaaaf17\") " pod="openshift-ovn-kubernetes/ovnkube-node-86758"
Oct 31 12:26:37.648781 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.648762    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee-etc-sysconfig\") pod \"tuned-qkt2l\" (UID: \"51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee\") " pod="openshift-cluster-node-tuning-operator/tuned-qkt2l"
Oct 31 12:26:37.648994 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.648974    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d43a4e16-3834-4d31-bed4-42fc0aaaaf17-ovnkube-script-lib\") pod \"ovnkube-node-86758\" (UID: \"d43a4e16-3834-4d31-bed4-42fc0aaaaf17\") " pod="openshift-ovn-kubernetes/ovnkube-node-86758"
Oct 31 12:26:37.649089 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.649020    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee-etc-sysctl-conf\") pod \"tuned-qkt2l\" (UID: \"51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee\") " pod="openshift-cluster-node-tuning-operator/tuned-qkt2l"
Oct 31 12:26:37.649442 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.649418    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3bb2c04e-55a2-4206-9530-7298f4469379-iptables-alerter-script\") pod \"iptables-alerter-plbnz\" (UID: \"3bb2c04e-55a2-4206-9530-7298f4469379\") " pod="openshift-network-operator/iptables-alerter-plbnz"
Oct 31 12:26:37.649553 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.649477    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a1c3175b-e32e-420e-877f-71bfe64d9c6c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-44zj2\" (UID: \"a1c3175b-e32e-420e-877f-71bfe64d9c6c\") " pod="openshift-multus/multus-additional-cni-plugins-44zj2"
Oct 31 12:26:37.650842 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.649757    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee-etc-tuned\") pod \"tuned-qkt2l\" (UID: \"51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee\") " pod="openshift-cluster-node-tuning-operator/tuned-qkt2l"
Oct 31 12:26:37.650842 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.649893    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee-tmp\") pod \"tuned-qkt2l\" (UID: \"51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee\") " pod="openshift-cluster-node-tuning-operator/tuned-qkt2l"
Oct 31 12:26:37.650842 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.649852    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d43a4e16-3834-4d31-bed4-42fc0aaaaf17-ovnkube-config\") pod \"ovnkube-node-86758\" (UID: \"d43a4e16-3834-4d31-bed4-42fc0aaaaf17\") " pod="openshift-ovn-kubernetes/ovnkube-node-86758"
Oct 31 12:26:37.651326 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.651288    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a1c3175b-e32e-420e-877f-71bfe64d9c6c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-44zj2\" (UID: \"a1c3175b-e32e-420e-877f-71bfe64d9c6c\") " pod="openshift-multus/multus-additional-cni-plugins-44zj2"
Oct 31 12:26:37.656004 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.655735    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwj8z\" (UniqueName: \"kubernetes.io/projected/a1c3175b-e32e-420e-877f-71bfe64d9c6c-kube-api-access-nwj8z\") pod \"multus-additional-cni-plugins-44zj2\" (UID: \"a1c3175b-e32e-420e-877f-71bfe64d9c6c\") " pod="openshift-multus/multus-additional-cni-plugins-44zj2"
Oct 31 12:26:37.656814 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.656767    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7j2rp\" (UniqueName: \"kubernetes.io/projected/6bbe13aa-df4b-40b0-8595-68fe1166a082-kube-api-access-7j2rp\") pod \"node-resolver-v9ks4\" (UID: \"6bbe13aa-df4b-40b0-8595-68fe1166a082\") " pod="openshift-dns/node-resolver-v9ks4"
Oct 31 12:26:37.657304 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.657276    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5nx4\" (UniqueName: \"kubernetes.io/projected/51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee-kube-api-access-t5nx4\") pod \"tuned-qkt2l\" (UID: \"51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee\") " pod="openshift-cluster-node-tuning-operator/tuned-qkt2l"
Oct 31 12:26:37.657595 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.657557    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzcgz\" (UniqueName: \"kubernetes.io/projected/d43a4e16-3834-4d31-bed4-42fc0aaaaf17-kube-api-access-hzcgz\") pod \"ovnkube-node-86758\" (UID: \"d43a4e16-3834-4d31-bed4-42fc0aaaaf17\") " pod="openshift-ovn-kubernetes/ovnkube-node-86758"
Oct 31 12:26:37.658500 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.658476    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xv9qt\" (UniqueName: \"kubernetes.io/projected/3bb2c04e-55a2-4206-9530-7298f4469379-kube-api-access-xv9qt\") pod \"iptables-alerter-plbnz\" (UID: \"3bb2c04e-55a2-4206-9530-7298f4469379\") " pod="openshift-network-operator/iptables-alerter-plbnz"
Oct 31 12:26:37.660582 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:26:37.660272    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 12:26:37.660582 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:26:37.660295    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 12:26:37.660582 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:26:37.660310    2576 projected.go:196] Error preparing data for projected volume kube-api-access-2tf6n for pod openshift-network-diagnostics/network-check-target-tdrmt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 12:26:37.660582 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:26:37.660372    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7a402472-ca33-44ae-b598-96fc8e7a5eb2-kube-api-access-2tf6n podName:7a402472-ca33-44ae-b598-96fc8e7a5eb2 nodeName:}" failed. No retries permitted until 2026-10-31 12:26:38.160351037 +0000 UTC m=+3.052998170 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-2tf6n" (UniqueName: "kubernetes.io/projected/7a402472-ca33-44ae-b598-96fc8e7a5eb2-kube-api-access-2tf6n") pod "network-check-target-tdrmt" (UID: "7a402472-ca33-44ae-b598-96fc8e7a5eb2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 12:26:37.662188 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.662120    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-190.us-east-2.compute.internal" event={"ID":"7fb532212935ad861267bbe741c1366b","Type":"ContainerStarted","Data":"b539c38438f2cfd785f2a2a6e023750667ea93e764840b11ad42f359c3687b08"}
Oct 31 12:26:37.663810 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.663782    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-190.us-east-2.compute.internal" event={"ID":"928a0065cb02eec7c5308f7ecf1e7e6c","Type":"ContainerStarted","Data":"b61159ca029793099370ff0779dda563c7d5600e24e1415f4ca6047a637d4ed7"}
Oct 31 12:26:37.665705 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.665681    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nx5hk\" (UniqueName: \"kubernetes.io/projected/6c661ce8-9f7b-4509-bd17-2380bcd1c425-kube-api-access-nx5hk\") pod \"node-ca-wdkzp\" (UID: \"6c661ce8-9f7b-4509-bd17-2380bcd1c425\") " pod="openshift-image-registry/node-ca-wdkzp"
Oct 31 12:26:37.748701 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.747688    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6b3e77ff-4445-42fa-a7c3-f35d442c276d-cnibin\") pod \"multus-ccps4\" (UID: \"6b3e77ff-4445-42fa-a7c3-f35d442c276d\") " pod="openshift-multus/multus-ccps4"
Oct 31 12:26:37.748701 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.747732    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6b3e77ff-4445-42fa-a7c3-f35d442c276d-multus-socket-dir-parent\") pod \"multus-ccps4\" (UID: \"6b3e77ff-4445-42fa-a7c3-f35d442c276d\") " pod="openshift-multus/multus-ccps4"
Oct 31 12:26:37.748701 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.747758    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6b3e77ff-4445-42fa-a7c3-f35d442c276d-multus-conf-dir\") pod \"multus-ccps4\" (UID: \"6b3e77ff-4445-42fa-a7c3-f35d442c276d\") " pod="openshift-multus/multus-ccps4"
Oct 31 12:26:37.748701 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.747776    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b5014b57-9b83-4fc9-b0ed-55793b2b3a98-konnectivity-ca\") pod \"konnectivity-agent-8jlvq\" (UID: \"b5014b57-9b83-4fc9-b0ed-55793b2b3a98\") " pod="kube-system/konnectivity-agent-8jlvq"
Oct 31 12:26:37.748701 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.747794    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a1e578b0-2e77-4f52-8a44-379b6059c3d9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jmlvc\" (UID: \"a1e578b0-2e77-4f52-8a44-379b6059c3d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jmlvc"
Oct 31 12:26:37.748701 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.747809    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a1e578b0-2e77-4f52-8a44-379b6059c3d9-etc-selinux\") pod \"aws-ebs-csi-driver-node-jmlvc\" (UID: \"a1e578b0-2e77-4f52-8a44-379b6059c3d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jmlvc"
Oct 31 12:26:37.748701 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.747830    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6b3e77ff-4445-42fa-a7c3-f35d442c276d-multus-cni-dir\") pod \"multus-ccps4\" (UID: \"6b3e77ff-4445-42fa-a7c3-f35d442c276d\") " pod="openshift-multus/multus-ccps4"
Oct 31 12:26:37.748701 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.747852    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6b3e77ff-4445-42fa-a7c3-f35d442c276d-host-var-lib-cni-bin\") pod \"multus-ccps4\" (UID: \"6b3e77ff-4445-42fa-a7c3-f35d442c276d\") " pod="openshift-multus/multus-ccps4"
Oct 31 12:26:37.748701 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.747876    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a2b450be-ca79-4edf-a1ea-377531a49519-dbus\") pod \"global-pull-secret-syncer-2bmzr\" (UID: \"a2b450be-ca79-4edf-a1ea-377531a49519\") " pod="kube-system/global-pull-secret-syncer-2bmzr"
Oct 31 12:26:37.748701 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.747902    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6b3e77ff-4445-42fa-a7c3-f35d442c276d-host-var-lib-kubelet\") pod \"multus-ccps4\" (UID: \"6b3e77ff-4445-42fa-a7c3-f35d442c276d\") " pod="openshift-multus/multus-ccps4"
Oct 31 12:26:37.748701 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.747929    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6b3e77ff-4445-42fa-a7c3-f35d442c276d-host-run-multus-certs\") pod \"multus-ccps4\" (UID: \"6b3e77ff-4445-42fa-a7c3-f35d442c276d\") " pod="openshift-multus/multus-ccps4"
Oct 31 12:26:37.748701 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.747957    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e64bd69d-946f-42ad-94ae-b87b7de3a581-metrics-certs\") pod \"network-metrics-daemon-94kh9\" (UID: \"e64bd69d-946f-42ad-94ae-b87b7de3a581\") " pod="openshift-multus/network-metrics-daemon-94kh9"
Oct 31 12:26:37.748701 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.747985    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqk48\" (UniqueName: \"kubernetes.io/projected/e64bd69d-946f-42ad-94ae-b87b7de3a581-kube-api-access-gqk48\") pod \"network-metrics-daemon-94kh9\" (UID: \"e64bd69d-946f-42ad-94ae-b87b7de3a581\") " pod="openshift-multus/network-metrics-daemon-94kh9"
Oct 31 12:26:37.748701 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.748017    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a1e578b0-2e77-4f52-8a44-379b6059c3d9-device-dir\") pod \"aws-ebs-csi-driver-node-jmlvc\" (UID: \"a1e578b0-2e77-4f52-8a44-379b6059c3d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jmlvc"
Oct 31 12:26:37.748701 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.748042    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6b3e77ff-4445-42fa-a7c3-f35d442c276d-host-run-netns\") pod \"multus-ccps4\" (UID: \"6b3e77ff-4445-42fa-a7c3-f35d442c276d\") " pod="openshift-multus/multus-ccps4"
Oct 31 12:26:37.748701 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.748065    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b5014b57-9b83-4fc9-b0ed-55793b2b3a98-agent-certs\") pod \"konnectivity-agent-8jlvq\" (UID: \"b5014b57-9b83-4fc9-b0ed-55793b2b3a98\") " pod="kube-system/konnectivity-agent-8jlvq"
Oct 31 12:26:37.748701 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.748090    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a1e578b0-2e77-4f52-8a44-379b6059c3d9-etc-selinux\") pod \"aws-ebs-csi-driver-node-jmlvc\" (UID: \"a1e578b0-2e77-4f52-8a44-379b6059c3d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jmlvc"
Oct 31 12:26:37.749670 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.748100    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6b3e77ff-4445-42fa-a7c3-f35d442c276d-system-cni-dir\") pod \"multus-ccps4\" (UID: \"6b3e77ff-4445-42fa-a7c3-f35d442c276d\") " pod="openshift-multus/multus-ccps4"
Oct 31 12:26:37.749670 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.748145    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6b3e77ff-4445-42fa-a7c3-f35d442c276d-cni-binary-copy\") pod \"multus-ccps4\" (UID: \"6b3e77ff-4445-42fa-a7c3-f35d442c276d\") " pod="openshift-multus/multus-ccps4"
Oct 31 12:26:37.749670 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.748173    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6b3e77ff-4445-42fa-a7c3-f35d442c276d-host-var-lib-cni-multus\") pod \"multus-ccps4\" (UID: \"6b3e77ff-4445-42fa-a7c3-f35d442c276d\") " pod="openshift-multus/multus-ccps4"
Oct 31 12:26:37.749670 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.748169    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6b3e77ff-4445-42fa-a7c3-f35d442c276d-system-cni-dir\") pod \"multus-ccps4\" (UID: \"6b3e77ff-4445-42fa-a7c3-f35d442c276d\") " pod="openshift-multus/multus-ccps4"
Oct 31 12:26:37.749670 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.748207    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6b3e77ff-4445-42fa-a7c3-f35d442c276d-multus-daemon-config\") pod \"multus-ccps4\" (UID: \"6b3e77ff-4445-42fa-a7c3-f35d442c276d\") " pod="openshift-multus/multus-ccps4"
Oct 31 12:26:37.749670 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.748233    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vknw2\" (UniqueName: \"kubernetes.io/projected/6b3e77ff-4445-42fa-a7c3-f35d442c276d-kube-api-access-vknw2\") pod \"multus-ccps4\" (UID: \"6b3e77ff-4445-42fa-a7c3-f35d442c276d\") " pod="openshift-multus/multus-ccps4"
Oct 31 12:26:37.749670 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.748241    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6b3e77ff-4445-42fa-a7c3-f35d442c276d-host-run-multus-certs\") pod \"multus-ccps4\" (UID: \"6b3e77ff-4445-42fa-a7c3-f35d442c276d\") " pod="openshift-multus/multus-ccps4"
Oct 31 12:26:37.749670 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.748267    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6b3e77ff-4445-42fa-a7c3-f35d442c276d-hostroot\") pod \"multus-ccps4\" (UID: \"6b3e77ff-4445-42fa-a7c3-f35d442c276d\") " pod="openshift-multus/multus-ccps4"
Oct 31 12:26:37.749670 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.748298    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a2b450be-ca79-4edf-a1ea-377531a49519-kubelet-config\") pod \"global-pull-secret-syncer-2bmzr\" (UID: \"a2b450be-ca79-4edf-a1ea-377531a49519\") " pod="kube-system/global-pull-secret-syncer-2bmzr"
Oct 31 12:26:37.749670 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.748321    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6b3e77ff-4445-42fa-a7c3-f35d442c276d-multus-cni-dir\") pod \"multus-ccps4\" (UID: \"6b3e77ff-4445-42fa-a7c3-f35d442c276d\") " pod="openshift-multus/multus-ccps4"
Oct 31 12:26:37.749670 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:26:37.748336    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 12:26:37.749670 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.748389    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a1e578b0-2e77-4f52-8a44-379b6059c3d9-registration-dir\") pod \"aws-ebs-csi-driver-node-jmlvc\" (UID: \"a1e578b0-2e77-4f52-8a44-379b6059c3d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jmlvc"
Oct 31 12:26:37.749670 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:26:37.748398    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e64bd69d-946f-42ad-94ae-b87b7de3a581-metrics-certs podName:e64bd69d-946f-42ad-94ae-b87b7de3a581 nodeName:}" failed. No retries permitted until 2026-10-31 12:26:38.248378507 +0000 UTC m=+3.141025665 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e64bd69d-946f-42ad-94ae-b87b7de3a581-metrics-certs") pod "network-metrics-daemon-94kh9" (UID: "e64bd69d-946f-42ad-94ae-b87b7de3a581") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 12:26:37.749670 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.748443    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6b3e77ff-4445-42fa-a7c3-f35d442c276d-cnibin\") pod \"multus-ccps4\" (UID: \"6b3e77ff-4445-42fa-a7c3-f35d442c276d\") " pod="openshift-multus/multus-ccps4"
Oct 31 12:26:37.749670 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.748501    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6b3e77ff-4445-42fa-a7c3-f35d442c276d-multus-socket-dir-parent\") pod \"multus-ccps4\" (UID: \"6b3e77ff-4445-42fa-a7c3-f35d442c276d\") " pod="openshift-multus/multus-ccps4"
Oct 31 12:26:37.749670 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.748511    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a1e578b0-2e77-4f52-8a44-379b6059c3d9-device-dir\") pod \"aws-ebs-csi-driver-node-jmlvc\" (UID: \"a1e578b0-2e77-4f52-8a44-379b6059c3d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jmlvc"
Oct 31 12:26:37.749670 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.748561    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6b3e77ff-4445-42fa-a7c3-f35d442c276d-multus-conf-dir\") pod \"multus-ccps4\" (UID: \"6b3e77ff-4445-42fa-a7c3-f35d442c276d\") " pod="openshift-multus/multus-ccps4"
Oct 31 12:26:37.750522 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.748330    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a1e578b0-2e77-4f52-8a44-379b6059c3d9-registration-dir\") pod \"aws-ebs-csi-driver-node-jmlvc\" (UID: \"a1e578b0-2e77-4f52-8a44-379b6059c3d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jmlvc"
Oct 31 12:26:37.750522 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.748599    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6b3e77ff-4445-42fa-a7c3-f35d442c276d-host-run-netns\") pod \"multus-ccps4\" (UID: \"6b3e77ff-4445-42fa-a7c3-f35d442c276d\") " pod="openshift-multus/multus-ccps4"
Oct 31 12:26:37.750522 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.748620    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6b3e77ff-4445-42fa-a7c3-f35d442c276d-host-var-lib-cni-multus\") pod \"multus-ccps4\" (UID: \"6b3e77ff-4445-42fa-a7c3-f35d442c276d\") " pod="openshift-multus/multus-ccps4"
Oct 31 12:26:37.750522 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.748628    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6b3e77ff-4445-42fa-a7c3-f35d442c276d-host-run-k8s-cni-cncf-io\") pod \"multus-ccps4\" (UID: \"6b3e77ff-4445-42fa-a7c3-f35d442c276d\") " pod="openshift-multus/multus-ccps4"
Oct 31 12:26:37.750522 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.748640    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a1e578b0-2e77-4f52-8a44-379b6059c3d9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jmlvc\" (UID: \"a1e578b0-2e77-4f52-8a44-379b6059c3d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jmlvc"
Oct 31 12:26:37.750522 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.748640    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6b3e77ff-4445-42fa-a7c3-f35d442c276d-host-var-lib-kubelet\") pod \"multus-ccps4\" (UID: \"6b3e77ff-4445-42fa-a7c3-f35d442c276d\") " pod="openshift-multus/multus-ccps4"
Oct 31 12:26:37.750522 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.748652    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6b3e77ff-4445-42fa-a7c3-f35d442c276d-hostroot\") pod \"multus-ccps4\" (UID: \"6b3e77ff-4445-42fa-a7c3-f35d442c276d\") " pod="openshift-multus/multus-ccps4"
Oct 31 12:26:37.750522 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.748666    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6b3e77ff-4445-42fa-a7c3-f35d442c276d-host-run-k8s-cni-cncf-io\") pod \"multus-ccps4\" (UID: \"6b3e77ff-4445-42fa-a7c3-f35d442c276d\") " pod="openshift-multus/multus-ccps4"
Oct 31 12:26:37.750522 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.748679    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a2b450be-ca79-4edf-a1ea-377531a49519-original-pull-secret\") pod \"global-pull-secret-syncer-2bmzr\" (UID: \"a2b450be-ca79-4edf-a1ea-377531a49519\") " pod="kube-system/global-pull-secret-syncer-2bmzr"
Oct 31 12:26:37.750522 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.748688    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6b3e77ff-4445-42fa-a7c3-f35d442c276d-host-var-lib-cni-bin\") pod \"multus-ccps4\" (UID: \"6b3e77ff-4445-42fa-a7c3-f35d442c276d\") " pod="openshift-multus/multus-ccps4"
Oct 31 12:26:37.750522 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.748704    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a1e578b0-2e77-4f52-8a44-379b6059c3d9-sys-fs\") pod \"aws-ebs-csi-driver-node-jmlvc\" (UID: \"a1e578b0-2e77-4f52-8a44-379b6059c3d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jmlvc"
Oct 31 12:26:37.750522 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.748720    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6b3e77ff-4445-42fa-a7c3-f35d442c276d-etc-kubernetes\") pod \"multus-ccps4\" (UID: \"6b3e77ff-4445-42fa-a7c3-f35d442c276d\") " pod="openshift-multus/multus-ccps4"
Oct 31 12:26:37.750522 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.748718    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a2b450be-ca79-4edf-a1ea-377531a49519-dbus\") pod \"global-pull-secret-syncer-2bmzr\" (UID: \"a2b450be-ca79-4edf-a1ea-377531a49519\") " pod="kube-system/global-pull-secret-syncer-2bmzr"
Oct 31 12:26:37.750522 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.748756    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6b3e77ff-4445-42fa-a7c3-f35d442c276d-etc-kubernetes\") pod \"multus-ccps4\" (UID: \"6b3e77ff-4445-42fa-a7c3-f35d442c276d\") " pod="openshift-multus/multus-ccps4"
Oct 31 12:26:37.750522 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.748761    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a1e578b0-2e77-4f52-8a44-379b6059c3d9-socket-dir\") pod \"aws-ebs-csi-driver-node-jmlvc\" (UID: \"a1e578b0-2e77-4f52-8a44-379b6059c3d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jmlvc"
Oct 31 12:26:37.750522 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.748798    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsg9j\" (UniqueName: \"kubernetes.io/projected/a1e578b0-2e77-4f52-8a44-379b6059c3d9-kube-api-access-dsg9j\") pod \"aws-ebs-csi-driver-node-jmlvc\" (UID: \"a1e578b0-2e77-4f52-8a44-379b6059c3d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jmlvc"
Oct 31 12:26:37.750522 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:26:37.748809    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 12:26:37.751382 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:26:37.748857    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a2b450be-ca79-4edf-a1ea-377531a49519-original-pull-secret podName:a2b450be-ca79-4edf-a1ea-377531a49519 nodeName:}" failed. No retries permitted until 2026-10-31 12:26:38.248838128 +0000 UTC m=+3.141485259 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a2b450be-ca79-4edf-a1ea-377531a49519-original-pull-secret") pod "global-pull-secret-syncer-2bmzr" (UID: "a2b450be-ca79-4edf-a1ea-377531a49519") : object "kube-system"/"original-pull-secret" not registered
Oct 31 12:26:37.751382 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.748869    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a1e578b0-2e77-4f52-8a44-379b6059c3d9-sys-fs\") pod \"aws-ebs-csi-driver-node-jmlvc\" (UID: \"a1e578b0-2e77-4f52-8a44-379b6059c3d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jmlvc"
Oct 31 12:26:37.751382 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.748992    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a1e578b0-2e77-4f52-8a44-379b6059c3d9-socket-dir\") pod \"aws-ebs-csi-driver-node-jmlvc\" (UID: \"a1e578b0-2e77-4f52-8a44-379b6059c3d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jmlvc"
Oct 31 12:26:37.751382 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.749047    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a2b450be-ca79-4edf-a1ea-377531a49519-kubelet-config\") pod \"global-pull-secret-syncer-2bmzr\" (UID: \"a2b450be-ca79-4edf-a1ea-377531a49519\") " pod="kube-system/global-pull-secret-syncer-2bmzr"
Oct 31 12:26:37.751382 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.750008    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6b3e77ff-4445-42fa-a7c3-f35d442c276d-multus-daemon-config\") pod \"multus-ccps4\" (UID: \"6b3e77ff-4445-42fa-a7c3-f35d442c276d\") " pod="openshift-multus/multus-ccps4"
Oct 31 12:26:37.751382 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.750032    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b5014b57-9b83-4fc9-b0ed-55793b2b3a98-konnectivity-ca\") pod \"konnectivity-agent-8jlvq\" (UID: \"b5014b57-9b83-4fc9-b0ed-55793b2b3a98\") " pod="kube-system/konnectivity-agent-8jlvq"
Oct 31 12:26:37.751382 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.750227    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6b3e77ff-4445-42fa-a7c3-f35d442c276d-cni-binary-copy\") pod \"multus-ccps4\" (UID: \"6b3e77ff-4445-42fa-a7c3-f35d442c276d\") " pod="openshift-multus/multus-ccps4"
Oct 31 12:26:37.755083 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.752301    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b5014b57-9b83-4fc9-b0ed-55793b2b3a98-agent-certs\") pod \"konnectivity-agent-8jlvq\" (UID: \"b5014b57-9b83-4fc9-b0ed-55793b2b3a98\") " pod="kube-system/konnectivity-agent-8jlvq"
Oct 31 12:26:37.774765 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.766771    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vknw2\" (UniqueName: \"kubernetes.io/projected/6b3e77ff-4445-42fa-a7c3-f35d442c276d-kube-api-access-vknw2\") pod \"multus-ccps4\" (UID: \"6b3e77ff-4445-42fa-a7c3-f35d442c276d\") " pod="openshift-multus/multus-ccps4"
Oct 31 12:26:37.774765 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.770741    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsg9j\" (UniqueName: \"kubernetes.io/projected/a1e578b0-2e77-4f52-8a44-379b6059c3d9-kube-api-access-dsg9j\") pod \"aws-ebs-csi-driver-node-jmlvc\" (UID: \"a1e578b0-2e77-4f52-8a44-379b6059c3d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jmlvc"
Oct 31 12:26:37.776080 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.776053    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqk48\" (UniqueName: \"kubernetes.io/projected/e64bd69d-946f-42ad-94ae-b87b7de3a581-kube-api-access-gqk48\") pod \"network-metrics-daemon-94kh9\" (UID: \"e64bd69d-946f-42ad-94ae-b87b7de3a581\") " pod="openshift-multus/network-metrics-daemon-94kh9"
Oct 31 12:26:37.828737 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.828699    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-plbnz"
Oct 31 12:26:37.839252 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.839221    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-qkt2l"
Oct 31 12:26:37.842006 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:37.841975    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bb2c04e_55a2_4206_9530_7298f4469379.slice/crio-8a79b2069dde730f773d0ac904f6e23f6db008b97e8fd24680a8e7fa053f2e9f WatchSource:0}: Error finding container 8a79b2069dde730f773d0ac904f6e23f6db008b97e8fd24680a8e7fa053f2e9f: Status 404 returned error can't find the container with id 8a79b2069dde730f773d0ac904f6e23f6db008b97e8fd24680a8e7fa053f2e9f
Oct 31 12:26:37.849977 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:37.849952    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51a5b8c5_e56a_4c0c_b270_b6a0ec0db9ee.slice/crio-1bf8d18acbe5580d4e21bb564614ffbafbeafb5e804a96c1fee8a4417b7bc588 WatchSource:0}: Error finding container 1bf8d18acbe5580d4e21bb564614ffbafbeafb5e804a96c1fee8a4417b7bc588: Status 404 returned error can't find the container with id 1bf8d18acbe5580d4e21bb564614ffbafbeafb5e804a96c1fee8a4417b7bc588
Oct 31 12:26:37.862699 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.862563    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-wdkzp"
Oct 31 12:26:37.867905 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.867887    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-86758"
Oct 31 12:26:37.876144 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:37.876116    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c661ce8_9f7b_4509_bd17_2380bcd1c425.slice/crio-1712b1a89fc033bb7af133fed6fccaed65abbd63124e20f0e3ea940d30aa7113 WatchSource:0}: Error finding container 1712b1a89fc033bb7af133fed6fccaed65abbd63124e20f0e3ea940d30aa7113: Status 404 returned error can't find the container with id 1712b1a89fc033bb7af133fed6fccaed65abbd63124e20f0e3ea940d30aa7113
Oct 31 12:26:37.876235 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.876160    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-v9ks4"
Oct 31 12:26:37.880926 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:37.880900    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd43a4e16_3834_4d31_bed4_42fc0aaaaf17.slice/crio-b530060702c06b4b43a5a8cdf318bc791c379fc3c905612530366b47f66d02c8 WatchSource:0}: Error finding container b530060702c06b4b43a5a8cdf318bc791c379fc3c905612530366b47f66d02c8: Status 404 returned error can't find the container with id b530060702c06b4b43a5a8cdf318bc791c379fc3c905612530366b47f66d02c8
Oct 31 12:26:37.887760 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:37.887734    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bbe13aa_df4b_40b0_8595_68fe1166a082.slice/crio-d14f2152e247518ab96f882b2d1c789e2985ca7b210789682e82d1d99b01a2ea WatchSource:0}: Error finding container d14f2152e247518ab96f882b2d1c789e2985ca7b210789682e82d1d99b01a2ea: Status 404 returned error can't find the container with id d14f2152e247518ab96f882b2d1c789e2985ca7b210789682e82d1d99b01a2ea
Oct 31 12:26:37.890205 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.890188    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jmlvc"
Oct 31 12:26:37.892805 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.892785    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-44zj2"
Oct 31 12:26:37.896061 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.896043    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-ccps4"
Oct 31 12:26:37.898029 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:37.898009    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-8jlvq"
Oct 31 12:26:37.903812 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:37.903761    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1e578b0_2e77_4f52_8a44_379b6059c3d9.slice/crio-c4cb00c97f37c42bcd3b91b32e6ec987fd6fd61186b49b744d6301563f405c70 WatchSource:0}: Error finding container c4cb00c97f37c42bcd3b91b32e6ec987fd6fd61186b49b744d6301563f405c70: Status 404 returned error can't find the container with id c4cb00c97f37c42bcd3b91b32e6ec987fd6fd61186b49b744d6301563f405c70
Oct 31 12:26:37.909009 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:37.908929    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1c3175b_e32e_420e_877f_71bfe64d9c6c.slice/crio-e4538bb23cd43573e0f29c86571dcd40acc1137dc3413f283057119ec8a4a608 WatchSource:0}: Error finding container e4538bb23cd43573e0f29c86571dcd40acc1137dc3413f283057119ec8a4a608: Status 404 returned error can't find the container with id e4538bb23cd43573e0f29c86571dcd40acc1137dc3413f283057119ec8a4a608
Oct 31 12:26:37.912094 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:26:37.911939    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5014b57_9b83_4fc9_b0ed_55793b2b3a98.slice/crio-1fe0c2cef9e318b723913d1aafd071a4c4a50766fab690a4b526ef633e216e36 WatchSource:0}: Error finding container 1fe0c2cef9e318b723913d1aafd071a4c4a50766fab690a4b526ef633e216e36: Status 404 returned error can't find the container with id 1fe0c2cef9e318b723913d1aafd071a4c4a50766fab690a4b526ef633e216e36
Oct 31 12:26:38.251829 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:38.251790    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tf6n\" (UniqueName: \"kubernetes.io/projected/7a402472-ca33-44ae-b598-96fc8e7a5eb2-kube-api-access-2tf6n\") pod \"network-check-target-tdrmt\" (UID: \"7a402472-ca33-44ae-b598-96fc8e7a5eb2\") " pod="openshift-network-diagnostics/network-check-target-tdrmt"
Oct 31 12:26:38.252017 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:38.251866    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e64bd69d-946f-42ad-94ae-b87b7de3a581-metrics-certs\") pod \"network-metrics-daemon-94kh9\" (UID: \"e64bd69d-946f-42ad-94ae-b87b7de3a581\") " pod="openshift-multus/network-metrics-daemon-94kh9"
Oct 31 12:26:38.252017 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:38.251916    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a2b450be-ca79-4edf-a1ea-377531a49519-original-pull-secret\") pod \"global-pull-secret-syncer-2bmzr\" (UID: \"a2b450be-ca79-4edf-a1ea-377531a49519\") " pod="kube-system/global-pull-secret-syncer-2bmzr"
Oct 31 12:26:38.252017 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:26:38.251930    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 12:26:38.252017 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:26:38.251954    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 12:26:38.252017 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:26:38.251969    2576 projected.go:196] Error preparing data for projected volume kube-api-access-2tf6n for pod openshift-network-diagnostics/network-check-target-tdrmt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 12:26:38.252221 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:26:38.252026    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 12:26:38.252221 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:26:38.252026    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 12:26:38.252221 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:26:38.252030    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7a402472-ca33-44ae-b598-96fc8e7a5eb2-kube-api-access-2tf6n podName:7a402472-ca33-44ae-b598-96fc8e7a5eb2 nodeName:}" failed. No retries permitted until 2026-10-31 12:26:39.252015124 +0000 UTC m=+4.144662255 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-2tf6n" (UniqueName: "kubernetes.io/projected/7a402472-ca33-44ae-b598-96fc8e7a5eb2-kube-api-access-2tf6n") pod "network-check-target-tdrmt" (UID: "7a402472-ca33-44ae-b598-96fc8e7a5eb2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 12:26:38.252221 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:26:38.252116    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a2b450be-ca79-4edf-a1ea-377531a49519-original-pull-secret podName:a2b450be-ca79-4edf-a1ea-377531a49519 nodeName:}" failed. No retries permitted until 2026-10-31 12:26:39.252098307 +0000 UTC m=+4.144745457 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a2b450be-ca79-4edf-a1ea-377531a49519-original-pull-secret") pod "global-pull-secret-syncer-2bmzr" (UID: "a2b450be-ca79-4edf-a1ea-377531a49519") : object "kube-system"/"original-pull-secret" not registered
Oct 31 12:26:38.252221 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:26:38.252131    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e64bd69d-946f-42ad-94ae-b87b7de3a581-metrics-certs podName:e64bd69d-946f-42ad-94ae-b87b7de3a581 nodeName:}" failed. No retries permitted until 2026-10-31 12:26:39.252123292 +0000 UTC m=+4.144770424 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e64bd69d-946f-42ad-94ae-b87b7de3a581-metrics-certs") pod "network-metrics-daemon-94kh9" (UID: "e64bd69d-946f-42ad-94ae-b87b7de3a581") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 12:26:38.579843 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:38.579722    2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 12:21:37 +0000 UTC" deadline="2028-06-20 16:55:51.574285054 +0000 UTC"
Oct 31 12:26:38.579843 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:38.579770    2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14356h29m12.994521185s"
Oct 31 12:26:38.656918 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:38.656875    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tdrmt"
Oct 31 12:26:38.657118 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:26:38.657029    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tdrmt" podUID="7a402472-ca33-44ae-b598-96fc8e7a5eb2"
Oct 31 12:26:38.672862 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:38.672824    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-ccps4" event={"ID":"6b3e77ff-4445-42fa-a7c3-f35d442c276d","Type":"ContainerStarted","Data":"3cd295581fbb43bd6a480a27b300dbdfe231373fcec52e0d8ceeb87c35bd6500"}
Oct 31 12:26:38.680047 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:38.680013    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-8jlvq" event={"ID":"b5014b57-9b83-4fc9-b0ed-55793b2b3a98","Type":"ContainerStarted","Data":"1fe0c2cef9e318b723913d1aafd071a4c4a50766fab690a4b526ef633e216e36"}
Oct 31 12:26:38.689381 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:38.689208    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-44zj2" event={"ID":"a1c3175b-e32e-420e-877f-71bfe64d9c6c","Type":"ContainerStarted","Data":"e4538bb23cd43573e0f29c86571dcd40acc1137dc3413f283057119ec8a4a608"}
Oct 31 12:26:38.698703 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:38.698674    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jmlvc" event={"ID":"a1e578b0-2e77-4f52-8a44-379b6059c3d9","Type":"ContainerStarted","Data":"c4cb00c97f37c42bcd3b91b32e6ec987fd6fd61186b49b744d6301563f405c70"}
Oct 31 12:26:38.712838 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:38.712761    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-wdkzp" event={"ID":"6c661ce8-9f7b-4509-bd17-2380bcd1c425","Type":"ContainerStarted","Data":"1712b1a89fc033bb7af133fed6fccaed65abbd63124e20f0e3ea940d30aa7113"}
Oct 31 12:26:38.721847 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:38.721811    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-v9ks4" event={"ID":"6bbe13aa-df4b-40b0-8595-68fe1166a082","Type":"ContainerStarted","Data":"d14f2152e247518ab96f882b2d1c789e2985ca7b210789682e82d1d99b01a2ea"}
Oct 31 12:26:38.726393 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:38.726364    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-86758" event={"ID":"d43a4e16-3834-4d31-bed4-42fc0aaaaf17","Type":"ContainerStarted","Data":"b530060702c06b4b43a5a8cdf318bc791c379fc3c905612530366b47f66d02c8"}
Oct 31 12:26:38.732976 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:38.732922    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-qkt2l" event={"ID":"51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee","Type":"ContainerStarted","Data":"1bf8d18acbe5580d4e21bb564614ffbafbeafb5e804a96c1fee8a4417b7bc588"}
Oct 31 12:26:38.753180 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:38.753151    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-plbnz" event={"ID":"3bb2c04e-55a2-4206-9530-7298f4469379","Type":"ContainerStarted","Data":"8a79b2069dde730f773d0ac904f6e23f6db008b97e8fd24680a8e7fa053f2e9f"}
Oct 31 12:26:39.260621 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:39.260583    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e64bd69d-946f-42ad-94ae-b87b7de3a581-metrics-certs\") pod \"network-metrics-daemon-94kh9\" (UID: \"e64bd69d-946f-42ad-94ae-b87b7de3a581\") " pod="openshift-multus/network-metrics-daemon-94kh9"
Oct 31 12:26:39.260826 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:39.260662    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a2b450be-ca79-4edf-a1ea-377531a49519-original-pull-secret\") pod \"global-pull-secret-syncer-2bmzr\" (UID: \"a2b450be-ca79-4edf-a1ea-377531a49519\") " pod="kube-system/global-pull-secret-syncer-2bmzr"
Oct 31 12:26:39.260826 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:39.260698    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tf6n\" (UniqueName: \"kubernetes.io/projected/7a402472-ca33-44ae-b598-96fc8e7a5eb2-kube-api-access-2tf6n\") pod \"network-check-target-tdrmt\" (UID: \"7a402472-ca33-44ae-b598-96fc8e7a5eb2\") " pod="openshift-network-diagnostics/network-check-target-tdrmt"
Oct 31 12:26:39.260957 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:26:39.260870    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 12:26:39.260957 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:26:39.260950    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 12:26:39.261071 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:26:39.260969    2576 projected.go:196] Error preparing data for projected volume kube-api-access-2tf6n for pod openshift-network-diagnostics/network-check-target-tdrmt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 12:26:39.261071 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:26:39.261036    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7a402472-ca33-44ae-b598-96fc8e7a5eb2-kube-api-access-2tf6n podName:7a402472-ca33-44ae-b598-96fc8e7a5eb2 nodeName:}" failed. No retries permitted until 2026-10-31 12:26:41.261014963 +0000 UTC m=+6.153662095 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-2tf6n" (UniqueName: "kubernetes.io/projected/7a402472-ca33-44ae-b598-96fc8e7a5eb2-kube-api-access-2tf6n") pod "network-check-target-tdrmt" (UID: "7a402472-ca33-44ae-b598-96fc8e7a5eb2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 12:26:39.261408 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:26:39.261311    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 12:26:39.261408 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:26:39.261350    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 12:26:39.261408 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:26:39.261375    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a2b450be-ca79-4edf-a1ea-377531a49519-original-pull-secret podName:a2b450be-ca79-4edf-a1ea-377531a49519 nodeName:}" failed. No retries permitted until 2026-10-31 12:26:41.261358629 +0000 UTC m=+6.154005787 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a2b450be-ca79-4edf-a1ea-377531a49519-original-pull-secret") pod "global-pull-secret-syncer-2bmzr" (UID: "a2b450be-ca79-4edf-a1ea-377531a49519") : object "kube-system"/"original-pull-secret" not registered
Oct 31 12:26:39.261408 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:26:39.261409    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e64bd69d-946f-42ad-94ae-b87b7de3a581-metrics-certs podName:e64bd69d-946f-42ad-94ae-b87b7de3a581 nodeName:}" failed. No retries permitted until 2026-10-31 12:26:41.261391773 +0000 UTC m=+6.154038921 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e64bd69d-946f-42ad-94ae-b87b7de3a581-metrics-certs") pod "network-metrics-daemon-94kh9" (UID: "e64bd69d-946f-42ad-94ae-b87b7de3a581") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 12:26:39.580919 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:39.580849    2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 12:21:37 +0000 UTC" deadline="2028-07-12 08:55:49.83371614 +0000 UTC"
Oct 31 12:26:39.580919 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:39.580916    2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14876h29m10.252803947s"
Oct 31 12:26:39.659755 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:39.658720    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-94kh9"
Oct 31 12:26:39.659755 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:26:39.659135    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-94kh9" podUID="e64bd69d-946f-42ad-94ae-b87b7de3a581"
Oct 31 12:26:39.660814 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:39.660214    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2bmzr"
Oct 31 12:26:39.660814 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:26:39.660328    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2bmzr" podUID="a2b450be-ca79-4edf-a1ea-377531a49519"
Oct 31 12:26:39.775445 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:39.774757    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-190.us-east-2.compute.internal" event={"ID":"7fb532212935ad861267bbe741c1366b","Type":"ContainerStarted","Data":"dfda9c75e01d4df0bb0b1f645cc9fd65c59496e8697e211e22f1e8daccc12d03"}
Oct 31 12:26:40.656334 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:40.656261    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tdrmt"
Oct 31 12:26:40.657186 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:26:40.656391    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tdrmt" podUID="7a402472-ca33-44ae-b598-96fc8e7a5eb2"
Oct 31 12:26:40.792434 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:40.791197    2576 generic.go:356] "Generic (PLEG): container finished" podID="928a0065cb02eec7c5308f7ecf1e7e6c" containerID="e8f994fa7a328558aca930a66289edc6895311b96cd49b2b7ce9af1967439f32" exitCode=0
Oct 31 12:26:40.792434 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:40.792138    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-190.us-east-2.compute.internal" event={"ID":"928a0065cb02eec7c5308f7ecf1e7e6c","Type":"ContainerDied","Data":"e8f994fa7a328558aca930a66289edc6895311b96cd49b2b7ce9af1967439f32"}
Oct 31 12:26:40.811356 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:40.811309    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-190.us-east-2.compute.internal" podStartSLOduration=3.811291513 podStartE2EDuration="3.811291513s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 12:26:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:26:39.794356867 +0000 UTC m=+4.687004024" watchObservedRunningTime="2026-10-31 12:26:40.811291513 +0000 UTC m=+5.703938664"
Oct 31 12:26:41.283507 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:41.280440    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e64bd69d-946f-42ad-94ae-b87b7de3a581-metrics-certs\") pod \"network-metrics-daemon-94kh9\" (UID: \"e64bd69d-946f-42ad-94ae-b87b7de3a581\") " pod="openshift-multus/network-metrics-daemon-94kh9"
Oct 31 12:26:41.283507 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:41.280524    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a2b450be-ca79-4edf-a1ea-377531a49519-original-pull-secret\") pod \"global-pull-secret-syncer-2bmzr\" (UID: \"a2b450be-ca79-4edf-a1ea-377531a49519\") " pod="kube-system/global-pull-secret-syncer-2bmzr"
Oct 31 12:26:41.283507 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:41.280564    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tf6n\" (UniqueName: \"kubernetes.io/projected/7a402472-ca33-44ae-b598-96fc8e7a5eb2-kube-api-access-2tf6n\") pod \"network-check-target-tdrmt\" (UID: \"7a402472-ca33-44ae-b598-96fc8e7a5eb2\") " pod="openshift-network-diagnostics/network-check-target-tdrmt"
Oct 31 12:26:41.283507 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:26:41.280728    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 12:26:41.283507 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:26:41.280746    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 12:26:41.283507 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:26:41.280758    2576 projected.go:196] Error preparing data for projected volume kube-api-access-2tf6n for pod openshift-network-diagnostics/network-check-target-tdrmt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 12:26:41.283507 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:26:41.280815    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7a402472-ca33-44ae-b598-96fc8e7a5eb2-kube-api-access-2tf6n podName:7a402472-ca33-44ae-b598-96fc8e7a5eb2 nodeName:}" failed. No retries permitted until 2026-10-31 12:26:45.280797275 +0000 UTC m=+10.173444426 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-2tf6n" (UniqueName: "kubernetes.io/projected/7a402472-ca33-44ae-b598-96fc8e7a5eb2-kube-api-access-2tf6n") pod "network-check-target-tdrmt" (UID: "7a402472-ca33-44ae-b598-96fc8e7a5eb2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 12:26:41.283507 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:26:41.281209    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 12:26:41.283507 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:26:41.281259    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e64bd69d-946f-42ad-94ae-b87b7de3a581-metrics-certs podName:e64bd69d-946f-42ad-94ae-b87b7de3a581 nodeName:}" failed. No retries permitted until 2026-10-31 12:26:45.281243617 +0000 UTC m=+10.173890747 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e64bd69d-946f-42ad-94ae-b87b7de3a581-metrics-certs") pod "network-metrics-daemon-94kh9" (UID: "e64bd69d-946f-42ad-94ae-b87b7de3a581") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 12:26:41.283507 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:26:41.281319    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 12:26:41.283507 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:26:41.281350    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a2b450be-ca79-4edf-a1ea-377531a49519-original-pull-secret podName:a2b450be-ca79-4edf-a1ea-377531a49519 nodeName:}" failed. No retries permitted until 2026-10-31 12:26:45.281339381 +0000 UTC m=+10.173986528 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a2b450be-ca79-4edf-a1ea-377531a49519-original-pull-secret") pod "global-pull-secret-syncer-2bmzr" (UID: "a2b450be-ca79-4edf-a1ea-377531a49519") : object "kube-system"/"original-pull-secret" not registered
Oct 31 12:26:41.658810 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:41.658740    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-94kh9"
Oct 31 12:26:41.659253 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:26:41.658864    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-94kh9" podUID="e64bd69d-946f-42ad-94ae-b87b7de3a581"
Oct 31 12:26:41.659253 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:41.659221    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2bmzr"
Oct 31 12:26:41.659359 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:26:41.659316    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2bmzr" podUID="a2b450be-ca79-4edf-a1ea-377531a49519"
Oct 31 12:26:41.798476 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:41.798441    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-190.us-east-2.compute.internal" event={"ID":"928a0065cb02eec7c5308f7ecf1e7e6c","Type":"ContainerStarted","Data":"c2e7f76aa5d274e97fce9326472fe173770b9da4acf41adfb8ecba48d8232731"}
Oct 31 12:26:41.804606 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:41.804015    2576 generic.go:356] "Generic (PLEG): container finished" podID="a1c3175b-e32e-420e-877f-71bfe64d9c6c" containerID="bb15aebbbc18ac0274c0f0e477b19cf85a26e0215d4399f57bb1f8e8750991bd" exitCode=0
Oct 31 12:26:41.804606 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:41.804057    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-44zj2" event={"ID":"a1c3175b-e32e-420e-877f-71bfe64d9c6c","Type":"ContainerDied","Data":"bb15aebbbc18ac0274c0f0e477b19cf85a26e0215d4399f57bb1f8e8750991bd"}
Oct 31 12:26:41.846435 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:41.846036    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-190.us-east-2.compute.internal" podStartSLOduration=4.846016937 podStartE2EDuration="4.846016937s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 12:26:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:26:41.818962371 +0000 UTC m=+6.711609523" watchObservedRunningTime="2026-10-31 12:26:41.846016937 +0000 UTC m=+6.738664090"
Oct 31 12:26:42.657165 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:42.657083    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tdrmt"
Oct 31 12:26:42.657323 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:26:42.657222    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tdrmt" podUID="7a402472-ca33-44ae-b598-96fc8e7a5eb2"
Oct 31 12:26:42.809467 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:42.809436    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-8jlvq" event={"ID":"b5014b57-9b83-4fc9-b0ed-55793b2b3a98","Type":"ContainerStarted","Data":"f09b6f4236accadad7b5765be2afb51a22c77ec28b615dbcf14a99c8d1b3b8ab"}
Oct 31 12:26:43.658937 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:43.658901    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-94kh9"
Oct 31 12:26:43.659112 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:26:43.659044    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-94kh9" podUID="e64bd69d-946f-42ad-94ae-b87b7de3a581"
Oct 31 12:26:43.659477 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:43.659447    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2bmzr"
Oct 31 12:26:43.659598 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:26:43.659559    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2bmzr" podUID="a2b450be-ca79-4edf-a1ea-377531a49519"
Oct 31 12:26:43.814244 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:43.814189    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-wdkzp" event={"ID":"6c661ce8-9f7b-4509-bd17-2380bcd1c425","Type":"ContainerStarted","Data":"4fda387b07ed3e65cb4fad599acdcd23c883938cae7a5cffe111c60b61d0a812"}
Oct 31 12:26:43.830844 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:43.830791    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-8jlvq" podStartSLOduration=3.381419354 podStartE2EDuration="7.83076803s" totalImagesPullingTime="4.449348676s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:26:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:26:42.826983105 +0000 UTC m=+7.719630256" watchObservedRunningTime="2026-10-31 12:26:43.83076803 +0000 UTC m=+8.723415181"
Oct 31 12:26:44.657650 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:44.656929    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tdrmt"
Oct 31 12:26:44.657650 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:26:44.657066    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tdrmt" podUID="7a402472-ca33-44ae-b598-96fc8e7a5eb2"
Oct 31 12:26:44.820232 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:44.820107    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jmlvc" event={"ID":"a1e578b0-2e77-4f52-8a44-379b6059c3d9","Type":"ContainerStarted","Data":"ab138ef24cdb3058d8632de1a416b5d9b1dabc9108b2667be26515a1816a6154"}
Oct 31 12:26:44.969165 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:44.969028    2576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-8jlvq"
Oct 31 12:26:44.970323 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:44.970295    2576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-8jlvq"
Oct 31 12:26:44.988022 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:44.987968    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-wdkzp" podStartSLOduration=3.644796167 podStartE2EDuration="8.987949685s" totalImagesPullingTime="5.343153518s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 12:26:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:26:43.83110341 +0000 UTC m=+8.723750564" watchObservedRunningTime="2026-10-31 12:26:44.987949685 +0000 UTC m=+9.880596840"
Oct 31 12:26:45.322793 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:45.322753    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a2b450be-ca79-4edf-a1ea-377531a49519-original-pull-secret\") pod \"global-pull-secret-syncer-2bmzr\" (UID: \"a2b450be-ca79-4edf-a1ea-377531a49519\") " pod="kube-system/global-pull-secret-syncer-2bmzr"
Oct 31 12:26:45.322971 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:45.322806    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tf6n\" (UniqueName: \"kubernetes.io/projected/7a402472-ca33-44ae-b598-96fc8e7a5eb2-kube-api-access-2tf6n\") pod \"network-check-target-tdrmt\" (UID: \"7a402472-ca33-44ae-b598-96fc8e7a5eb2\") " pod="openshift-network-diagnostics/network-check-target-tdrmt"
Oct 31 12:26:45.322971 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:45.322844    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e64bd69d-946f-42ad-94ae-b87b7de3a581-metrics-certs\") pod \"network-metrics-daemon-94kh9\" (UID: \"e64bd69d-946f-42ad-94ae-b87b7de3a581\") " pod="openshift-multus/network-metrics-daemon-94kh9"
Oct 31 12:26:45.323084 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:26:45.322967    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 12:26:45.323084 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:26:45.323030    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e64bd69d-946f-42ad-94ae-b87b7de3a581-metrics-certs podName:e64bd69d-946f-42ad-94ae-b87b7de3a581 nodeName:}" failed. No retries permitted until 2026-10-31 12:26:53.323011074 +0000 UTC m=+18.215658219 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e64bd69d-946f-42ad-94ae-b87b7de3a581-metrics-certs") pod "network-metrics-daemon-94kh9" (UID: "e64bd69d-946f-42ad-94ae-b87b7de3a581") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 12:26:45.323440 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:26:45.323419    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 12:26:45.323619 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:26:45.323479    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a2b450be-ca79-4edf-a1ea-377531a49519-original-pull-secret podName:a2b450be-ca79-4edf-a1ea-377531a49519 nodeName:}" failed. No retries permitted until 2026-10-31 12:26:53.323462688 +0000 UTC m=+18.216109831 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a2b450be-ca79-4edf-a1ea-377531a49519-original-pull-secret") pod "global-pull-secret-syncer-2bmzr" (UID: "a2b450be-ca79-4edf-a1ea-377531a49519") : object "kube-system"/"original-pull-secret" not registered
Oct 31 12:26:45.323619 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:26:45.323581    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 12:26:45.323619 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:26:45.323600    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 12:26:45.323619 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:26:45.323611    2576 projected.go:196] Error preparing data for projected volume kube-api-access-2tf6n for pod openshift-network-diagnostics/network-check-target-tdrmt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 12:26:45.323799 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:26:45.323655    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7a402472-ca33-44ae-b598-96fc8e7a5eb2-kube-api-access-2tf6n podName:7a402472-ca33-44ae-b598-96fc8e7a5eb2 nodeName:}" failed. No retries permitted until 2026-10-31 12:26:53.323643457 +0000 UTC m=+18.216290600 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-2tf6n" (UniqueName: "kubernetes.io/projected/7a402472-ca33-44ae-b598-96fc8e7a5eb2-kube-api-access-2tf6n") pod "network-check-target-tdrmt" (UID: "7a402472-ca33-44ae-b598-96fc8e7a5eb2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 12:26:45.657645 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:45.657615    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2bmzr"
Oct 31 12:26:45.658454 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:26:45.658424    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2bmzr" podUID="a2b450be-ca79-4edf-a1ea-377531a49519"
Oct 31 12:26:45.658952 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:45.658791    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-94kh9"
Oct 31 12:26:45.658952 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:26:45.658909    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-94kh9" podUID="e64bd69d-946f-42ad-94ae-b87b7de3a581"
Oct 31 12:26:45.688903 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:45.688854    2576 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 31 12:26:45.826889 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:45.826856    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jmlvc" event={"ID":"a1e578b0-2e77-4f52-8a44-379b6059c3d9","Type":"ContainerStarted","Data":"1c304a626f580257e9ba95a1c0bb2b2706cc5627d08cacbfea3e5b23ca2412df"}
Oct 31 12:26:46.595336 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:46.595058    2576 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-31T12:26:45.688871887Z","UUID":"f429b254-b0b7-4660-b542-7cddb452d408","Handler":null,"Name":"","Endpoint":""}
Oct 31 12:26:46.596841 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:46.596814    2576 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 31 12:26:46.596841 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:46.596840    2576 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 31 12:26:46.656447 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:46.656414    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tdrmt"
Oct 31 12:26:46.656636 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:26:46.656569    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tdrmt" podUID="7a402472-ca33-44ae-b598-96fc8e7a5eb2"
Oct 31 12:26:46.832722 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:46.832166    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-ccps4" event={"ID":"6b3e77ff-4445-42fa-a7c3-f35d442c276d","Type":"ContainerStarted","Data":"7386ab03c0114a2e96a01cf1d1ac765516da95220b5b2649c797bdccfad6c59e"}
Oct 31 12:26:46.835610 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:46.835584    2576 generic.go:356] "Generic (PLEG): container finished" podID="a1c3175b-e32e-420e-877f-71bfe64d9c6c" containerID="5277aadaa97b46ce60e04e87b5b25f8b8ffaef3c16a182a3798a6c0523fb2caf" exitCode=0
Oct 31 12:26:46.835748 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:46.835729    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-44zj2" event={"ID":"a1c3175b-e32e-420e-877f-71bfe64d9c6c","Type":"ContainerDied","Data":"5277aadaa97b46ce60e04e87b5b25f8b8ffaef3c16a182a3798a6c0523fb2caf"}
Oct 31 12:26:46.837922 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:46.837899    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-v9ks4" event={"ID":"6bbe13aa-df4b-40b0-8595-68fe1166a082","Type":"ContainerStarted","Data":"49b8c119744e3a5e48d80929f982fecdb8ed75bbd711907cf81c2949c27f4db2"}
Oct 31 12:26:46.853148 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:46.853100    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-ccps4" podStartSLOduration=2.657752749 podStartE2EDuration="10.853085665s" totalImagesPullingTime="8.195332916s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 12:26:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:26:46.853063133 +0000 UTC m=+11.745710295" watchObservedRunningTime="2026-10-31 12:26:46.853085665 +0000 UTC m=+11.745732816"
Oct 31 12:26:47.659123 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:47.659089    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-94kh9"
Oct 31 12:26:47.659333 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:26:47.659221    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-94kh9" podUID="e64bd69d-946f-42ad-94ae-b87b7de3a581"
Oct 31 12:26:47.660033 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:47.659615    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2bmzr"
Oct 31 12:26:47.660033 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:26:47.659712    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2bmzr" podUID="a2b450be-ca79-4edf-a1ea-377531a49519"
Oct 31 12:26:47.846183 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:47.845987    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-44zj2" event={"ID":"a1c3175b-e32e-420e-877f-71bfe64d9c6c","Type":"ContainerStarted","Data":"e09b86dbee27e9abd088acc93d8bff8dcf71d5ec422aae1a04d82c64bdef0d3d"}
Oct 31 12:26:47.849656 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:47.849629    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jmlvc" event={"ID":"a1e578b0-2e77-4f52-8a44-379b6059c3d9","Type":"ContainerStarted","Data":"9c6c6e8162f76431bc4bcbb0f48d598bec3e5bc79992c3dcefc42c75759e9b7c"}
Oct 31 12:26:47.852950 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:47.852523    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-plbnz" event={"ID":"3bb2c04e-55a2-4206-9530-7298f4469379","Type":"ContainerStarted","Data":"5565ffc762da9c309bba827654d61b9b03d94855d123018c888843cade9d5294"}
Oct 31 12:26:47.874089 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:47.874030    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-v9ks4" podStartSLOduration=3.771780464 podStartE2EDuration="11.874012653s" totalImagesPullingTime="8.102232189s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 12:26:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:26:46.896958107 +0000 UTC m=+11.789605257" watchObservedRunningTime="2026-10-31 12:26:47.874012653 +0000 UTC m=+12.766659807"
Oct 31 12:26:47.894294 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:47.894245    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jmlvc" podStartSLOduration=3.344200133 podStartE2EDuration="11.894226239s" totalImagesPullingTime="8.550026106s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 12:26:36 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:26:47.893893899 +0000 UTC m=+12.786541053" watchObservedRunningTime="2026-10-31 12:26:47.894226239 +0000 UTC m=+12.786873398"
Oct 31 12:26:48.656291 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:48.656261    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tdrmt"
Oct 31 12:26:48.656448 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:26:48.656366    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tdrmt" podUID="7a402472-ca33-44ae-b598-96fc8e7a5eb2"
Oct 31 12:26:48.855717 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:48.855676    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-qkt2l" event={"ID":"51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee","Type":"ContainerStarted","Data":"f0a7bfc53caf021af6772e69bece8ad7e3f7395cb22d54ebe3241c42c23f8d0e"}
Oct 31 12:26:48.857897 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:48.857872    2576 generic.go:356] "Generic (PLEG): container finished" podID="a1c3175b-e32e-420e-877f-71bfe64d9c6c" containerID="e09b86dbee27e9abd088acc93d8bff8dcf71d5ec422aae1a04d82c64bdef0d3d" exitCode=0
Oct 31 12:26:48.858003 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:48.857939    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-44zj2" event={"ID":"a1c3175b-e32e-420e-877f-71bfe64d9c6c","Type":"ContainerDied","Data":"e09b86dbee27e9abd088acc93d8bff8dcf71d5ec422aae1a04d82c64bdef0d3d"}
Oct 31 12:26:48.876036 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:48.875991    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-plbnz" podStartSLOduration=4.576612327 podStartE2EDuration="12.875973484s" totalImagesPullingTime="8.299361157s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 12:26:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:26:47.909890094 +0000 UTC m=+12.802537244" watchObservedRunningTime="2026-10-31 12:26:48.875973484 +0000 UTC m=+13.768620637"
Oct 31 12:26:48.876254 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:48.876232    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-qkt2l" podStartSLOduration=2.628028842 podStartE2EDuration="12.876227554s" totalImagesPullingTime="10.248198712s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 12:26:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:26:48.875612851 +0000 UTC m=+13.768260001" watchObservedRunningTime="2026-10-31 12:26:48.876227554 +0000 UTC m=+13.768874705"
Oct 31 12:26:49.659134 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:49.659101    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-94kh9"
Oct 31 12:26:49.659284 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:49.659101    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2bmzr"
Oct 31 12:26:49.659284 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:26:49.659240    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-94kh9" podUID="e64bd69d-946f-42ad-94ae-b87b7de3a581"
Oct 31 12:26:49.659378 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:26:49.659306    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2bmzr" podUID="a2b450be-ca79-4edf-a1ea-377531a49519"
Oct 31 12:26:50.656350 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:50.656155    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tdrmt"
Oct 31 12:26:50.656841 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:26:50.656436    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tdrmt" podUID="7a402472-ca33-44ae-b598-96fc8e7a5eb2"
Oct 31 12:26:50.865086 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:50.865054    2576 generic.go:356] "Generic (PLEG): container finished" podID="a1c3175b-e32e-420e-877f-71bfe64d9c6c" containerID="1a72cc05db4951f01ee9048390ecbb98a21b223630b33c9fcb888f790ed78755" exitCode=0
Oct 31 12:26:50.865268 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:50.865108    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-44zj2" event={"ID":"a1c3175b-e32e-420e-877f-71bfe64d9c6c","Type":"ContainerDied","Data":"1a72cc05db4951f01ee9048390ecbb98a21b223630b33c9fcb888f790ed78755"}
Oct 31 12:26:51.660048 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:51.660018    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2bmzr"
Oct 31 12:26:51.660417 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:26:51.660147    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2bmzr" podUID="a2b450be-ca79-4edf-a1ea-377531a49519"
Oct 31 12:26:51.660417 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:51.660189    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-94kh9"
Oct 31 12:26:51.660417 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:26:51.660299    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-94kh9" podUID="e64bd69d-946f-42ad-94ae-b87b7de3a581"
Oct 31 12:26:52.656786 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:52.656749    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tdrmt"
Oct 31 12:26:52.656999 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:26:52.656856    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tdrmt" podUID="7a402472-ca33-44ae-b598-96fc8e7a5eb2"
Oct 31 12:26:52.871217 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:52.871183    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-86758" event={"ID":"d43a4e16-3834-4d31-bed4-42fc0aaaaf17","Type":"ContainerStarted","Data":"576e2b612f36003bf235657d527f273ec1dcd64359e146570dde919d4f0b6d16"}
Oct 31 12:26:52.871647 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:52.871300    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-86758" event={"ID":"d43a4e16-3834-4d31-bed4-42fc0aaaaf17","Type":"ContainerStarted","Data":"6bb8a3489fc1c5353d74c0927698043f475c920a88d2901fd64c2bcc503d3f8a"}
Oct 31 12:26:53.380284 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:53.380234    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e64bd69d-946f-42ad-94ae-b87b7de3a581-metrics-certs\") pod \"network-metrics-daemon-94kh9\" (UID: \"e64bd69d-946f-42ad-94ae-b87b7de3a581\") " pod="openshift-multus/network-metrics-daemon-94kh9"
Oct 31 12:26:53.380284 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:53.380300    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a2b450be-ca79-4edf-a1ea-377531a49519-original-pull-secret\") pod \"global-pull-secret-syncer-2bmzr\" (UID: \"a2b450be-ca79-4edf-a1ea-377531a49519\") " pod="kube-system/global-pull-secret-syncer-2bmzr"
Oct 31 12:26:53.380624 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:26:53.380424    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 12:26:53.380624 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:53.380420    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tf6n\" (UniqueName: \"kubernetes.io/projected/7a402472-ca33-44ae-b598-96fc8e7a5eb2-kube-api-access-2tf6n\") pod \"network-check-target-tdrmt\" (UID: \"7a402472-ca33-44ae-b598-96fc8e7a5eb2\") " pod="openshift-network-diagnostics/network-check-target-tdrmt"
Oct 31 12:26:53.380624 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:26:53.380517    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e64bd69d-946f-42ad-94ae-b87b7de3a581-metrics-certs podName:e64bd69d-946f-42ad-94ae-b87b7de3a581 nodeName:}" failed. No retries permitted until 2026-10-31 12:27:09.380479156 +0000 UTC m=+34.273126291 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e64bd69d-946f-42ad-94ae-b87b7de3a581-metrics-certs") pod "network-metrics-daemon-94kh9" (UID: "e64bd69d-946f-42ad-94ae-b87b7de3a581") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 12:26:53.380624 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:26:53.380527    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 12:26:53.380624 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:26:53.380542    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 12:26:53.380624 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:26:53.380550    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 12:26:53.380624 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:26:53.380562    2576 projected.go:196] Error preparing data for projected volume kube-api-access-2tf6n for pod openshift-network-diagnostics/network-check-target-tdrmt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 12:26:53.380624 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:26:53.380588    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a2b450be-ca79-4edf-a1ea-377531a49519-original-pull-secret podName:a2b450be-ca79-4edf-a1ea-377531a49519 nodeName:}" failed. No retries permitted until 2026-10-31 12:27:09.380572654 +0000 UTC m=+34.273219786 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a2b450be-ca79-4edf-a1ea-377531a49519-original-pull-secret") pod "global-pull-secret-syncer-2bmzr" (UID: "a2b450be-ca79-4edf-a1ea-377531a49519") : object "kube-system"/"original-pull-secret" not registered
Oct 31 12:26:53.380624 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:26:53.380605    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7a402472-ca33-44ae-b598-96fc8e7a5eb2-kube-api-access-2tf6n podName:7a402472-ca33-44ae-b598-96fc8e7a5eb2 nodeName:}" failed. No retries permitted until 2026-10-31 12:27:09.380596785 +0000 UTC m=+34.273243914 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-2tf6n" (UniqueName: "kubernetes.io/projected/7a402472-ca33-44ae-b598-96fc8e7a5eb2-kube-api-access-2tf6n") pod "network-check-target-tdrmt" (UID: "7a402472-ca33-44ae-b598-96fc8e7a5eb2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 12:26:53.656883 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:53.656795    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2bmzr"
Oct 31 12:26:53.656883 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:53.656843    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-94kh9"
Oct 31 12:26:53.657038 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:26:53.656923    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2bmzr" podUID="a2b450be-ca79-4edf-a1ea-377531a49519"
Oct 31 12:26:53.657083 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:26:53.657044    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-94kh9" podUID="e64bd69d-946f-42ad-94ae-b87b7de3a581"
Oct 31 12:26:53.876140 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:53.876107    2576 generic.go:356] "Generic (PLEG): container finished" podID="a1c3175b-e32e-420e-877f-71bfe64d9c6c" containerID="44ccdd536d82848d2351206309ee00963dc164980658786d52f9f51997905e40" exitCode=0
Oct 31 12:26:53.876707 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:53.876192    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-44zj2" event={"ID":"a1c3175b-e32e-420e-877f-71bfe64d9c6c","Type":"ContainerDied","Data":"44ccdd536d82848d2351206309ee00963dc164980658786d52f9f51997905e40"}
Oct 31 12:26:53.879257 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:53.879237    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-86758" event={"ID":"d43a4e16-3834-4d31-bed4-42fc0aaaaf17","Type":"ContainerStarted","Data":"c63d83561266ed26f9b86ebfef5b34ba0a5d011f15ab9d14fe5fa7246c826167"}
Oct 31 12:26:53.879332 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:53.879296    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-86758" event={"ID":"d43a4e16-3834-4d31-bed4-42fc0aaaaf17","Type":"ContainerStarted","Data":"3463d58a6f8a2c4688a8b1d3d8b68f1ea9d1981be5a576c2711b216ac5de3ce9"}
Oct 31 12:26:53.879385 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:53.879334    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-86758" event={"ID":"d43a4e16-3834-4d31-bed4-42fc0aaaaf17","Type":"ContainerStarted","Data":"129e52b8b16c2cf564f2edb94453f8f1c352028a9ce890dcf121acf9be319082"}
Oct 31 12:26:53.879385 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:53.879364    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-86758" event={"ID":"d43a4e16-3834-4d31-bed4-42fc0aaaaf17","Type":"ContainerStarted","Data":"edb67204e41d79e542f089210953bfb429370382005f7c67f9d97246219da747"}
Oct 31 12:26:54.656791 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:54.656757    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tdrmt"
Oct 31 12:26:54.656932 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:26:54.656860    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tdrmt" podUID="7a402472-ca33-44ae-b598-96fc8e7a5eb2"
Oct 31 12:26:54.883574 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:54.883536    2576 generic.go:356] "Generic (PLEG): container finished" podID="a1c3175b-e32e-420e-877f-71bfe64d9c6c" containerID="39c01b342d9309b09ba86218154e82594fafe0cebfa7ecc46f44bda8d2167341" exitCode=0
Oct 31 12:26:54.883935 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:54.883596    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-44zj2" event={"ID":"a1c3175b-e32e-420e-877f-71bfe64d9c6c","Type":"ContainerDied","Data":"39c01b342d9309b09ba86218154e82594fafe0cebfa7ecc46f44bda8d2167341"}
Oct 31 12:26:55.657023 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:55.656986    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2bmzr"
Oct 31 12:26:55.657209 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:26:55.657071    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2bmzr" podUID="a2b450be-ca79-4edf-a1ea-377531a49519"
Oct 31 12:26:55.657209 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:55.657184    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-94kh9"
Oct 31 12:26:55.657358 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:26:55.657337    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-94kh9" podUID="e64bd69d-946f-42ad-94ae-b87b7de3a581"
Oct 31 12:26:55.888952 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:55.888918    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-86758" event={"ID":"d43a4e16-3834-4d31-bed4-42fc0aaaaf17","Type":"ContainerStarted","Data":"9d68037023e145858968ae8342c7a8f7bd0c0c84a672a241b4c0da67aea8732d"}
Oct 31 12:26:55.892127 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:55.892103    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-44zj2" event={"ID":"a1c3175b-e32e-420e-877f-71bfe64d9c6c","Type":"ContainerStarted","Data":"6b006abce1cbd52aaa9357ec0bfa2be3961d4b0dd065967c1d0787a191fef858"}
Oct 31 12:26:55.920231 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:55.920130    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-44zj2" podStartSLOduration=8.495222631 podStartE2EDuration="19.920116515s" totalImagesPullingTime="11.424893884s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 12:26:36 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:26:55.918010214 +0000 UTC m=+20.810657368" watchObservedRunningTime="2026-10-31 12:26:55.920116515 +0000 UTC m=+20.812763665"
Oct 31 12:26:56.415573 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:56.415532    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-8jlvq"
Oct 31 12:26:56.415732 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:56.415660    2576 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 31 12:26:56.416082 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:56.416067    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-8jlvq"
Oct 31 12:26:56.656606 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:56.656576    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tdrmt"
Oct 31 12:26:56.656762 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:26:56.656664    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tdrmt" podUID="7a402472-ca33-44ae-b598-96fc8e7a5eb2"
Oct 31 12:26:57.656968 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:57.656936    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-94kh9"
Oct 31 12:26:57.657456 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:26:57.657048    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-94kh9" podUID="e64bd69d-946f-42ad-94ae-b87b7de3a581"
Oct 31 12:26:57.657456 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:57.657102    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2bmzr"
Oct 31 12:26:57.657456 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:26:57.657190    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2bmzr" podUID="a2b450be-ca79-4edf-a1ea-377531a49519"
Oct 31 12:26:57.899251 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:57.899215    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-86758" event={"ID":"d43a4e16-3834-4d31-bed4-42fc0aaaaf17","Type":"ContainerStarted","Data":"a4eeafb3e9e3329998ae0bdc391fd7c366cca46b0f6d4ea1ec8e685c35fe72fd"}
Oct 31 12:26:57.899519 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:57.899486    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-86758"
Oct 31 12:26:57.916028 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:57.915967    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-86758"
Oct 31 12:26:57.943302 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:57.943258    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-86758" podStartSLOduration=7.157283974 podStartE2EDuration="21.943243117s" totalImagesPullingTime="14.785959143s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 12:26:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:26:57.943212727 +0000 UTC m=+22.835859877" watchObservedRunningTime="2026-10-31 12:26:57.943243117 +0000 UTC m=+22.835890268"
Oct 31 12:26:58.656868 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:58.656827    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tdrmt"
Oct 31 12:26:58.657003 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:26:58.656966    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tdrmt" podUID="7a402472-ca33-44ae-b598-96fc8e7a5eb2"
Oct 31 12:26:58.901969 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:58.901935    2576 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 31 12:26:58.902454 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:58.902436    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-86758"
Oct 31 12:26:58.918975 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:58.918907    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-86758"
Oct 31 12:26:59.014102 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:59.014069    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-tdrmt"]
Oct 31 12:26:59.014269 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:59.014237    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tdrmt"
Oct 31 12:26:59.014368 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:26:59.014344    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tdrmt" podUID="7a402472-ca33-44ae-b598-96fc8e7a5eb2"
Oct 31 12:26:59.021935 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:59.021898    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-2bmzr"]
Oct 31 12:26:59.022059 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:59.022023    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2bmzr"
Oct 31 12:26:59.022171 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:26:59.022143    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2bmzr" podUID="a2b450be-ca79-4edf-a1ea-377531a49519"
Oct 31 12:26:59.022618 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:59.022595    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-94kh9"]
Oct 31 12:26:59.022705 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:59.022693    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-94kh9"
Oct 31 12:26:59.022790 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:26:59.022773    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-94kh9" podUID="e64bd69d-946f-42ad-94ae-b87b7de3a581"
Oct 31 12:26:59.903535 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:26:59.903507    2576 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 31 12:27:00.656545 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:00.656503    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-94kh9"
Oct 31 12:27:00.656545 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:00.656547    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tdrmt"
Oct 31 12:27:00.656743 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:00.656555    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2bmzr"
Oct 31 12:27:00.656743 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:27:00.656642    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-94kh9" podUID="e64bd69d-946f-42ad-94ae-b87b7de3a581"
Oct 31 12:27:00.656743 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:27:00.656720    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tdrmt" podUID="7a402472-ca33-44ae-b598-96fc8e7a5eb2"
Oct 31 12:27:00.656836 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:27:00.656768    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2bmzr" podUID="a2b450be-ca79-4edf-a1ea-377531a49519"
Oct 31 12:27:00.905427 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:00.905399    2576 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 31 12:27:02.656962 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:02.656928    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-94kh9"
Oct 31 12:27:02.657410 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:02.657037    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2bmzr"
Oct 31 12:27:02.657410 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:27:02.657043    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-94kh9" podUID="e64bd69d-946f-42ad-94ae-b87b7de3a581"
Oct 31 12:27:02.657410 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:27:02.657138    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2bmzr" podUID="a2b450be-ca79-4edf-a1ea-377531a49519"
Oct 31 12:27:02.657410 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:02.657183    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tdrmt"
Oct 31 12:27:02.657410 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:27:02.657234    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tdrmt" podUID="7a402472-ca33-44ae-b598-96fc8e7a5eb2"
Oct 31 12:27:03.508752 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:03.508709    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-86758"
Oct 31 12:27:03.508998 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:03.508967    2576 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 31 12:27:03.520005 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:03.519976    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-86758"
Oct 31 12:27:04.656541 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:04.656333    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-94kh9"
Oct 31 12:27:04.656961 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:04.656390    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tdrmt"
Oct 31 12:27:04.656961 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:27:04.656621    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-94kh9" podUID="e64bd69d-946f-42ad-94ae-b87b7de3a581"
Oct 31 12:27:04.656961 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:04.656397    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2bmzr"
Oct 31 12:27:04.656961 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:27:04.656722    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tdrmt" podUID="7a402472-ca33-44ae-b598-96fc8e7a5eb2"
Oct 31 12:27:04.656961 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:27:04.656753    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2bmzr" podUID="a2b450be-ca79-4edf-a1ea-377531a49519"
Oct 31 12:27:04.874037 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:04.874008    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-190.us-east-2.compute.internal" event="NodeReady"
Oct 31 12:27:04.874199 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:04.874129    2576 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Oct 31 12:27:04.908193 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:04.908163    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-vvzpg"]
Oct 31 12:27:04.912464 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:04.912057    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-vvzpg"
Oct 31 12:27:04.912617 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:04.912422    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-2bfvp"]
Oct 31 12:27:04.917442 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:04.917420    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-k64bz"]
Oct 31 12:27:04.917575 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:04.917544    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-2bfvp"
Oct 31 12:27:04.923784 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:04.923749    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-659589798c-776bx"]
Oct 31 12:27:04.924065 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:04.924019    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-k64bz"
Oct 31 12:27:04.926081 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:04.926045    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\""
Oct 31 12:27:04.926183 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:04.926081    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\""
Oct 31 12:27:04.926938 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:04.926920    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Oct 31 12:27:04.928310 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:04.928286    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Oct 31 12:27:04.929080 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:04.928351    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-vq8lr\""
Oct 31 12:27:04.938582 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:04.931516    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-rrc8k"]
Oct 31 12:27:04.938582 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:04.933013    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-659589798c-776bx"
Oct 31 12:27:04.943472 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:04.942645    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6dd647c7b7-9255m"]
Oct 31 12:27:04.943472 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:04.942755    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-rrc8k"
Oct 31 12:27:04.946302 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:04.946282    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6cfdbfcdc4-2rq84"]
Oct 31 12:27:04.946579 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:04.946558    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6dd647c7b7-9255m"
Oct 31 12:27:04.950481 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:04.949647    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-z879s"]
Oct 31 12:27:04.950481 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:04.949732    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6cfdbfcdc4-2rq84"
Oct 31 12:27:04.952832 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:04.952811    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-gt9rn"]
Oct 31 12:27:04.953964 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:04.953918    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-z879s"
Oct 31 12:27:04.956133 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:04.956105    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-vqsv9"]
Oct 31 12:27:04.956907 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:04.956214    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gt9rn"
Oct 31 12:27:04.959271 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:04.959252    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-hf9hr"]
Oct 31 12:27:04.959397 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:04.959368    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-vqsv9"
Oct 31 12:27:04.962232 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:04.962210    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-pcvvm"]
Oct 31 12:27:04.962388 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:04.962370    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-hf9hr"
Oct 31 12:27:04.965251 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:04.965231    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-c789s"]
Oct 31 12:27:04.965364 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:04.965352    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-pcvvm"
Oct 31 12:27:04.968419 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:04.968398    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-tcnrw"]
Oct 31 12:27:04.968559 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:04.968542    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-c789s"
Oct 31 12:27:04.971209 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:04.971190    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-m9xxh"]
Oct 31 12:27:04.971302 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:04.971250    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-tcnrw"
Oct 31 12:27:04.978458 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:04.978440    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-vvzpg"]
Oct 31 12:27:04.978553 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:04.978470    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-rrc8k"]
Oct 31 12:27:04.978553 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:04.978484    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-2bfvp"]
Oct 31 12:27:04.978553 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:04.978511    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-k64bz"]
Oct 31 12:27:04.978553 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:04.978522    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-659589798c-776bx"]
Oct 31 12:27:04.978553 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:04.978536    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-r6fvt"]
Oct 31 12:27:04.978703 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:04.978592    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-m9xxh"
Oct 31 12:27:04.981603 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:04.981586    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-pcvvm"]
Oct 31 12:27:04.981699 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:04.981613    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-rl55s"]
Oct 31 12:27:04.981764 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:04.981698    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-r6fvt"
Oct 31 12:27:04.984631 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:04.984611    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-gt9rn"]
Oct 31 12:27:04.984706 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:04.984650    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-z879s"]
Oct 31 12:27:04.984706 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:04.984677    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-tcnrw"]
Oct 31 12:27:04.984706 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:04.984705    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6dd647c7b7-9255m"]
Oct 31 12:27:04.984793 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:04.984730    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-hf9hr"]
Oct 31 12:27:04.984793 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:04.984750    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-c789s"]
Oct 31 12:27:04.984793 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:04.984768    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-r6fvt"]
Oct 31 12:27:04.984793 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:04.984775    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-rl55s"
Oct 31 12:27:04.984920 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:04.984794    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-m9xxh"]
Oct 31 12:27:04.984920 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:04.984814    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-rl55s"]
Oct 31 12:27:04.984920 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:04.984835    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6cfdbfcdc4-2rq84"]
Oct 31 12:27:04.984920 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:04.984855    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-vqsv9"]
Oct 31 12:27:05.024446 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.024372    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Oct 31 12:27:05.024646 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.024559    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Oct 31 12:27:05.024766 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.024749    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Oct 31 12:27:05.024839 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.024788    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Oct 31 12:27:05.024899 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.024832    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Oct 31 12:27:05.024899 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.024870    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Oct 31 12:27:05.024899 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.024878    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Oct 31 12:27:05.025041 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.024911    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-ght96\""
Oct 31 12:27:05.025041 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.024924    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Oct 31 12:27:05.025041 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.024981    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Oct 31 12:27:05.025041 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.024999    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Oct 31 12:27:05.025041 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.025019    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 31 12:27:05.025041 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.025031    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 31 12:27:05.025041 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.025036    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 31 12:27:05.025240 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.025064    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-nzzrg\""
Oct 31 12:27:05.025240 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.025064    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Oct 31 12:27:05.025240 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.025075    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 31 12:27:05.025423 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.025404    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Oct 31 12:27:05.025423 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.025416    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\""
Oct 31 12:27:05.025538 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.025437    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Oct 31 12:27:05.025538 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.025450    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Oct 31 12:27:05.025652 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.025631    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Oct 31 12:27:05.026581 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.026563    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 31 12:27:05.026671 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.026605    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 31 12:27:05.026671 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.026629    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Oct 31 12:27:05.026671 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.026659    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-vcqch\""
Oct 31 12:27:05.026819 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.026675    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 31 12:27:05.026819 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.026678    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Oct 31 12:27:05.026819 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.026715    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 31 12:27:05.026819 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.026724    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Oct 31 12:27:05.026819 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.026741    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 31 12:27:05.026819 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.026756    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Oct 31 12:27:05.026819 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.026765    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Oct 31 12:27:05.026819 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.026788    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Oct 31 12:27:05.026819 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.026759    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Oct 31 12:27:05.026819 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.026802    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 31 12:27:05.026819 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.026820    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Oct 31 12:27:05.027170 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.026843    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Oct 31 12:27:05.027170 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.026842    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 31 12:27:05.027170 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.026957    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Oct 31 12:27:05.027170 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.026970    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Oct 31 12:27:05.028451 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.028435    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Oct 31 12:27:05.028546 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.028479    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 31 12:27:05.028546 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.028479    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 31 12:27:05.036534 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.036508    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Oct 31 12:27:05.046662 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.046642    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Oct 31 12:27:05.047158 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.047143    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Oct 31 12:27:05.047283 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.047269    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Oct 31 12:27:05.063776 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.063751    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9083c35a-9739-4f3e-8415-bad1e6433d40-image-registry-private-configuration\") pod \"image-registry-6cfdbfcdc4-2rq84\" (UID: \"9083c35a-9739-4f3e-8415-bad1e6433d40\") " pod="openshift-image-registry/image-registry-6cfdbfcdc4-2rq84"
Oct 31 12:27:05.063950 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.063781    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/556dda7b-d8e6-4558-b5f3-994b01d6a866-service-ca-bundle\") pod \"router-default-659589798c-776bx\" (UID: \"556dda7b-d8e6-4558-b5f3-994b01d6a866\") " pod="openshift-ingress/router-default-659589798c-776bx"
Oct 31 12:27:05.063950 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.063799    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5c8865f9-fdf6-49b9-8b4f-69dbc7787084-metrics-tls\") pod \"dns-default-r6fvt\" (UID: \"5c8865f9-fdf6-49b9-8b4f-69dbc7787084\") " pod="openshift-dns/dns-default-r6fvt"
Oct 31 12:27:05.063950 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.063816    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvqpw\" (UniqueName: \"kubernetes.io/projected/25691fa3-7017-429a-b94c-c2dde4d29e55-kube-api-access-gvqpw\") pod \"network-check-source-6f78d59f99-m9xxh\" (UID: \"25691fa3-7017-429a-b94c-c2dde4d29e55\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-m9xxh"
Oct 31 12:27:05.063950 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.063934    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0db116b4-d971-4966-bb05-db28d077b6fe-ca-trust-extracted\") pod \"image-registry-6dd647c7b7-9255m\" (UID: \"0db116b4-d971-4966-bb05-db28d077b6fe\") " pod="openshift-image-registry/image-registry-6dd647c7b7-9255m"
Oct 31 12:27:05.064122 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.064034    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/dbbd2d70-8911-4aa5-96cf-3a16dcdf246d-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-tcnrw\" (UID: \"dbbd2d70-8911-4aa5-96cf-3a16dcdf246d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-tcnrw"
Oct 31 12:27:05.064122 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.064093    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/556dda7b-d8e6-4558-b5f3-994b01d6a866-default-certificate\") pod \"router-default-659589798c-776bx\" (UID: \"556dda7b-d8e6-4558-b5f3-994b01d6a866\") " pod="openshift-ingress/router-default-659589798c-776bx"
Oct 31 12:27:05.064194 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.064163    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/febbc28e-bbcc-4961-bde8-09944d9ca680-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-vqsv9\" (UID: \"febbc28e-bbcc-4961-bde8-09944d9ca680\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-vqsv9"
Oct 31 12:27:05.064242 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.064193    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0db116b4-d971-4966-bb05-db28d077b6fe-registry-certificates\") pod \"image-registry-6dd647c7b7-9255m\" (UID: \"0db116b4-d971-4966-bb05-db28d077b6fe\") " pod="openshift-image-registry/image-registry-6dd647c7b7-9255m"
Oct 31 12:27:05.064242 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.064224    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jw4wt\" (UniqueName: \"kubernetes.io/projected/dda0c26f-8225-4239-bc27-866b66cbb919-kube-api-access-jw4wt\") pod \"cluster-monitoring-operator-884576c57-gt9rn\" (UID: \"dda0c26f-8225-4239-bc27-866b66cbb919\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gt9rn"
Oct 31 12:27:05.064333 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.064252    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/99a606a8-546b-4145-8bae-a95061241069-serving-cert\") pod \"service-ca-operator-544d749c8c-c789s\" (UID: \"99a606a8-546b-4145-8bae-a95061241069\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-c789s"
Oct 31 12:27:05.064333 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.064275    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wq6d\" (UniqueName: \"kubernetes.io/projected/ddb04b2e-2724-44e9-948a-877a90d8a071-kube-api-access-5wq6d\") pod \"insights-operator-7c4986d449-z879s\" (UID: \"ddb04b2e-2724-44e9-948a-877a90d8a071\") " pod="openshift-insights/insights-operator-7c4986d449-z879s"
Oct 31 12:27:05.064425 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.064345    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrpck\" (UniqueName: \"kubernetes.io/projected/891b3c47-ff7c-4a7a-a09b-411361a843f7-kube-api-access-mrpck\") pod \"ingress-canary-rl55s\" (UID: \"891b3c47-ff7c-4a7a-a09b-411361a843f7\") " pod="openshift-ingress-canary/ingress-canary-rl55s"
Oct 31 12:27:05.064855 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.064717    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvzh5\" (UniqueName: \"kubernetes.io/projected/613125d4-fbb4-4b43-9c47-24e039bd4697-kube-api-access-mvzh5\") pod \"kas-connection-checker-685895c7c9-k64bz\" (UID: \"613125d4-fbb4-4b43-9c47-24e039bd4697\") " pod="kube-system/kas-connection-checker-685895c7c9-k64bz"
Oct 31 12:27:05.064855 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.064778    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ddb04b2e-2724-44e9-948a-877a90d8a071-service-ca-bundle\") pod \"insights-operator-7c4986d449-z879s\" (UID: \"ddb04b2e-2724-44e9-948a-877a90d8a071\") " pod="openshift-insights/insights-operator-7c4986d449-z879s"
Oct 31 12:27:05.064855 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.064835    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/febbc28e-bbcc-4961-bde8-09944d9ca680-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-vqsv9\" (UID: \"febbc28e-bbcc-4961-bde8-09944d9ca680\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-vqsv9"
Oct 31 12:27:05.065012 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.064866    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0db116b4-d971-4966-bb05-db28d077b6fe-trusted-ca\") pod \"image-registry-6dd647c7b7-9255m\" (UID: \"0db116b4-d971-4966-bb05-db28d077b6fe\") " pod="openshift-image-registry/image-registry-6dd647c7b7-9255m"
Oct 31 12:27:05.065012 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.064886    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/dda0c26f-8225-4239-bc27-866b66cbb919-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-gt9rn\" (UID: \"dda0c26f-8225-4239-bc27-866b66cbb919\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gt9rn"
Oct 31 12:27:05.065012 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.064907    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f032909d-bc39-4c9e-99b7-f3d800842119-trusted-ca\") pod \"console-operator-7cd56b8dbc-pcvvm\" (UID: \"f032909d-bc39-4c9e-99b7-f3d800842119\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-pcvvm"
Oct 31 12:27:05.065012 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.064946    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bs5p9\" (UniqueName: \"kubernetes.io/projected/dbbd2d70-8911-4aa5-96cf-3a16dcdf246d-kube-api-access-bs5p9\") pod \"cluster-samples-operator-6ddb6c8b94-tcnrw\" (UID: \"dbbd2d70-8911-4aa5-96cf-3a16dcdf246d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-tcnrw"
Oct 31 12:27:05.065012 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.064968    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqmhd\" (UniqueName: \"kubernetes.io/projected/07714f20-a70e-49e8-8e51-d03a8e00005a-kube-api-access-hqmhd\") pod \"kas-connection-checker-685895c7c9-vvzpg\" (UID: \"07714f20-a70e-49e8-8e51-d03a8e00005a\") " pod="kube-system/kas-connection-checker-685895c7c9-vvzpg"
Oct 31 12:27:05.065323 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.065306    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/556dda7b-d8e6-4558-b5f3-994b01d6a866-stats-auth\") pod \"router-default-659589798c-776bx\" (UID: \"556dda7b-d8e6-4558-b5f3-994b01d6a866\") " pod="openshift-ingress/router-default-659589798c-776bx"
Oct 31 12:27:05.065364 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.065352    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6t4dc\" (UniqueName: \"kubernetes.io/projected/99a606a8-546b-4145-8bae-a95061241069-kube-api-access-6t4dc\") pod \"service-ca-operator-544d749c8c-c789s\" (UID: \"99a606a8-546b-4145-8bae-a95061241069\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-c789s"
Oct 31 12:27:05.065419 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.065392    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/50ad7816-358d-461e-a9bd-fadce6246e8f-nginx-conf\") pod \"networking-console-plugin-86c478bb88-2bfvp\" (UID: \"50ad7816-358d-461e-a9bd-fadce6246e8f\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-2bfvp"
Oct 31 12:27:05.065457 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.065444    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0db116b4-d971-4966-bb05-db28d077b6fe-image-registry-private-configuration\") pod \"image-registry-6dd647c7b7-9255m\" (UID: \"0db116b4-d971-4966-bb05-db28d077b6fe\") " pod="openshift-image-registry/image-registry-6dd647c7b7-9255m"
Oct 31 12:27:05.065535 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.065516    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlgsp\" (UniqueName: \"kubernetes.io/projected/0db116b4-d971-4966-bb05-db28d077b6fe-kube-api-access-nlgsp\") pod \"image-registry-6dd647c7b7-9255m\" (UID: \"0db116b4-d971-4966-bb05-db28d077b6fe\") " pod="openshift-image-registry/image-registry-6dd647c7b7-9255m"
Oct 31 12:27:05.065612 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.065589    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f032909d-bc39-4c9e-99b7-f3d800842119-config\") pod \"console-operator-7cd56b8dbc-pcvvm\" (UID: \"f032909d-bc39-4c9e-99b7-f3d800842119\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-pcvvm"
Oct 31 12:27:05.065673 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.065655    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ddb04b2e-2724-44e9-948a-877a90d8a071-tmp\") pod \"insights-operator-7c4986d449-z879s\" (UID: \"ddb04b2e-2724-44e9-948a-877a90d8a071\") " pod="openshift-insights/insights-operator-7c4986d449-z879s"
Oct 31 12:27:05.065720 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.065697    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f032909d-bc39-4c9e-99b7-f3d800842119-serving-cert\") pod \"console-operator-7cd56b8dbc-pcvvm\" (UID: \"f032909d-bc39-4c9e-99b7-f3d800842119\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-pcvvm"
Oct 31 12:27:05.065770 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.065732    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9083c35a-9739-4f3e-8415-bad1e6433d40-registry-certificates\") pod \"image-registry-6cfdbfcdc4-2rq84\" (UID: \"9083c35a-9739-4f3e-8415-bad1e6433d40\") " pod="openshift-image-registry/image-registry-6cfdbfcdc4-2rq84"
Oct 31 12:27:05.065816 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.065768    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9083c35a-9739-4f3e-8415-bad1e6433d40-trusted-ca\") pod \"image-registry-6cfdbfcdc4-2rq84\" (UID: \"9083c35a-9739-4f3e-8415-bad1e6433d40\") " pod="openshift-image-registry/image-registry-6cfdbfcdc4-2rq84"
Oct 31 12:27:05.065816 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.065799    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9083c35a-9739-4f3e-8415-bad1e6433d40-bound-sa-token\") pod \"image-registry-6cfdbfcdc4-2rq84\" (UID: \"9083c35a-9739-4f3e-8415-bad1e6433d40\") " pod="openshift-image-registry/image-registry-6cfdbfcdc4-2rq84"
Oct 31 12:27:05.065906 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.065832    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/99a606a8-546b-4145-8bae-a95061241069-config\") pod \"service-ca-operator-544d749c8c-c789s\" (UID: \"99a606a8-546b-4145-8bae-a95061241069\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-c789s"
Oct 31 12:27:05.065906 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.065866    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9083c35a-9739-4f3e-8415-bad1e6433d40-registry-tls\") pod \"image-registry-6cfdbfcdc4-2rq84\" (UID: \"9083c35a-9739-4f3e-8415-bad1e6433d40\") " pod="openshift-image-registry/image-registry-6cfdbfcdc4-2rq84"
Oct 31 12:27:05.065906 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.065896    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xwrq\" (UniqueName: \"kubernetes.io/projected/9083c35a-9739-4f3e-8415-bad1e6433d40-kube-api-access-5xwrq\") pod \"image-registry-6cfdbfcdc4-2rq84\" (UID: \"9083c35a-9739-4f3e-8415-bad1e6433d40\") " pod="openshift-image-registry/image-registry-6cfdbfcdc4-2rq84"
Oct 31 12:27:05.066040 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.065927    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rblt\" (UniqueName: \"kubernetes.io/projected/f032909d-bc39-4c9e-99b7-f3d800842119-kube-api-access-9rblt\") pod \"console-operator-7cd56b8dbc-pcvvm\" (UID: \"f032909d-bc39-4c9e-99b7-f3d800842119\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-pcvvm"
Oct 31 12:27:05.066040 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.065959    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0db116b4-d971-4966-bb05-db28d077b6fe-registry-tls\") pod \"image-registry-6dd647c7b7-9255m\" (UID: \"0db116b4-d971-4966-bb05-db28d077b6fe\") " pod="openshift-image-registry/image-registry-6dd647c7b7-9255m"
Oct 31 12:27:05.066040 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.065984    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9083c35a-9739-4f3e-8415-bad1e6433d40-ca-trust-extracted\") pod \"image-registry-6cfdbfcdc4-2rq84\" (UID: \"9083c35a-9739-4f3e-8415-bad1e6433d40\") " pod="openshift-image-registry/image-registry-6cfdbfcdc4-2rq84"
Oct 31 12:27:05.066040 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.066023    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/dbbd2d70-8911-4aa5-96cf-3a16dcdf246d-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-tcnrw\" (UID: \"dbbd2d70-8911-4aa5-96cf-3a16dcdf246d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-tcnrw"
Oct 31 12:27:05.066208 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.066053    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5c8865f9-fdf6-49b9-8b4f-69dbc7787084-config-volume\") pod \"dns-default-r6fvt\" (UID: \"5c8865f9-fdf6-49b9-8b4f-69dbc7787084\") " pod="openshift-dns/dns-default-r6fvt"
Oct 31 12:27:05.066208 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.066084    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/dda0c26f-8225-4239-bc27-866b66cbb919-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-gt9rn\" (UID: \"dda0c26f-8225-4239-bc27-866b66cbb919\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gt9rn"
Oct 31 12:27:05.066208 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.066122    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jsrnd\" (UniqueName: \"kubernetes.io/projected/febbc28e-bbcc-4961-bde8-09944d9ca680-kube-api-access-jsrnd\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-vqsv9\" (UID: \"febbc28e-bbcc-4961-bde8-09944d9ca680\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-vqsv9"
Oct 31 12:27:05.066208 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.066179    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/556dda7b-d8e6-4558-b5f3-994b01d6a866-metrics-certs\") pod \"router-default-659589798c-776bx\" (UID: \"556dda7b-d8e6-4558-b5f3-994b01d6a866\") " pod="openshift-ingress/router-default-659589798c-776bx"
Oct 31 12:27:05.066371 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.066208    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zk92l\" (UniqueName: \"kubernetes.io/projected/ff424288-19ca-4db0-9f16-fed9fb58404a-kube-api-access-zk92l\") pod \"kas-connection-checker-685895c7c9-rrc8k\" (UID: \"ff424288-19ca-4db0-9f16-fed9fb58404a\") " pod="kube-system/kas-connection-checker-685895c7c9-rrc8k"
Oct 31 12:27:05.066371 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.066232    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0db116b4-d971-4966-bb05-db28d077b6fe-installation-pull-secrets\") pod \"image-registry-6dd647c7b7-9255m\" (UID: \"0db116b4-d971-4966-bb05-db28d077b6fe\") " pod="openshift-image-registry/image-registry-6dd647c7b7-9255m"
Oct 31 12:27:05.066371 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.066264    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ddb04b2e-2724-44e9-948a-877a90d8a071-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-z879s\" (UID: \"ddb04b2e-2724-44e9-948a-877a90d8a071\") " pod="openshift-insights/insights-operator-7c4986d449-z879s"
Oct 31 12:27:05.066371 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.066299    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/50ad7816-358d-461e-a9bd-fadce6246e8f-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-2bfvp\" (UID: \"50ad7816-358d-461e-a9bd-fadce6246e8f\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-2bfvp"
Oct 31 12:27:05.066371 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.066329    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9083c35a-9739-4f3e-8415-bad1e6433d40-installation-pull-secrets\") pod \"image-registry-6cfdbfcdc4-2rq84\" (UID: \"9083c35a-9739-4f3e-8415-bad1e6433d40\") " pod="openshift-image-registry/image-registry-6cfdbfcdc4-2rq84"
Oct 31 12:27:05.066656 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.066371    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9g26m\" (UniqueName: \"kubernetes.io/projected/556dda7b-d8e6-4558-b5f3-994b01d6a866-kube-api-access-9g26m\") pod \"router-default-659589798c-776bx\" (UID: \"556dda7b-d8e6-4558-b5f3-994b01d6a866\") " pod="openshift-ingress/router-default-659589798c-776bx"
Oct 31 12:27:05.066656 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.066395    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/891b3c47-ff7c-4a7a-a09b-411361a843f7-cert\") pod \"ingress-canary-rl55s\" (UID: \"891b3c47-ff7c-4a7a-a09b-411361a843f7\") " pod="openshift-ingress-canary/ingress-canary-rl55s"
Oct 31 12:27:05.066656 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.066425    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mnlt\" (UniqueName: \"kubernetes.io/projected/b28ee2bb-0387-4d2d-8ca7-3fef91619c6b-kube-api-access-8mnlt\") pod \"volume-data-source-validator-58c9f5f46d-hf9hr\" (UID: \"b28ee2bb-0387-4d2d-8ca7-3fef91619c6b\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-hf9hr"
Oct 31 12:27:05.066656 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.066456    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0db116b4-d971-4966-bb05-db28d077b6fe-bound-sa-token\") pod \"image-registry-6dd647c7b7-9255m\" (UID: \"0db116b4-d971-4966-bb05-db28d077b6fe\") " pod="openshift-image-registry/image-registry-6dd647c7b7-9255m"
Oct 31 12:27:05.066656 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.066485    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c8865f9-fdf6-49b9-8b4f-69dbc7787084-tmp-dir\") pod \"dns-default-r6fvt\" (UID: \"5c8865f9-fdf6-49b9-8b4f-69dbc7787084\") " pod="openshift-dns/dns-default-r6fvt"
Oct 31 12:27:05.066656 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.066528    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhm56\" (UniqueName: \"kubernetes.io/projected/5c8865f9-fdf6-49b9-8b4f-69dbc7787084-kube-api-access-mhm56\") pod \"dns-default-r6fvt\" (UID: \"5c8865f9-fdf6-49b9-8b4f-69dbc7787084\") " pod="openshift-dns/dns-default-r6fvt"
Oct 31 12:27:05.066656 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.066560    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ddb04b2e-2724-44e9-948a-877a90d8a071-serving-cert\") pod \"insights-operator-7c4986d449-z879s\" (UID: \"ddb04b2e-2724-44e9-948a-877a90d8a071\") " pod="openshift-insights/insights-operator-7c4986d449-z879s"
Oct 31 12:27:05.066656 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.066595    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/ddb04b2e-2724-44e9-948a-877a90d8a071-snapshots\") pod \"insights-operator-7c4986d449-z879s\" (UID: \"ddb04b2e-2724-44e9-948a-877a90d8a071\") " pod="openshift-insights/insights-operator-7c4986d449-z879s"
Oct 31 12:27:05.167557 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.167466    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9083c35a-9739-4f3e-8415-bad1e6433d40-bound-sa-token\") pod \"image-registry-6cfdbfcdc4-2rq84\" (UID: \"9083c35a-9739-4f3e-8415-bad1e6433d40\") " pod="openshift-image-registry/image-registry-6cfdbfcdc4-2rq84"
Oct 31 12:27:05.167557 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.167521    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/99a606a8-546b-4145-8bae-a95061241069-config\") pod \"service-ca-operator-544d749c8c-c789s\" (UID: \"99a606a8-546b-4145-8bae-a95061241069\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-c789s"
Oct 31 12:27:05.167557 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.167543    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9083c35a-9739-4f3e-8415-bad1e6433d40-registry-tls\") pod \"image-registry-6cfdbfcdc4-2rq84\" (UID: \"9083c35a-9739-4f3e-8415-bad1e6433d40\") " pod="openshift-image-registry/image-registry-6cfdbfcdc4-2rq84"
Oct 31 12:27:05.167557 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.167559    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xwrq\" (UniqueName: \"kubernetes.io/projected/9083c35a-9739-4f3e-8415-bad1e6433d40-kube-api-access-5xwrq\") pod \"image-registry-6cfdbfcdc4-2rq84\" (UID: \"9083c35a-9739-4f3e-8415-bad1e6433d40\") " pod="openshift-image-registry/image-registry-6cfdbfcdc4-2rq84"
Oct 31 12:27:05.167859 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.167578    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rblt\" (UniqueName: \"kubernetes.io/projected/f032909d-bc39-4c9e-99b7-f3d800842119-kube-api-access-9rblt\") pod \"console-operator-7cd56b8dbc-pcvvm\" (UID: \"f032909d-bc39-4c9e-99b7-f3d800842119\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-pcvvm"
Oct 31 12:27:05.167859 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.167601    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0db116b4-d971-4966-bb05-db28d077b6fe-registry-tls\") pod \"image-registry-6dd647c7b7-9255m\" (UID: \"0db116b4-d971-4966-bb05-db28d077b6fe\") " pod="openshift-image-registry/image-registry-6dd647c7b7-9255m"
Oct 31 12:27:05.167859 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:27:05.167677    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 31 12:27:05.167859 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:27:05.167693    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6dd647c7b7-9255m: secret "image-registry-tls" not found
Oct 31 12:27:05.167859 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:27:05.167677    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 31 12:27:05.167859 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:27:05.167741    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0db116b4-d971-4966-bb05-db28d077b6fe-registry-tls podName:0db116b4-d971-4966-bb05-db28d077b6fe nodeName:}" failed. No retries permitted until 2026-10-31 12:27:05.667722777 +0000 UTC m=+30.560369906 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0db116b4-d971-4966-bb05-db28d077b6fe-registry-tls") pod "image-registry-6dd647c7b7-9255m" (UID: "0db116b4-d971-4966-bb05-db28d077b6fe") : secret "image-registry-tls" not found
Oct 31 12:27:05.167859 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:27:05.167741    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6cfdbfcdc4-2rq84: secret "image-registry-tls" not found
Oct 31 12:27:05.167859 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.167757    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9083c35a-9739-4f3e-8415-bad1e6433d40-ca-trust-extracted\") pod \"image-registry-6cfdbfcdc4-2rq84\" (UID: \"9083c35a-9739-4f3e-8415-bad1e6433d40\") " pod="openshift-image-registry/image-registry-6cfdbfcdc4-2rq84"
Oct 31 12:27:05.167859 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.167788    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/dbbd2d70-8911-4aa5-96cf-3a16dcdf246d-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-tcnrw\" (UID: \"dbbd2d70-8911-4aa5-96cf-3a16dcdf246d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-tcnrw"
Oct 31 12:27:05.167859 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:27:05.167800    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9083c35a-9739-4f3e-8415-bad1e6433d40-registry-tls podName:9083c35a-9739-4f3e-8415-bad1e6433d40 nodeName:}" failed. No retries permitted until 2026-10-31 12:27:05.667782532 +0000 UTC m=+30.560429661 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9083c35a-9739-4f3e-8415-bad1e6433d40-registry-tls") pod "image-registry-6cfdbfcdc4-2rq84" (UID: "9083c35a-9739-4f3e-8415-bad1e6433d40") : secret "image-registry-tls" not found
Oct 31 12:27:05.167859 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.167834    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5c8865f9-fdf6-49b9-8b4f-69dbc7787084-config-volume\") pod \"dns-default-r6fvt\" (UID: \"5c8865f9-fdf6-49b9-8b4f-69dbc7787084\") " pod="openshift-dns/dns-default-r6fvt"
Oct 31 12:27:05.167859 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:27:05.167850    2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 31 12:27:05.168368 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.167862    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/dda0c26f-8225-4239-bc27-866b66cbb919-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-gt9rn\" (UID: \"dda0c26f-8225-4239-bc27-866b66cbb919\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gt9rn"
Oct 31 12:27:05.168368 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:27:05.167887    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dbbd2d70-8911-4aa5-96cf-3a16dcdf246d-samples-operator-tls podName:dbbd2d70-8911-4aa5-96cf-3a16dcdf246d nodeName:}" failed. No retries permitted until 2026-10-31 12:27:05.667875147 +0000 UTC m=+30.560522295 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/dbbd2d70-8911-4aa5-96cf-3a16dcdf246d-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-tcnrw" (UID: "dbbd2d70-8911-4aa5-96cf-3a16dcdf246d") : secret "samples-operator-tls" not found
Oct 31 12:27:05.168368 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.167918    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsrnd\" (UniqueName: \"kubernetes.io/projected/febbc28e-bbcc-4961-bde8-09944d9ca680-kube-api-access-jsrnd\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-vqsv9\" (UID: \"febbc28e-bbcc-4961-bde8-09944d9ca680\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-vqsv9"
Oct 31 12:27:05.168368 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:27:05.167931    2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 31 12:27:05.168368 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.167959    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/556dda7b-d8e6-4558-b5f3-994b01d6a866-metrics-certs\") pod \"router-default-659589798c-776bx\" (UID: \"556dda7b-d8e6-4558-b5f3-994b01d6a866\") " pod="openshift-ingress/router-default-659589798c-776bx"
Oct 31 12:27:05.168368 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:27:05.167969    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dda0c26f-8225-4239-bc27-866b66cbb919-cluster-monitoring-operator-tls podName:dda0c26f-8225-4239-bc27-866b66cbb919 nodeName:}" failed. No retries permitted until 2026-10-31 12:27:05.667957231 +0000 UTC m=+30.560604376 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/dda0c26f-8225-4239-bc27-866b66cbb919-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-gt9rn" (UID: "dda0c26f-8225-4239-bc27-866b66cbb919") : secret "cluster-monitoring-operator-tls" not found
Oct 31 12:27:05.168368 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.168000    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zk92l\" (UniqueName: \"kubernetes.io/projected/ff424288-19ca-4db0-9f16-fed9fb58404a-kube-api-access-zk92l\") pod \"kas-connection-checker-685895c7c9-rrc8k\" (UID: \"ff424288-19ca-4db0-9f16-fed9fb58404a\") " pod="kube-system/kas-connection-checker-685895c7c9-rrc8k"
Oct 31 12:27:05.168368 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.168030    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0db116b4-d971-4966-bb05-db28d077b6fe-installation-pull-secrets\") pod \"image-registry-6dd647c7b7-9255m\" (UID: \"0db116b4-d971-4966-bb05-db28d077b6fe\") " pod="openshift-image-registry/image-registry-6dd647c7b7-9255m"
Oct 31 12:27:05.168368 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.168056    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ddb04b2e-2724-44e9-948a-877a90d8a071-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-z879s\" (UID: \"ddb04b2e-2724-44e9-948a-877a90d8a071\") " pod="openshift-insights/insights-operator-7c4986d449-z879s"
Oct 31 12:27:05.168368 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.168085    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/50ad7816-358d-461e-a9bd-fadce6246e8f-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-2bfvp\" (UID: \"50ad7816-358d-461e-a9bd-fadce6246e8f\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-2bfvp"
Oct 31 12:27:05.168368 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:27:05.168102    2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 31 12:27:05.168368 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.168109    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9083c35a-9739-4f3e-8415-bad1e6433d40-installation-pull-secrets\") pod \"image-registry-6cfdbfcdc4-2rq84\" (UID: \"9083c35a-9739-4f3e-8415-bad1e6433d40\") " pod="openshift-image-registry/image-registry-6cfdbfcdc4-2rq84"
Oct 31 12:27:05.168368 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:27:05.168140    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/556dda7b-d8e6-4558-b5f3-994b01d6a866-metrics-certs podName:556dda7b-d8e6-4558-b5f3-994b01d6a866 nodeName:}" failed. No retries permitted until 2026-10-31 12:27:05.668128091 +0000 UTC m=+30.560775220 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/556dda7b-d8e6-4558-b5f3-994b01d6a866-metrics-certs") pod "router-default-659589798c-776bx" (UID: "556dda7b-d8e6-4558-b5f3-994b01d6a866") : secret "router-metrics-certs-default" not found
Oct 31 12:27:05.168368 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.168171    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9g26m\" (UniqueName: \"kubernetes.io/projected/556dda7b-d8e6-4558-b5f3-994b01d6a866-kube-api-access-9g26m\") pod \"router-default-659589798c-776bx\" (UID: \"556dda7b-d8e6-4558-b5f3-994b01d6a866\") " pod="openshift-ingress/router-default-659589798c-776bx"
Oct 31 12:27:05.169081 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.168193    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9083c35a-9739-4f3e-8415-bad1e6433d40-ca-trust-extracted\") pod \"image-registry-6cfdbfcdc4-2rq84\" (UID: \"9083c35a-9739-4f3e-8415-bad1e6433d40\") " pod="openshift-image-registry/image-registry-6cfdbfcdc4-2rq84"
Oct 31 12:27:05.169081 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.168199    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/891b3c47-ff7c-4a7a-a09b-411361a843f7-cert\") pod \"ingress-canary-rl55s\" (UID: \"891b3c47-ff7c-4a7a-a09b-411361a843f7\") " pod="openshift-ingress-canary/ingress-canary-rl55s"
Oct 31 12:27:05.169081 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.168238    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mnlt\" (UniqueName: \"kubernetes.io/projected/b28ee2bb-0387-4d2d-8ca7-3fef91619c6b-kube-api-access-8mnlt\") pod \"volume-data-source-validator-58c9f5f46d-hf9hr\" (UID: \"b28ee2bb-0387-4d2d-8ca7-3fef91619c6b\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-hf9hr"
Oct 31 12:27:05.169081 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.168269    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0db116b4-d971-4966-bb05-db28d077b6fe-bound-sa-token\") pod \"image-registry-6dd647c7b7-9255m\" (UID: \"0db116b4-d971-4966-bb05-db28d077b6fe\") " pod="openshift-image-registry/image-registry-6dd647c7b7-9255m"
Oct 31 12:27:05.169081 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.168297    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c8865f9-fdf6-49b9-8b4f-69dbc7787084-tmp-dir\") pod \"dns-default-r6fvt\" (UID: \"5c8865f9-fdf6-49b9-8b4f-69dbc7787084\") " pod="openshift-dns/dns-default-r6fvt"
Oct 31 12:27:05.169081 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.168320    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhm56\" (UniqueName: \"kubernetes.io/projected/5c8865f9-fdf6-49b9-8b4f-69dbc7787084-kube-api-access-mhm56\") pod \"dns-default-r6fvt\" (UID: \"5c8865f9-fdf6-49b9-8b4f-69dbc7787084\") " pod="openshift-dns/dns-default-r6fvt"
Oct 31 12:27:05.169081 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.168337    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/99a606a8-546b-4145-8bae-a95061241069-config\") pod \"service-ca-operator-544d749c8c-c789s\" (UID: \"99a606a8-546b-4145-8bae-a95061241069\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-c789s"
Oct 31 12:27:05.169081 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.168348    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ddb04b2e-2724-44e9-948a-877a90d8a071-serving-cert\") pod \"insights-operator-7c4986d449-z879s\" (UID: \"ddb04b2e-2724-44e9-948a-877a90d8a071\") " pod="openshift-insights/insights-operator-7c4986d449-z879s"
Oct 31 12:27:05.169081 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.168373    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/ddb04b2e-2724-44e9-948a-877a90d8a071-snapshots\") pod \"insights-operator-7c4986d449-z879s\" (UID: \"ddb04b2e-2724-44e9-948a-877a90d8a071\") " pod="openshift-insights/insights-operator-7c4986d449-z879s"
Oct 31 12:27:05.169081 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.168411    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9083c35a-9739-4f3e-8415-bad1e6433d40-image-registry-private-configuration\") pod \"image-registry-6cfdbfcdc4-2rq84\" (UID: \"9083c35a-9739-4f3e-8415-bad1e6433d40\") " pod="openshift-image-registry/image-registry-6cfdbfcdc4-2rq84"
Oct 31 12:27:05.169081 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:27:05.168428    2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 31 12:27:05.169081 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.168432    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5c8865f9-fdf6-49b9-8b4f-69dbc7787084-config-volume\") pod \"dns-default-r6fvt\" (UID: \"5c8865f9-fdf6-49b9-8b4f-69dbc7787084\") " pod="openshift-dns/dns-default-r6fvt"
Oct 31 12:27:05.169081 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:27:05.168505    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50ad7816-358d-461e-a9bd-fadce6246e8f-networking-console-plugin-cert podName:50ad7816-358d-461e-a9bd-fadce6246e8f nodeName:}" failed. No retries permitted until 2026-10-31 12:27:05.668469921 +0000 UTC m=+30.561117067 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/50ad7816-358d-461e-a9bd-fadce6246e8f-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-2bfvp" (UID: "50ad7816-358d-461e-a9bd-fadce6246e8f") : secret "networking-console-plugin-cert" not found
Oct 31 12:27:05.169081 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.168437    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/556dda7b-d8e6-4558-b5f3-994b01d6a866-service-ca-bundle\") pod \"router-default-659589798c-776bx\" (UID: \"556dda7b-d8e6-4558-b5f3-994b01d6a866\") " pod="openshift-ingress/router-default-659589798c-776bx"
Oct 31 12:27:05.169081 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.168553    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5c8865f9-fdf6-49b9-8b4f-69dbc7787084-metrics-tls\") pod \"dns-default-r6fvt\" (UID: \"5c8865f9-fdf6-49b9-8b4f-69dbc7787084\") " pod="openshift-dns/dns-default-r6fvt"
Oct 31 12:27:05.169794 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:27:05.168572    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/556dda7b-d8e6-4558-b5f3-994b01d6a866-service-ca-bundle podName:556dda7b-d8e6-4558-b5f3-994b01d6a866 nodeName:}" failed. No retries permitted until 2026-10-31 12:27:05.668557813 +0000 UTC m=+30.561204948 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/556dda7b-d8e6-4558-b5f3-994b01d6a866-service-ca-bundle") pod "router-default-659589798c-776bx" (UID: "556dda7b-d8e6-4558-b5f3-994b01d6a866") : configmap references non-existent config key: service-ca.crt
Oct 31 12:27:05.169794 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:27:05.168604    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 31 12:27:05.169794 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:27:05.168632    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5c8865f9-fdf6-49b9-8b4f-69dbc7787084-metrics-tls podName:5c8865f9-fdf6-49b9-8b4f-69dbc7787084 nodeName:}" failed. No retries permitted until 2026-10-31 12:27:05.668624054 +0000 UTC m=+30.561271187 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5c8865f9-fdf6-49b9-8b4f-69dbc7787084-metrics-tls") pod "dns-default-r6fvt" (UID: "5c8865f9-fdf6-49b9-8b4f-69dbc7787084") : secret "dns-default-metrics-tls" not found
Oct 31 12:27:05.169794 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.168603    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvqpw\" (UniqueName: \"kubernetes.io/projected/25691fa3-7017-429a-b94c-c2dde4d29e55-kube-api-access-gvqpw\") pod \"network-check-source-6f78d59f99-m9xxh\" (UID: \"25691fa3-7017-429a-b94c-c2dde4d29e55\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-m9xxh"
Oct 31 12:27:05.169794 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.168658    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0db116b4-d971-4966-bb05-db28d077b6fe-ca-trust-extracted\") pod \"image-registry-6dd647c7b7-9255m\" (UID: \"0db116b4-d971-4966-bb05-db28d077b6fe\") " pod="openshift-image-registry/image-registry-6dd647c7b7-9255m"
Oct 31 12:27:05.169794 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.168691    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/dbbd2d70-8911-4aa5-96cf-3a16dcdf246d-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-tcnrw\" (UID: \"dbbd2d70-8911-4aa5-96cf-3a16dcdf246d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-tcnrw"
Oct 31 12:27:05.169794 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.168710    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/556dda7b-d8e6-4558-b5f3-994b01d6a866-default-certificate\") pod \"router-default-659589798c-776bx\" (UID: \"556dda7b-d8e6-4558-b5f3-994b01d6a866\") " pod="openshift-ingress/router-default-659589798c-776bx"
Oct 31 12:27:05.169794 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.168729    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/febbc28e-bbcc-4961-bde8-09944d9ca680-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-vqsv9\" (UID: \"febbc28e-bbcc-4961-bde8-09944d9ca680\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-vqsv9"
Oct 31 12:27:05.169794 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.168755    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0db116b4-d971-4966-bb05-db28d077b6fe-registry-certificates\") pod \"image-registry-6dd647c7b7-9255m\" (UID: \"0db116b4-d971-4966-bb05-db28d077b6fe\") " pod="openshift-image-registry/image-registry-6dd647c7b7-9255m"
Oct 31 12:27:05.169794 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.168785    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jw4wt\" (UniqueName: \"kubernetes.io/projected/dda0c26f-8225-4239-bc27-866b66cbb919-kube-api-access-jw4wt\") pod \"cluster-monitoring-operator-884576c57-gt9rn\" (UID: \"dda0c26f-8225-4239-bc27-866b66cbb919\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gt9rn"
Oct 31 12:27:05.169794 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.168816    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/99a606a8-546b-4145-8bae-a95061241069-serving-cert\") pod \"service-ca-operator-544d749c8c-c789s\" (UID: \"99a606a8-546b-4145-8bae-a95061241069\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-c789s"
Oct 31 12:27:05.169794 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.168842    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wq6d\" (UniqueName: \"kubernetes.io/projected/ddb04b2e-2724-44e9-948a-877a90d8a071-kube-api-access-5wq6d\") pod \"insights-operator-7c4986d449-z879s\" (UID: \"ddb04b2e-2724-44e9-948a-877a90d8a071\") " pod="openshift-insights/insights-operator-7c4986d449-z879s"
Oct 31 12:27:05.169794 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.168870    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrpck\" (UniqueName: \"kubernetes.io/projected/891b3c47-ff7c-4a7a-a09b-411361a843f7-kube-api-access-mrpck\") pod \"ingress-canary-rl55s\" (UID: \"891b3c47-ff7c-4a7a-a09b-411361a843f7\") " pod="openshift-ingress-canary/ingress-canary-rl55s"
Oct 31 12:27:05.169794 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.168892    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvzh5\" (UniqueName: \"kubernetes.io/projected/613125d4-fbb4-4b43-9c47-24e039bd4697-kube-api-access-mvzh5\") pod \"kas-connection-checker-685895c7c9-k64bz\" (UID: \"613125d4-fbb4-4b43-9c47-24e039bd4697\") " pod="kube-system/kas-connection-checker-685895c7c9-k64bz"
Oct 31 12:27:05.169794 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.168914    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ddb04b2e-2724-44e9-948a-877a90d8a071-service-ca-bundle\") pod \"insights-operator-7c4986d449-z879s\" (UID: \"ddb04b2e-2724-44e9-948a-877a90d8a071\") " pod="openshift-insights/insights-operator-7c4986d449-z879s"
Oct 31 12:27:05.170522 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.168939    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/febbc28e-bbcc-4961-bde8-09944d9ca680-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-vqsv9\" (UID: \"febbc28e-bbcc-4961-bde8-09944d9ca680\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-vqsv9"
Oct 31 12:27:05.170522 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.169002    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0db116b4-d971-4966-bb05-db28d077b6fe-trusted-ca\") pod \"image-registry-6dd647c7b7-9255m\" (UID: \"0db116b4-d971-4966-bb05-db28d077b6fe\") " pod="openshift-image-registry/image-registry-6dd647c7b7-9255m"
Oct 31 12:27:05.170522 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.169024    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/dda0c26f-8225-4239-bc27-866b66cbb919-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-gt9rn\" (UID: \"dda0c26f-8225-4239-bc27-866b66cbb919\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gt9rn"
Oct 31 12:27:05.170522 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.169047    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f032909d-bc39-4c9e-99b7-f3d800842119-trusted-ca\") pod \"console-operator-7cd56b8dbc-pcvvm\" (UID: \"f032909d-bc39-4c9e-99b7-f3d800842119\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-pcvvm"
Oct 31 12:27:05.170522 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.169084    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bs5p9\" (UniqueName: \"kubernetes.io/projected/dbbd2d70-8911-4aa5-96cf-3a16dcdf246d-kube-api-access-bs5p9\") pod \"cluster-samples-operator-6ddb6c8b94-tcnrw\" (UID: \"dbbd2d70-8911-4aa5-96cf-3a16dcdf246d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-tcnrw"
Oct 31 12:27:05.170522 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.169109    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqmhd\" (UniqueName: \"kubernetes.io/projected/07714f20-a70e-49e8-8e51-d03a8e00005a-kube-api-access-hqmhd\") pod \"kas-connection-checker-685895c7c9-vvzpg\" (UID: \"07714f20-a70e-49e8-8e51-d03a8e00005a\") " pod="kube-system/kas-connection-checker-685895c7c9-vvzpg"
Oct 31 12:27:05.170522 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.169131    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/556dda7b-d8e6-4558-b5f3-994b01d6a866-stats-auth\") pod \"router-default-659589798c-776bx\" (UID: \"556dda7b-d8e6-4558-b5f3-994b01d6a866\") " pod="openshift-ingress/router-default-659589798c-776bx"
Oct 31 12:27:05.170522 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.169158    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6t4dc\" (UniqueName: \"kubernetes.io/projected/99a606a8-546b-4145-8bae-a95061241069-kube-api-access-6t4dc\") pod \"service-ca-operator-544d749c8c-c789s\" (UID: \"99a606a8-546b-4145-8bae-a95061241069\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-c789s"
Oct 31 12:27:05.170522 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.169155    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c8865f9-fdf6-49b9-8b4f-69dbc7787084-tmp-dir\") pod \"dns-default-r6fvt\" (UID: \"5c8865f9-fdf6-49b9-8b4f-69dbc7787084\") " pod="openshift-dns/dns-default-r6fvt"
Oct 31 12:27:05.170522 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.169654    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0db116b4-d971-4966-bb05-db28d077b6fe-ca-trust-extracted\") pod \"image-registry-6dd647c7b7-9255m\" (UID: \"0db116b4-d971-4966-bb05-db28d077b6fe\") " pod="openshift-image-registry/image-registry-6dd647c7b7-9255m"
Oct 31 12:27:05.170522 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.169713    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ddb04b2e-2724-44e9-948a-877a90d8a071-service-ca-bundle\") pod \"insights-operator-7c4986d449-z879s\" (UID: \"ddb04b2e-2724-44e9-948a-877a90d8a071\") " pod="openshift-insights/insights-operator-7c4986d449-z879s"
Oct 31 12:27:05.170522 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.170206    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/febbc28e-bbcc-4961-bde8-09944d9ca680-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-vqsv9\" (UID: \"febbc28e-bbcc-4961-bde8-09944d9ca680\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-vqsv9"
Oct 31 12:27:05.170522 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.170298    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/dbbd2d70-8911-4aa5-96cf-3a16dcdf246d-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-tcnrw\" (UID: \"dbbd2d70-8911-4aa5-96cf-3a16dcdf246d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-tcnrw"
Oct 31 12:27:05.170522 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.170316    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ddb04b2e-2724-44e9-948a-877a90d8a071-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-z879s\" (UID: \"ddb04b2e-2724-44e9-948a-877a90d8a071\") " pod="openshift-insights/insights-operator-7c4986d449-z879s"
Oct 31 12:27:05.171182 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.170837    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/ddb04b2e-2724-44e9-948a-877a90d8a071-snapshots\") pod \"insights-operator-7c4986d449-z879s\" (UID: \"ddb04b2e-2724-44e9-948a-877a90d8a071\") " pod="openshift-insights/insights-operator-7c4986d449-z879s"
Oct 31 12:27:05.171182 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.171034    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0db116b4-d971-4966-bb05-db28d077b6fe-trusted-ca\") pod \"image-registry-6dd647c7b7-9255m\" (UID: \"0db116b4-d971-4966-bb05-db28d077b6fe\") " pod="openshift-image-registry/image-registry-6dd647c7b7-9255m"
Oct 31 12:27:05.173398 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.171513    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f032909d-bc39-4c9e-99b7-f3d800842119-trusted-ca\") pod \"console-operator-7cd56b8dbc-pcvvm\" (UID: \"f032909d-bc39-4c9e-99b7-f3d800842119\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-pcvvm"
Oct 31 12:27:05.173398 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.172111    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0db116b4-d971-4966-bb05-db28d077b6fe-registry-certificates\") pod \"image-registry-6dd647c7b7-9255m\" (UID: \"0db116b4-d971-4966-bb05-db28d077b6fe\") " pod="openshift-image-registry/image-registry-6dd647c7b7-9255m"
Oct 31 12:27:05.173398 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.172821    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/dda0c26f-8225-4239-bc27-866b66cbb919-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-gt9rn\" (UID: \"dda0c26f-8225-4239-bc27-866b66cbb919\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gt9rn"
Oct 31 12:27:05.173398 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.172866    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/50ad7816-358d-461e-a9bd-fadce6246e8f-nginx-conf\") pod \"networking-console-plugin-86c478bb88-2bfvp\" (UID: \"50ad7816-358d-461e-a9bd-fadce6246e8f\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-2bfvp"
Oct 31 12:27:05.173398 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.172895    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0db116b4-d971-4966-bb05-db28d077b6fe-image-registry-private-configuration\") pod \"image-registry-6dd647c7b7-9255m\" (UID: \"0db116b4-d971-4966-bb05-db28d077b6fe\") " pod="openshift-image-registry/image-registry-6dd647c7b7-9255m"
Oct 31 12:27:05.173398 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.172921    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlgsp\" (UniqueName: \"kubernetes.io/projected/0db116b4-d971-4966-bb05-db28d077b6fe-kube-api-access-nlgsp\") pod \"image-registry-6dd647c7b7-9255m\" (UID: \"0db116b4-d971-4966-bb05-db28d077b6fe\") " pod="openshift-image-registry/image-registry-6dd647c7b7-9255m"
Oct 31 12:27:05.173398 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.172946    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f032909d-bc39-4c9e-99b7-f3d800842119-config\") pod \"console-operator-7cd56b8dbc-pcvvm\" (UID: \"f032909d-bc39-4c9e-99b7-f3d800842119\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-pcvvm"
Oct 31 12:27:05.173398 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.172972    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ddb04b2e-2724-44e9-948a-877a90d8a071-tmp\") pod \"insights-operator-7c4986d449-z879s\" (UID: \"ddb04b2e-2724-44e9-948a-877a90d8a071\") " pod="openshift-insights/insights-operator-7c4986d449-z879s"
Oct 31 12:27:05.173398 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.172997    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f032909d-bc39-4c9e-99b7-f3d800842119-serving-cert\") pod \"console-operator-7cd56b8dbc-pcvvm\" (UID: \"f032909d-bc39-4c9e-99b7-f3d800842119\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-pcvvm"
Oct 31 12:27:05.173398 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.173028    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9083c35a-9739-4f3e-8415-bad1e6433d40-registry-certificates\") pod \"image-registry-6cfdbfcdc4-2rq84\" (UID: \"9083c35a-9739-4f3e-8415-bad1e6433d40\") " pod="openshift-image-registry/image-registry-6cfdbfcdc4-2rq84"
Oct 31 12:27:05.173398 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.173055    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9083c35a-9739-4f3e-8415-bad1e6433d40-trusted-ca\") pod \"image-registry-6cfdbfcdc4-2rq84\" (UID: \"9083c35a-9739-4f3e-8415-bad1e6433d40\") " pod="openshift-image-registry/image-registry-6cfdbfcdc4-2rq84"
Oct 31 12:27:05.173398 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.173157    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ddb04b2e-2724-44e9-948a-877a90d8a071-serving-cert\") pod \"insights-operator-7c4986d449-z879s\" (UID: \"ddb04b2e-2724-44e9-948a-877a90d8a071\") " pod="openshift-insights/insights-operator-7c4986d449-z879s"
Oct 31 12:27:05.173398 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.173374    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/891b3c47-ff7c-4a7a-a09b-411361a843f7-cert\") pod \"ingress-canary-rl55s\" (UID: \"891b3c47-ff7c-4a7a-a09b-411361a843f7\") " pod="openshift-ingress-canary/ingress-canary-rl55s"
Oct 31 12:27:05.177075 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.173744    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f032909d-bc39-4c9e-99b7-f3d800842119-config\") pod \"console-operator-7cd56b8dbc-pcvvm\" (UID: \"f032909d-bc39-4c9e-99b7-f3d800842119\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-pcvvm"
Oct 31 12:27:05.177075 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.173814    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9083c35a-9739-4f3e-8415-bad1e6433d40-installation-pull-secrets\") pod \"image-registry-6cfdbfcdc4-2rq84\" (UID: \"9083c35a-9739-4f3e-8415-bad1e6433d40\") " pod="openshift-image-registry/image-registry-6cfdbfcdc4-2rq84"
Oct 31 12:27:05.177075 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.173968    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9083c35a-9739-4f3e-8415-bad1e6433d40-trusted-ca\") pod \"image-registry-6cfdbfcdc4-2rq84\" (UID: \"9083c35a-9739-4f3e-8415-bad1e6433d40\") " pod="openshift-image-registry/image-registry-6cfdbfcdc4-2rq84"
Oct 31 12:27:05.177075 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.173994    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/99a606a8-546b-4145-8bae-a95061241069-serving-cert\") pod \"service-ca-operator-544d749c8c-c789s\" (UID: \"99a606a8-546b-4145-8bae-a95061241069\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-c789s"
Oct 31 12:27:05.177075 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.174190    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9083c35a-9739-4f3e-8415-bad1e6433d40-image-registry-private-configuration\") pod \"image-registry-6cfdbfcdc4-2rq84\" (UID: \"9083c35a-9739-4f3e-8415-bad1e6433d40\") " pod="openshift-image-registry/image-registry-6cfdbfcdc4-2rq84"
Oct 31 12:27:05.177075 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.174222    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ddb04b2e-2724-44e9-948a-877a90d8a071-tmp\") pod \"insights-operator-7c4986d449-z879s\" (UID: \"ddb04b2e-2724-44e9-948a-877a90d8a071\") " pod="openshift-insights/insights-operator-7c4986d449-z879s"
Oct 31 12:27:05.177075 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.174328    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/50ad7816-358d-461e-a9bd-fadce6246e8f-nginx-conf\") pod \"networking-console-plugin-86c478bb88-2bfvp\" (UID: \"50ad7816-358d-461e-a9bd-fadce6246e8f\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-2bfvp"
Oct 31 12:27:05.177075 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.174708    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9083c35a-9739-4f3e-8415-bad1e6433d40-registry-certificates\") pod \"image-registry-6cfdbfcdc4-2rq84\" (UID: \"9083c35a-9739-4f3e-8415-bad1e6433d40\") " pod="openshift-image-registry/image-registry-6cfdbfcdc4-2rq84"
Oct 31 12:27:05.177075 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.175451    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0db116b4-d971-4966-bb05-db28d077b6fe-installation-pull-secrets\") pod \"image-registry-6dd647c7b7-9255m\" (UID: \"0db116b4-d971-4966-bb05-db28d077b6fe\") " pod="openshift-image-registry/image-registry-6dd647c7b7-9255m"
Oct 31 12:27:05.177075 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.175649    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/febbc28e-bbcc-4961-bde8-09944d9ca680-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-vqsv9\" (UID: \"febbc28e-bbcc-4961-bde8-09944d9ca680\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-vqsv9"
Oct 31 12:27:05.177075 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.176607    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/556dda7b-d8e6-4558-b5f3-994b01d6a866-stats-auth\") pod \"router-default-659589798c-776bx\" (UID: \"556dda7b-d8e6-4558-b5f3-994b01d6a866\") " pod="openshift-ingress/router-default-659589798c-776bx"
Oct 31 12:27:05.177075 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.176754    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0db116b4-d971-4966-bb05-db28d077b6fe-image-registry-private-configuration\") pod \"image-registry-6dd647c7b7-9255m\" (UID: \"0db116b4-d971-4966-bb05-db28d077b6fe\") " pod="openshift-image-registry/image-registry-6dd647c7b7-9255m"
Oct 31 12:27:05.177075 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.176959    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/556dda7b-d8e6-4558-b5f3-994b01d6a866-default-certificate\") pod \"router-default-659589798c-776bx\" (UID: \"556dda7b-d8e6-4558-b5f3-994b01d6a866\") " pod="openshift-ingress/router-default-659589798c-776bx"
Oct 31 12:27:05.177664 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.177646    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f032909d-bc39-4c9e-99b7-f3d800842119-serving-cert\") pod \"console-operator-7cd56b8dbc-pcvvm\" (UID: \"f032909d-bc39-4c9e-99b7-f3d800842119\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-pcvvm"
Oct 31 12:27:05.184098 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.184016    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsrnd\" (UniqueName: \"kubernetes.io/projected/febbc28e-bbcc-4961-bde8-09944d9ca680-kube-api-access-jsrnd\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-vqsv9\" (UID: \"febbc28e-bbcc-4961-bde8-09944d9ca680\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-vqsv9"
Oct 31 12:27:05.184845 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.184579    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9083c35a-9739-4f3e-8415-bad1e6433d40-bound-sa-token\") pod \"image-registry-6cfdbfcdc4-2rq84\" (UID: \"9083c35a-9739-4f3e-8415-bad1e6433d40\") " pod="openshift-image-registry/image-registry-6cfdbfcdc4-2rq84"
Oct 31 12:27:05.185031 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.185005    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvzh5\" (UniqueName: \"kubernetes.io/projected/613125d4-fbb4-4b43-9c47-24e039bd4697-kube-api-access-mvzh5\") pod \"kas-connection-checker-685895c7c9-k64bz\" (UID: \"613125d4-fbb4-4b43-9c47-24e039bd4697\") " pod="kube-system/kas-connection-checker-685895c7c9-k64bz"
Oct 31 12:27:05.185637 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.185610    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mnlt\" (UniqueName: \"kubernetes.io/projected/b28ee2bb-0387-4d2d-8ca7-3fef91619c6b-kube-api-access-8mnlt\") pod \"volume-data-source-validator-58c9f5f46d-hf9hr\" (UID: \"b28ee2bb-0387-4d2d-8ca7-3fef91619c6b\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-hf9hr"
Oct 31 12:27:05.186623 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.186600    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvqpw\" (UniqueName: \"kubernetes.io/projected/25691fa3-7017-429a-b94c-c2dde4d29e55-kube-api-access-gvqpw\") pod \"network-check-source-6f78d59f99-m9xxh\" (UID: \"25691fa3-7017-429a-b94c-c2dde4d29e55\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-m9xxh"
Oct 31 12:27:05.186772 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.186663    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xwrq\" (UniqueName: \"kubernetes.io/projected/9083c35a-9739-4f3e-8415-bad1e6433d40-kube-api-access-5xwrq\") pod \"image-registry-6cfdbfcdc4-2rq84\" (UID: \"9083c35a-9739-4f3e-8415-bad1e6433d40\") " pod="openshift-image-registry/image-registry-6cfdbfcdc4-2rq84"
Oct 31 12:27:05.186970 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.186924    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rblt\" (UniqueName: \"kubernetes.io/projected/f032909d-bc39-4c9e-99b7-f3d800842119-kube-api-access-9rblt\") pod \"console-operator-7cd56b8dbc-pcvvm\" (UID: \"f032909d-bc39-4c9e-99b7-f3d800842119\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-pcvvm"
Oct 31 12:27:05.190265 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.190241    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9g26m\" (UniqueName: \"kubernetes.io/projected/556dda7b-d8e6-4558-b5f3-994b01d6a866-kube-api-access-9g26m\") pod \"router-default-659589798c-776bx\" (UID: \"556dda7b-d8e6-4558-b5f3-994b01d6a866\") " pod="openshift-ingress/router-default-659589798c-776bx"
Oct 31 12:27:05.191846 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.191157    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqmhd\" (UniqueName: \"kubernetes.io/projected/07714f20-a70e-49e8-8e51-d03a8e00005a-kube-api-access-hqmhd\") pod \"kas-connection-checker-685895c7c9-vvzpg\" (UID: \"07714f20-a70e-49e8-8e51-d03a8e00005a\") " pod="kube-system/kas-connection-checker-685895c7c9-vvzpg"
Oct 31 12:27:05.191846 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.191169    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6t4dc\" (UniqueName: \"kubernetes.io/projected/99a606a8-546b-4145-8bae-a95061241069-kube-api-access-6t4dc\") pod \"service-ca-operator-544d749c8c-c789s\" (UID: \"99a606a8-546b-4145-8bae-a95061241069\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-c789s"
Oct 31 12:27:05.191846 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.191184    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0db116b4-d971-4966-bb05-db28d077b6fe-bound-sa-token\") pod \"image-registry-6dd647c7b7-9255m\" (UID: \"0db116b4-d971-4966-bb05-db28d077b6fe\") " pod="openshift-image-registry/image-registry-6dd647c7b7-9255m"
Oct 31 12:27:05.191846 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.191283    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bs5p9\" (UniqueName: \"kubernetes.io/projected/dbbd2d70-8911-4aa5-96cf-3a16dcdf246d-kube-api-access-bs5p9\") pod \"cluster-samples-operator-6ddb6c8b94-tcnrw\" (UID: \"dbbd2d70-8911-4aa5-96cf-3a16dcdf246d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-tcnrw"
Oct 31 12:27:05.191846 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.191286    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jw4wt\" (UniqueName: \"kubernetes.io/projected/dda0c26f-8225-4239-bc27-866b66cbb919-kube-api-access-jw4wt\") pod \"cluster-monitoring-operator-884576c57-gt9rn\" (UID: \"dda0c26f-8225-4239-bc27-866b66cbb919\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gt9rn"
Oct 31 12:27:05.191846 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.191611    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zk92l\" (UniqueName: \"kubernetes.io/projected/ff424288-19ca-4db0-9f16-fed9fb58404a-kube-api-access-zk92l\") pod \"kas-connection-checker-685895c7c9-rrc8k\" (UID: \"ff424288-19ca-4db0-9f16-fed9fb58404a\") " pod="kube-system/kas-connection-checker-685895c7c9-rrc8k"
Oct 31 12:27:05.191846 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.191750    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlgsp\" (UniqueName: \"kubernetes.io/projected/0db116b4-d971-4966-bb05-db28d077b6fe-kube-api-access-nlgsp\") pod \"image-registry-6dd647c7b7-9255m\" (UID: \"0db116b4-d971-4966-bb05-db28d077b6fe\") " pod="openshift-image-registry/image-registry-6dd647c7b7-9255m"
Oct 31 12:27:05.192245 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.192098    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrpck\" (UniqueName: \"kubernetes.io/projected/891b3c47-ff7c-4a7a-a09b-411361a843f7-kube-api-access-mrpck\") pod \"ingress-canary-rl55s\" (UID: \"891b3c47-ff7c-4a7a-a09b-411361a843f7\") " pod="openshift-ingress-canary/ingress-canary-rl55s"
Oct 31 12:27:05.192245 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.192219    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhm56\" (UniqueName: \"kubernetes.io/projected/5c8865f9-fdf6-49b9-8b4f-69dbc7787084-kube-api-access-mhm56\") pod \"dns-default-r6fvt\" (UID: \"5c8865f9-fdf6-49b9-8b4f-69dbc7787084\") " pod="openshift-dns/dns-default-r6fvt"
Oct 31 12:27:05.192986 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.192964    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wq6d\" (UniqueName: \"kubernetes.io/projected/ddb04b2e-2724-44e9-948a-877a90d8a071-kube-api-access-5wq6d\") pod \"insights-operator-7c4986d449-z879s\" (UID: \"ddb04b2e-2724-44e9-948a-877a90d8a071\") " pod="openshift-insights/insights-operator-7c4986d449-z879s"
Oct 31 12:27:05.228160 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.228131    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-vvzpg"
Oct 31 12:27:05.246318 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.246294    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-k64bz"
Oct 31 12:27:05.267327 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.267288    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-rrc8k"
Oct 31 12:27:05.287824 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.287668    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-z879s"
Oct 31 12:27:05.300569 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.299625    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-vqsv9"
Oct 31 12:27:05.307628 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.306683    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-hf9hr"
Oct 31 12:27:05.310979 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.310665    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-pcvvm"
Oct 31 12:27:05.317261 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.317192    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-c789s"
Oct 31 12:27:05.330434 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.330348    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-m9xxh"
Oct 31 12:27:05.348645 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.340691    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-rl55s"
Oct 31 12:27:05.523434 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.523406    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-vvzpg"]
Oct 31 12:27:05.526007 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.525985    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-k64bz"]
Oct 31 12:27:05.526858 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:27:05.526836    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07714f20_a70e_49e8_8e51_d03a8e00005a.slice/crio-8a59f3fca9f665e623be85a15c0d77fe56925fd62aae37db535bb9a23c596a6f WatchSource:0}: Error finding container 8a59f3fca9f665e623be85a15c0d77fe56925fd62aae37db535bb9a23c596a6f: Status 404 returned error can't find the container with id 8a59f3fca9f665e623be85a15c0d77fe56925fd62aae37db535bb9a23c596a6f
Oct 31 12:27:05.528040 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:27:05.528013    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod613125d4_fbb4_4b43_9c47_24e039bd4697.slice/crio-5a4af82d83c4e2d729012b35f36754f633c7d8edae532b658e217176aeed8416 WatchSource:0}: Error finding container 5a4af82d83c4e2d729012b35f36754f633c7d8edae532b658e217176aeed8416: Status 404 returned error can't find the container with id 5a4af82d83c4e2d729012b35f36754f633c7d8edae532b658e217176aeed8416
Oct 31 12:27:05.590423 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.590388    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-rrc8k"]
Oct 31 12:27:05.595974 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.595725    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-z879s"]
Oct 31 12:27:05.597597 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:27:05.597575    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff424288_19ca_4db0_9f16_fed9fb58404a.slice/crio-432c0f75ce4f14ee6e21e05374e78fc51fdb2f101584925475c6900c584e1409 WatchSource:0}: Error finding container 432c0f75ce4f14ee6e21e05374e78fc51fdb2f101584925475c6900c584e1409: Status 404 returned error can't find the container with id 432c0f75ce4f14ee6e21e05374e78fc51fdb2f101584925475c6900c584e1409
Oct 31 12:27:05.602904 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:27:05.602802    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddb04b2e_2724_44e9_948a_877a90d8a071.slice/crio-c1849c803f134d969c39443fe4e01b9f3a600259e4ff5749f6d3ca25a2a168b1 WatchSource:0}: Error finding container c1849c803f134d969c39443fe4e01b9f3a600259e4ff5749f6d3ca25a2a168b1: Status 404 returned error can't find the container with id c1849c803f134d969c39443fe4e01b9f3a600259e4ff5749f6d3ca25a2a168b1
Oct 31 12:27:05.635173 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.635143    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-hf9hr"]
Oct 31 12:27:05.638805 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:27:05.638781    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb28ee2bb_0387_4d2d_8ca7_3fef91619c6b.slice/crio-a130b0e336cef0401531b357c6118d6c3ec61ce95a7b67c9eda4cf43d7a4f443 WatchSource:0}: Error finding container a130b0e336cef0401531b357c6118d6c3ec61ce95a7b67c9eda4cf43d7a4f443: Status 404 returned error can't find the container with id a130b0e336cef0401531b357c6118d6c3ec61ce95a7b67c9eda4cf43d7a4f443
Oct 31 12:27:05.653525 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.651156    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-vqsv9"]
Oct 31 12:27:05.665806 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:27:05.665680    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfebbc28e_bbcc_4961_bde8_09944d9ca680.slice/crio-9ec8c5086bd55eeaa1cf3cd8ee3146133e8915c24281d20fe7de74ac684c8429 WatchSource:0}: Error finding container 9ec8c5086bd55eeaa1cf3cd8ee3146133e8915c24281d20fe7de74ac684c8429: Status 404 returned error can't find the container with id 9ec8c5086bd55eeaa1cf3cd8ee3146133e8915c24281d20fe7de74ac684c8429
Oct 31 12:27:05.678891 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.678860    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/556dda7b-d8e6-4558-b5f3-994b01d6a866-service-ca-bundle\") pod \"router-default-659589798c-776bx\" (UID: \"556dda7b-d8e6-4558-b5f3-994b01d6a866\") " pod="openshift-ingress/router-default-659589798c-776bx"
Oct 31 12:27:05.679026 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.678895    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5c8865f9-fdf6-49b9-8b4f-69dbc7787084-metrics-tls\") pod \"dns-default-r6fvt\" (UID: \"5c8865f9-fdf6-49b9-8b4f-69dbc7787084\") " pod="openshift-dns/dns-default-r6fvt"
Oct 31 12:27:05.679026 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.678947    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9083c35a-9739-4f3e-8415-bad1e6433d40-registry-tls\") pod \"image-registry-6cfdbfcdc4-2rq84\" (UID: \"9083c35a-9739-4f3e-8415-bad1e6433d40\") " pod="openshift-image-registry/image-registry-6cfdbfcdc4-2rq84"
Oct 31 12:27:05.679026 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.678964    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0db116b4-d971-4966-bb05-db28d077b6fe-registry-tls\") pod \"image-registry-6dd647c7b7-9255m\" (UID: \"0db116b4-d971-4966-bb05-db28d077b6fe\") " pod="openshift-image-registry/image-registry-6dd647c7b7-9255m"
Oct 31 12:27:05.679026 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.678983    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/dbbd2d70-8911-4aa5-96cf-3a16dcdf246d-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-tcnrw\" (UID: \"dbbd2d70-8911-4aa5-96cf-3a16dcdf246d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-tcnrw"
Oct 31 12:27:05.679026 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.679001    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/dda0c26f-8225-4239-bc27-866b66cbb919-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-gt9rn\" (UID: \"dda0c26f-8225-4239-bc27-866b66cbb919\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gt9rn"
Oct 31 12:27:05.679026 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.679026    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/556dda7b-d8e6-4558-b5f3-994b01d6a866-metrics-certs\") pod \"router-default-659589798c-776bx\" (UID: \"556dda7b-d8e6-4558-b5f3-994b01d6a866\") " pod="openshift-ingress/router-default-659589798c-776bx"
Oct 31 12:27:05.682161 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.679044    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/50ad7816-358d-461e-a9bd-fadce6246e8f-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-2bfvp\" (UID: \"50ad7816-358d-461e-a9bd-fadce6246e8f\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-2bfvp"
Oct 31 12:27:05.682161 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:27:05.679131    2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 31 12:27:05.682161 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:27:05.679174    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50ad7816-358d-461e-a9bd-fadce6246e8f-networking-console-plugin-cert podName:50ad7816-358d-461e-a9bd-fadce6246e8f nodeName:}" failed. No retries permitted until 2026-10-31 12:27:06.67916252 +0000 UTC m=+31.571809662 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/50ad7816-358d-461e-a9bd-fadce6246e8f-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-2bfvp" (UID: "50ad7816-358d-461e-a9bd-fadce6246e8f") : secret "networking-console-plugin-cert" not found
Oct 31 12:27:05.682161 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:27:05.679246    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/556dda7b-d8e6-4558-b5f3-994b01d6a866-service-ca-bundle podName:556dda7b-d8e6-4558-b5f3-994b01d6a866 nodeName:}" failed. No retries permitted until 2026-10-31 12:27:06.679238971 +0000 UTC m=+31.571886113 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/556dda7b-d8e6-4558-b5f3-994b01d6a866-service-ca-bundle") pod "router-default-659589798c-776bx" (UID: "556dda7b-d8e6-4558-b5f3-994b01d6a866") : configmap references non-existent config key: service-ca.crt
Oct 31 12:27:05.682161 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:27:05.679301    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 31 12:27:05.682161 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:27:05.679348    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5c8865f9-fdf6-49b9-8b4f-69dbc7787084-metrics-tls podName:5c8865f9-fdf6-49b9-8b4f-69dbc7787084 nodeName:}" failed. No retries permitted until 2026-10-31 12:27:06.67933203 +0000 UTC m=+31.571979159 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5c8865f9-fdf6-49b9-8b4f-69dbc7787084-metrics-tls") pod "dns-default-r6fvt" (UID: "5c8865f9-fdf6-49b9-8b4f-69dbc7787084") : secret "dns-default-metrics-tls" not found
Oct 31 12:27:05.682161 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:27:05.679428    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 31 12:27:05.682161 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:27:05.679440    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6cfdbfcdc4-2rq84: secret "image-registry-tls" not found
Oct 31 12:27:05.682161 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:27:05.679475    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9083c35a-9739-4f3e-8415-bad1e6433d40-registry-tls podName:9083c35a-9739-4f3e-8415-bad1e6433d40 nodeName:}" failed. No retries permitted until 2026-10-31 12:27:06.679462065 +0000 UTC m=+31.572109208 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9083c35a-9739-4f3e-8415-bad1e6433d40-registry-tls") pod "image-registry-6cfdbfcdc4-2rq84" (UID: "9083c35a-9739-4f3e-8415-bad1e6433d40") : secret "image-registry-tls" not found
Oct 31 12:27:05.682161 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:27:05.679588    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 31 12:27:05.682161 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:27:05.679600    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6dd647c7b7-9255m: secret "image-registry-tls" not found
Oct 31 12:27:05.682161 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:27:05.679632    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0db116b4-d971-4966-bb05-db28d077b6fe-registry-tls podName:0db116b4-d971-4966-bb05-db28d077b6fe nodeName:}" failed. No retries permitted until 2026-10-31 12:27:06.679621322 +0000 UTC m=+31.572268464 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0db116b4-d971-4966-bb05-db28d077b6fe-registry-tls") pod "image-registry-6dd647c7b7-9255m" (UID: "0db116b4-d971-4966-bb05-db28d077b6fe") : secret "image-registry-tls" not found
Oct 31 12:27:05.682161 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:27:05.679689    2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 31 12:27:05.682161 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:27:05.679718    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dbbd2d70-8911-4aa5-96cf-3a16dcdf246d-samples-operator-tls podName:dbbd2d70-8911-4aa5-96cf-3a16dcdf246d nodeName:}" failed. No retries permitted until 2026-10-31 12:27:06.679708279 +0000 UTC m=+31.572355422 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/dbbd2d70-8911-4aa5-96cf-3a16dcdf246d-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-tcnrw" (UID: "dbbd2d70-8911-4aa5-96cf-3a16dcdf246d") : secret "samples-operator-tls" not found
Oct 31 12:27:05.682161 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:27:05.679762    2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 31 12:27:05.683013 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:27:05.679789    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dda0c26f-8225-4239-bc27-866b66cbb919-cluster-monitoring-operator-tls podName:dda0c26f-8225-4239-bc27-866b66cbb919 nodeName:}" failed. No retries permitted until 2026-10-31 12:27:06.679781563 +0000 UTC m=+31.572428706 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/dda0c26f-8225-4239-bc27-866b66cbb919-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-gt9rn" (UID: "dda0c26f-8225-4239-bc27-866b66cbb919") : secret "cluster-monitoring-operator-tls" not found
Oct 31 12:27:05.683013 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:27:05.679835    2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 31 12:27:05.683013 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:27:05.679857    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/556dda7b-d8e6-4558-b5f3-994b01d6a866-metrics-certs podName:556dda7b-d8e6-4558-b5f3-994b01d6a866 nodeName:}" failed. No retries permitted until 2026-10-31 12:27:06.679850744 +0000 UTC m=+31.572497872 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/556dda7b-d8e6-4558-b5f3-994b01d6a866-metrics-certs") pod "router-default-659589798c-776bx" (UID: "556dda7b-d8e6-4558-b5f3-994b01d6a866") : secret "router-metrics-certs-default" not found
Oct 31 12:27:05.686806 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:27:05.686775    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf032909d_bc39_4c9e_99b7_f3d800842119.slice/crio-68516ca2184b4f7cae9b513be83b9d11699718f173a9dfde5a4ffb4a1031cddc WatchSource:0}: Error finding container 68516ca2184b4f7cae9b513be83b9d11699718f173a9dfde5a4ffb4a1031cddc: Status 404 returned error can't find the container with id 68516ca2184b4f7cae9b513be83b9d11699718f173a9dfde5a4ffb4a1031cddc
Oct 31 12:27:05.692577 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.692556    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-pcvvm"]
Oct 31 12:27:05.692661 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.692619    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-c789s"]
Oct 31 12:27:05.693925 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.693860    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-rl55s"]
Oct 31 12:27:05.694468 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:27:05.694451    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod891b3c47_ff7c_4a7a_a09b_411361a843f7.slice/crio-24f74c6f47a4176d951ed9f1e88a53b5b9928f802c402c0074106d93796b3218 WatchSource:0}: Error finding container 24f74c6f47a4176d951ed9f1e88a53b5b9928f802c402c0074106d93796b3218: Status 404 returned error can't find the container with id 24f74c6f47a4176d951ed9f1e88a53b5b9928f802c402c0074106d93796b3218
Oct 31 12:27:05.696361 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:27:05.696339    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99a606a8_546b_4145_8bae_a95061241069.slice/crio-dd140492a16d3d96e416a11f11f04bef073fc5e086c5ae9eeeb1a019605577c1 WatchSource:0}: Error finding container dd140492a16d3d96e416a11f11f04bef073fc5e086c5ae9eeeb1a019605577c1: Status 404 returned error can't find the container with id dd140492a16d3d96e416a11f11f04bef073fc5e086c5ae9eeeb1a019605577c1
Oct 31 12:27:05.704292 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.704271    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-m9xxh"]
Oct 31 12:27:05.712444 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:27:05.712422    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25691fa3_7017_429a_b94c_c2dde4d29e55.slice/crio-432c216b017de6fb2dc9ee38001b85f4604b2715e988e0d643b46af056985f7d WatchSource:0}: Error finding container 432c216b017de6fb2dc9ee38001b85f4604b2715e988e0d643b46af056985f7d: Status 404 returned error can't find the container with id 432c216b017de6fb2dc9ee38001b85f4604b2715e988e0d643b46af056985f7d
Oct 31 12:27:05.918531 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.918469    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-c789s" event={"ID":"99a606a8-546b-4145-8bae-a95061241069","Type":"ContainerStarted","Data":"dd140492a16d3d96e416a11f11f04bef073fc5e086c5ae9eeeb1a019605577c1"}
Oct 31 12:27:05.919635 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.919608    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-hf9hr" event={"ID":"b28ee2bb-0387-4d2d-8ca7-3fef91619c6b","Type":"ContainerStarted","Data":"a130b0e336cef0401531b357c6118d6c3ec61ce95a7b67c9eda4cf43d7a4f443"}
Oct 31 12:27:05.921235 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.921178    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-z879s" event={"ID":"ddb04b2e-2724-44e9-948a-877a90d8a071","Type":"ContainerStarted","Data":"c1849c803f134d969c39443fe4e01b9f3a600259e4ff5749f6d3ca25a2a168b1"}
Oct 31 12:27:05.922519 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.922482    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-rrc8k" event={"ID":"ff424288-19ca-4db0-9f16-fed9fb58404a","Type":"ContainerStarted","Data":"44ee58d4c11a7acc5a35630d01dc307b97816b4f11fa1d2100ee117e8e39fe57"}
Oct 31 12:27:05.922588 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.922534    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-rrc8k" event={"ID":"ff424288-19ca-4db0-9f16-fed9fb58404a","Type":"ContainerStarted","Data":"432c0f75ce4f14ee6e21e05374e78fc51fdb2f101584925475c6900c584e1409"}
Oct 31 12:27:05.923819 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.923799    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-k64bz" event={"ID":"613125d4-fbb4-4b43-9c47-24e039bd4697","Type":"ContainerStarted","Data":"870f03cc04905b5d7cdfc59accbc77c75d7889b339779c832aedf49fc4edb1b1"}
Oct 31 12:27:05.923913 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.923835    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-k64bz" event={"ID":"613125d4-fbb4-4b43-9c47-24e039bd4697","Type":"ContainerStarted","Data":"5a4af82d83c4e2d729012b35f36754f633c7d8edae532b658e217176aeed8416"}
Oct 31 12:27:05.924895 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.924876    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-m9xxh" event={"ID":"25691fa3-7017-429a-b94c-c2dde4d29e55","Type":"ContainerStarted","Data":"432c216b017de6fb2dc9ee38001b85f4604b2715e988e0d643b46af056985f7d"}
Oct 31 12:27:05.925932 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.925913    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-vqsv9" event={"ID":"febbc28e-bbcc-4961-bde8-09944d9ca680","Type":"ContainerStarted","Data":"9ec8c5086bd55eeaa1cf3cd8ee3146133e8915c24281d20fe7de74ac684c8429"}
Oct 31 12:27:05.927085 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.927064    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-pcvvm" event={"ID":"f032909d-bc39-4c9e-99b7-f3d800842119","Type":"ContainerStarted","Data":"68516ca2184b4f7cae9b513be83b9d11699718f173a9dfde5a4ffb4a1031cddc"}
Oct 31 12:27:05.928184 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.928164    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-rl55s" event={"ID":"891b3c47-ff7c-4a7a-a09b-411361a843f7","Type":"ContainerStarted","Data":"24f74c6f47a4176d951ed9f1e88a53b5b9928f802c402c0074106d93796b3218"}
Oct 31 12:27:05.929473 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.929454    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-vvzpg" event={"ID":"07714f20-a70e-49e8-8e51-d03a8e00005a","Type":"ContainerStarted","Data":"a0f9aa0c6b23b1c2539c91425dc227eea335c744b22ee5cf38c2096c90ac09d0"}
Oct 31 12:27:05.929551 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.929484    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-vvzpg" event={"ID":"07714f20-a70e-49e8-8e51-d03a8e00005a","Type":"ContainerStarted","Data":"8a59f3fca9f665e623be85a15c0d77fe56925fd62aae37db535bb9a23c596a6f"}
Oct 31 12:27:05.941147 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.941102    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-rrc8k" podStartSLOduration=424.941091232 podStartE2EDuration="7m4.941091232s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:20:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:27:05.940646956 +0000 UTC m=+30.833294111" watchObservedRunningTime="2026-10-31 12:27:05.941091232 +0000 UTC m=+30.833738382"
Oct 31 12:27:05.959508 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.959449    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-vvzpg" podStartSLOduration=424.959437517 podStartE2EDuration="7m4.959437517s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:20:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:27:05.95883229 +0000 UTC m=+30.851479442" watchObservedRunningTime="2026-10-31 12:27:05.959437517 +0000 UTC m=+30.852084679"
Oct 31 12:27:05.976928 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:05.976887    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-k64bz" podStartSLOduration=424.976874334 podStartE2EDuration="7m4.976874334s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:20:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:27:05.97643035 +0000 UTC m=+30.869077501" watchObservedRunningTime="2026-10-31 12:27:05.976874334 +0000 UTC m=+30.869521486"
Oct 31 12:27:06.656808 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:06.656770    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-94kh9"
Oct 31 12:27:06.657322 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:06.657307    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tdrmt"
Oct 31 12:27:06.657715 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:06.657696    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2bmzr"
Oct 31 12:27:06.661994 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:06.661969    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 31 12:27:06.662179 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:06.662163    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-mwkk2\""
Oct 31 12:27:06.663875 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:06.663695    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-ghjl2\""
Oct 31 12:27:06.663875 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:06.663746    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-s7td2\""
Oct 31 12:27:06.663875 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:06.663792    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 31 12:27:06.689507 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:06.689468    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/556dda7b-d8e6-4558-b5f3-994b01d6a866-metrics-certs\") pod \"router-default-659589798c-776bx\" (UID: \"556dda7b-d8e6-4558-b5f3-994b01d6a866\") " pod="openshift-ingress/router-default-659589798c-776bx"
Oct 31 12:27:06.689901 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:06.689537    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/50ad7816-358d-461e-a9bd-fadce6246e8f-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-2bfvp\" (UID: \"50ad7816-358d-461e-a9bd-fadce6246e8f\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-2bfvp"
Oct 31 12:27:06.689901 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:06.689611    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/556dda7b-d8e6-4558-b5f3-994b01d6a866-service-ca-bundle\") pod \"router-default-659589798c-776bx\" (UID: \"556dda7b-d8e6-4558-b5f3-994b01d6a866\") " pod="openshift-ingress/router-default-659589798c-776bx"
Oct 31 12:27:06.690126 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:27:06.690092    2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 31 12:27:06.690186 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:27:06.690161    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50ad7816-358d-461e-a9bd-fadce6246e8f-networking-console-plugin-cert podName:50ad7816-358d-461e-a9bd-fadce6246e8f nodeName:}" failed. No retries permitted until 2026-10-31 12:27:08.690141679 +0000 UTC m=+33.582788824 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/50ad7816-358d-461e-a9bd-fadce6246e8f-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-2bfvp" (UID: "50ad7816-358d-461e-a9bd-fadce6246e8f") : secret "networking-console-plugin-cert" not found
Oct 31 12:27:06.690692 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:27:06.690674    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/556dda7b-d8e6-4558-b5f3-994b01d6a866-service-ca-bundle podName:556dda7b-d8e6-4558-b5f3-994b01d6a866 nodeName:}" failed. No retries permitted until 2026-10-31 12:27:08.69065677 +0000 UTC m=+33.583303916 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/556dda7b-d8e6-4558-b5f3-994b01d6a866-service-ca-bundle") pod "router-default-659589798c-776bx" (UID: "556dda7b-d8e6-4558-b5f3-994b01d6a866") : configmap references non-existent config key: service-ca.crt
Oct 31 12:27:06.690783 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:27:06.690757    2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 31 12:27:06.690834 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:27:06.690795    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/556dda7b-d8e6-4558-b5f3-994b01d6a866-metrics-certs podName:556dda7b-d8e6-4558-b5f3-994b01d6a866 nodeName:}" failed. No retries permitted until 2026-10-31 12:27:08.690783345 +0000 UTC m=+33.583430514 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/556dda7b-d8e6-4558-b5f3-994b01d6a866-metrics-certs") pod "router-default-659589798c-776bx" (UID: "556dda7b-d8e6-4558-b5f3-994b01d6a866") : secret "router-metrics-certs-default" not found
Oct 31 12:27:06.690834 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:06.689640    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5c8865f9-fdf6-49b9-8b4f-69dbc7787084-metrics-tls\") pod \"dns-default-r6fvt\" (UID: \"5c8865f9-fdf6-49b9-8b4f-69dbc7787084\") " pod="openshift-dns/dns-default-r6fvt"
Oct 31 12:27:06.690991 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:06.690973    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9083c35a-9739-4f3e-8415-bad1e6433d40-registry-tls\") pod \"image-registry-6cfdbfcdc4-2rq84\" (UID: \"9083c35a-9739-4f3e-8415-bad1e6433d40\") " pod="openshift-image-registry/image-registry-6cfdbfcdc4-2rq84"
Oct 31 12:27:06.691031 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:06.691012    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0db116b4-d971-4966-bb05-db28d077b6fe-registry-tls\") pod \"image-registry-6dd647c7b7-9255m\" (UID: \"0db116b4-d971-4966-bb05-db28d077b6fe\") " pod="openshift-image-registry/image-registry-6dd647c7b7-9255m"
Oct 31 12:27:06.691061 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:06.691046    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/dbbd2d70-8911-4aa5-96cf-3a16dcdf246d-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-tcnrw\" (UID: \"dbbd2d70-8911-4aa5-96cf-3a16dcdf246d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-tcnrw"
Oct 31 12:27:06.691093 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:06.691075    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/dda0c26f-8225-4239-bc27-866b66cbb919-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-gt9rn\" (UID: \"dda0c26f-8225-4239-bc27-866b66cbb919\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gt9rn"
Oct 31 12:27:06.691226 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:27:06.691211    2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 31 12:27:06.691285 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:27:06.691257    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dda0c26f-8225-4239-bc27-866b66cbb919-cluster-monitoring-operator-tls podName:dda0c26f-8225-4239-bc27-866b66cbb919 nodeName:}" failed. No retries permitted until 2026-10-31 12:27:08.691245339 +0000 UTC m=+33.583892474 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/dda0c26f-8225-4239-bc27-866b66cbb919-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-gt9rn" (UID: "dda0c26f-8225-4239-bc27-866b66cbb919") : secret "cluster-monitoring-operator-tls" not found
Oct 31 12:27:06.691338 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:27:06.691317    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 31 12:27:06.691384 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:27:06.691348    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5c8865f9-fdf6-49b9-8b4f-69dbc7787084-metrics-tls podName:5c8865f9-fdf6-49b9-8b4f-69dbc7787084 nodeName:}" failed. No retries permitted until 2026-10-31 12:27:08.691337972 +0000 UTC m=+33.583985116 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5c8865f9-fdf6-49b9-8b4f-69dbc7787084-metrics-tls") pod "dns-default-r6fvt" (UID: "5c8865f9-fdf6-49b9-8b4f-69dbc7787084") : secret "dns-default-metrics-tls" not found
Oct 31 12:27:06.691435 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:27:06.691426    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 31 12:27:06.691504 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:27:06.691437    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6cfdbfcdc4-2rq84: secret "image-registry-tls" not found
Oct 31 12:27:06.691504 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:27:06.691470    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9083c35a-9739-4f3e-8415-bad1e6433d40-registry-tls podName:9083c35a-9739-4f3e-8415-bad1e6433d40 nodeName:}" failed. No retries permitted until 2026-10-31 12:27:08.691459261 +0000 UTC m=+33.584106404 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9083c35a-9739-4f3e-8415-bad1e6433d40-registry-tls") pod "image-registry-6cfdbfcdc4-2rq84" (UID: "9083c35a-9739-4f3e-8415-bad1e6433d40") : secret "image-registry-tls" not found
Oct 31 12:27:06.691609 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:27:06.691541    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 31 12:27:06.691609 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:27:06.691552    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6dd647c7b7-9255m: secret "image-registry-tls" not found
Oct 31 12:27:06.691609 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:27:06.691594    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0db116b4-d971-4966-bb05-db28d077b6fe-registry-tls podName:0db116b4-d971-4966-bb05-db28d077b6fe nodeName:}" failed. No retries permitted until 2026-10-31 12:27:08.691584141 +0000 UTC m=+33.584231284 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0db116b4-d971-4966-bb05-db28d077b6fe-registry-tls") pod "image-registry-6dd647c7b7-9255m" (UID: "0db116b4-d971-4966-bb05-db28d077b6fe") : secret "image-registry-tls" not found
Oct 31 12:27:06.691753 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:27:06.691682    2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 31 12:27:06.691753 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:27:06.691725    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dbbd2d70-8911-4aa5-96cf-3a16dcdf246d-samples-operator-tls podName:dbbd2d70-8911-4aa5-96cf-3a16dcdf246d nodeName:}" failed. No retries permitted until 2026-10-31 12:27:08.691713042 +0000 UTC m=+33.584360171 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/dbbd2d70-8911-4aa5-96cf-3a16dcdf246d-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-tcnrw" (UID: "dbbd2d70-8911-4aa5-96cf-3a16dcdf246d") : secret "samples-operator-tls" not found
Oct 31 12:27:08.715481 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:08.715438    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/556dda7b-d8e6-4558-b5f3-994b01d6a866-service-ca-bundle\") pod \"router-default-659589798c-776bx\" (UID: \"556dda7b-d8e6-4558-b5f3-994b01d6a866\") " pod="openshift-ingress/router-default-659589798c-776bx"
Oct 31 12:27:08.716074 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:08.715505    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5c8865f9-fdf6-49b9-8b4f-69dbc7787084-metrics-tls\") pod \"dns-default-r6fvt\" (UID: \"5c8865f9-fdf6-49b9-8b4f-69dbc7787084\") " pod="openshift-dns/dns-default-r6fvt"
Oct 31 12:27:08.716074 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:08.715600    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9083c35a-9739-4f3e-8415-bad1e6433d40-registry-tls\") pod \"image-registry-6cfdbfcdc4-2rq84\" (UID: \"9083c35a-9739-4f3e-8415-bad1e6433d40\") " pod="openshift-image-registry/image-registry-6cfdbfcdc4-2rq84"
Oct 31 12:27:08.716074 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:08.715630    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0db116b4-d971-4966-bb05-db28d077b6fe-registry-tls\") pod \"image-registry-6dd647c7b7-9255m\" (UID: \"0db116b4-d971-4966-bb05-db28d077b6fe\") " pod="openshift-image-registry/image-registry-6dd647c7b7-9255m"
Oct 31 12:27:08.716074 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:08.715660    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/dbbd2d70-8911-4aa5-96cf-3a16dcdf246d-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-tcnrw\" (UID: \"dbbd2d70-8911-4aa5-96cf-3a16dcdf246d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-tcnrw"
Oct 31 12:27:08.716074 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:08.715690    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/dda0c26f-8225-4239-bc27-866b66cbb919-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-gt9rn\" (UID: \"dda0c26f-8225-4239-bc27-866b66cbb919\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gt9rn"
Oct 31 12:27:08.716074 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:08.715732    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/556dda7b-d8e6-4558-b5f3-994b01d6a866-metrics-certs\") pod \"router-default-659589798c-776bx\" (UID: \"556dda7b-d8e6-4558-b5f3-994b01d6a866\") " pod="openshift-ingress/router-default-659589798c-776bx"
Oct 31 12:27:08.716074 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:08.715761    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/50ad7816-358d-461e-a9bd-fadce6246e8f-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-2bfvp\" (UID: \"50ad7816-358d-461e-a9bd-fadce6246e8f\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-2bfvp"
Oct 31 12:27:08.716074 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:27:08.715889    2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 31 12:27:08.716074 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:27:08.715934    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50ad7816-358d-461e-a9bd-fadce6246e8f-networking-console-plugin-cert podName:50ad7816-358d-461e-a9bd-fadce6246e8f nodeName:}" failed. No retries permitted until 2026-10-31 12:27:12.715919451 +0000 UTC m=+37.608566580 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/50ad7816-358d-461e-a9bd-fadce6246e8f-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-2bfvp" (UID: "50ad7816-358d-461e-a9bd-fadce6246e8f") : secret "networking-console-plugin-cert" not found
Oct 31 12:27:08.716074 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:27:08.715987    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 31 12:27:08.716074 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:27:08.715998    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/556dda7b-d8e6-4558-b5f3-994b01d6a866-service-ca-bundle podName:556dda7b-d8e6-4558-b5f3-994b01d6a866 nodeName:}" failed. No retries permitted until 2026-10-31 12:27:12.715976948 +0000 UTC m=+37.608624084 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/556dda7b-d8e6-4558-b5f3-994b01d6a866-service-ca-bundle") pod "router-default-659589798c-776bx" (UID: "556dda7b-d8e6-4558-b5f3-994b01d6a866") : configmap references non-existent config key: service-ca.crt
Oct 31 12:27:08.716074 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:27:08.716025    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5c8865f9-fdf6-49b9-8b4f-69dbc7787084-metrics-tls podName:5c8865f9-fdf6-49b9-8b4f-69dbc7787084 nodeName:}" failed. No retries permitted until 2026-10-31 12:27:12.716013984 +0000 UTC m=+37.608661188 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5c8865f9-fdf6-49b9-8b4f-69dbc7787084-metrics-tls") pod "dns-default-r6fvt" (UID: "5c8865f9-fdf6-49b9-8b4f-69dbc7787084") : secret "dns-default-metrics-tls" not found
Oct 31 12:27:08.716074 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:27:08.716036    2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 31 12:27:08.716074 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:27:08.716056    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dbbd2d70-8911-4aa5-96cf-3a16dcdf246d-samples-operator-tls podName:dbbd2d70-8911-4aa5-96cf-3a16dcdf246d nodeName:}" failed. No retries permitted until 2026-10-31 12:27:12.71604923 +0000 UTC m=+37.608696359 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/dbbd2d70-8911-4aa5-96cf-3a16dcdf246d-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-tcnrw" (UID: "dbbd2d70-8911-4aa5-96cf-3a16dcdf246d") : secret "samples-operator-tls" not found
Oct 31 12:27:08.716770 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:27:08.716102    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 31 12:27:08.716770 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:27:08.716112    2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 31 12:27:08.716770 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:27:08.716135    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dda0c26f-8225-4239-bc27-866b66cbb919-cluster-monitoring-operator-tls podName:dda0c26f-8225-4239-bc27-866b66cbb919 nodeName:}" failed. No retries permitted until 2026-10-31 12:27:12.716128413 +0000 UTC m=+37.608775543 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/dda0c26f-8225-4239-bc27-866b66cbb919-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-gt9rn" (UID: "dda0c26f-8225-4239-bc27-866b66cbb919") : secret "cluster-monitoring-operator-tls" not found
Oct 31 12:27:08.716770 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:27:08.716163    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 31 12:27:08.716770 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:27:08.716173    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6dd647c7b7-9255m: secret "image-registry-tls" not found
Oct 31 12:27:08.716770 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:27:08.716213    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0db116b4-d971-4966-bb05-db28d077b6fe-registry-tls podName:0db116b4-d971-4966-bb05-db28d077b6fe nodeName:}" failed. No retries permitted until 2026-10-31 12:27:12.716202166 +0000 UTC m=+37.608849310 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0db116b4-d971-4966-bb05-db28d077b6fe-registry-tls") pod "image-registry-6dd647c7b7-9255m" (UID: "0db116b4-d971-4966-bb05-db28d077b6fe") : secret "image-registry-tls" not found
Oct 31 12:27:08.716770 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:27:08.716166    2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 31 12:27:08.716770 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:27:08.716114    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6cfdbfcdc4-2rq84: secret "image-registry-tls" not found
Oct 31 12:27:08.716770 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:27:08.716249    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/556dda7b-d8e6-4558-b5f3-994b01d6a866-metrics-certs podName:556dda7b-d8e6-4558-b5f3-994b01d6a866 nodeName:}" failed. No retries permitted until 2026-10-31 12:27:12.716240188 +0000 UTC m=+37.608887323 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/556dda7b-d8e6-4558-b5f3-994b01d6a866-metrics-certs") pod "router-default-659589798c-776bx" (UID: "556dda7b-d8e6-4558-b5f3-994b01d6a866") : secret "router-metrics-certs-default" not found
Oct 31 12:27:08.716770 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:27:08.716314    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9083c35a-9739-4f3e-8415-bad1e6433d40-registry-tls podName:9083c35a-9739-4f3e-8415-bad1e6433d40 nodeName:}" failed. No retries permitted until 2026-10-31 12:27:12.716296586 +0000 UTC m=+37.608943718 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9083c35a-9739-4f3e-8415-bad1e6433d40-registry-tls") pod "image-registry-6cfdbfcdc4-2rq84" (UID: "9083c35a-9739-4f3e-8415-bad1e6433d40") : secret "image-registry-tls" not found
Oct 31 12:27:08.944719 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:08.944678    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-hf9hr" event={"ID":"b28ee2bb-0387-4d2d-8ca7-3fef91619c6b","Type":"ContainerStarted","Data":"9b546832dc12d9a6980921cef086b1960216a846242d5f0087301afd2be069f2"}
Oct 31 12:27:08.967166 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:08.966934    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-hf9hr" podStartSLOduration=392.35988681 podStartE2EDuration="6m34.966914639s" totalImagesPullingTime="2.607027829s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:20:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:27:08.965084375 +0000 UTC m=+33.857731525" watchObservedRunningTime="2026-10-31 12:27:08.966914639 +0000 UTC m=+33.859561791"
Oct 31 12:27:09.424542 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:09.423529    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tf6n\" (UniqueName: \"kubernetes.io/projected/7a402472-ca33-44ae-b598-96fc8e7a5eb2-kube-api-access-2tf6n\") pod \"network-check-target-tdrmt\" (UID: \"7a402472-ca33-44ae-b598-96fc8e7a5eb2\") " pod="openshift-network-diagnostics/network-check-target-tdrmt"
Oct 31 12:27:09.424542 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:09.423651    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a2b450be-ca79-4edf-a1ea-377531a49519-original-pull-secret\") pod \"global-pull-secret-syncer-2bmzr\" (UID: \"a2b450be-ca79-4edf-a1ea-377531a49519\") " pod="kube-system/global-pull-secret-syncer-2bmzr"
Oct 31 12:27:09.424542 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:09.423709    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e64bd69d-946f-42ad-94ae-b87b7de3a581-metrics-certs\") pod \"network-metrics-daemon-94kh9\" (UID: \"e64bd69d-946f-42ad-94ae-b87b7de3a581\") " pod="openshift-multus/network-metrics-daemon-94kh9"
Oct 31 12:27:09.424542 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:27:09.423849    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 31 12:27:09.424542 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:27:09.423900    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e64bd69d-946f-42ad-94ae-b87b7de3a581-metrics-certs podName:e64bd69d-946f-42ad-94ae-b87b7de3a581 nodeName:}" failed. No retries permitted until 2026-10-31 12:27:41.423887157 +0000 UTC m=+66.316534286 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e64bd69d-946f-42ad-94ae-b87b7de3a581-metrics-certs") pod "network-metrics-daemon-94kh9" (UID: "e64bd69d-946f-42ad-94ae-b87b7de3a581") : secret "metrics-daemon-secret" not found
Oct 31 12:27:09.428879 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:09.428849    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a2b450be-ca79-4edf-a1ea-377531a49519-original-pull-secret\") pod \"global-pull-secret-syncer-2bmzr\" (UID: \"a2b450be-ca79-4edf-a1ea-377531a49519\") " pod="kube-system/global-pull-secret-syncer-2bmzr"
Oct 31 12:27:09.429022 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:09.428945    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tf6n\" (UniqueName: \"kubernetes.io/projected/7a402472-ca33-44ae-b598-96fc8e7a5eb2-kube-api-access-2tf6n\") pod \"network-check-target-tdrmt\" (UID: \"7a402472-ca33-44ae-b598-96fc8e7a5eb2\") " pod="openshift-network-diagnostics/network-check-target-tdrmt"
Oct 31 12:27:09.712500 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:09.712461    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2bmzr"
Oct 31 12:27:09.722827 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:09.722639    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tdrmt"
Oct 31 12:27:09.960286 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:09.960205    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-z879s" event={"ID":"ddb04b2e-2724-44e9-948a-877a90d8a071","Type":"ContainerStarted","Data":"5c2ba5d038743ef59b4681530bba2d1485c47084c72bebb1b6d6c53c482fdebf"}
Oct 31 12:27:09.981484 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:09.981434    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-z879s" podStartSLOduration=392.027408991 podStartE2EDuration="6m35.981415919s" totalImagesPullingTime="3.954006928s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:20:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:27:09.981186018 +0000 UTC m=+34.873833195" watchObservedRunningTime="2026-10-31 12:27:09.981415919 +0000 UTC m=+34.874063072"
Oct 31 12:27:10.267814 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:10.267763    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-p7phb"]
Oct 31 12:27:10.302771 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:10.302739    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-p7phb"]
Oct 31 12:27:10.302940 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:10.302794    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-87ncb-cnk5g"]
Oct 31 12:27:10.302940 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:10.302886    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-p7phb"
Oct 31 12:27:10.308222 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:10.307691    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 31 12:27:10.308222 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:10.307785    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 31 12:27:10.308222 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:10.308007    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-c6kzr\""
Oct 31 12:27:10.323543 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:10.322680    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-87ncb-cnk5g"]
Oct 31 12:27:10.323543 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:10.322792    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-87ncb-cnk5g"
Oct 31 12:27:10.338320 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:10.337342    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9557a3f0-ae0f-41f9-a9b5-15aeda410a38-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p7phb\" (UID: \"9557a3f0-ae0f-41f9-a9b5-15aeda410a38\") " pod="openshift-insights/insights-runtime-extractor-p7phb"
Oct 31 12:27:10.338320 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:10.337385    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9557a3f0-ae0f-41f9-a9b5-15aeda410a38-data-volume\") pod \"insights-runtime-extractor-p7phb\" (UID: \"9557a3f0-ae0f-41f9-a9b5-15aeda410a38\") " pod="openshift-insights/insights-runtime-extractor-p7phb"
Oct 31 12:27:10.338320 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:10.337428    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9557a3f0-ae0f-41f9-a9b5-15aeda410a38-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p7phb\" (UID: \"9557a3f0-ae0f-41f9-a9b5-15aeda410a38\") " pod="openshift-insights/insights-runtime-extractor-p7phb"
Oct 31 12:27:10.338320 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:10.337511    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/eb20e317-3ebe-4f5c-9b37-1de9ea5f063f-archives-path\") pod \"periodic-gathering-87ncb-cnk5g\" (UID: \"eb20e317-3ebe-4f5c-9b37-1de9ea5f063f\") " pod="openshift-insights/periodic-gathering-87ncb-cnk5g"
Oct 31 12:27:10.338320 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:10.337536    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jl4xd\" (UniqueName: \"kubernetes.io/projected/9557a3f0-ae0f-41f9-a9b5-15aeda410a38-kube-api-access-jl4xd\") pod \"insights-runtime-extractor-p7phb\" (UID: \"9557a3f0-ae0f-41f9-a9b5-15aeda410a38\") " pod="openshift-insights/insights-runtime-extractor-p7phb"
Oct 31 12:27:10.338320 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:10.337561    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb20e317-3ebe-4f5c-9b37-1de9ea5f063f-service-ca-bundle\") pod \"periodic-gathering-87ncb-cnk5g\" (UID: \"eb20e317-3ebe-4f5c-9b37-1de9ea5f063f\") " pod="openshift-insights/periodic-gathering-87ncb-cnk5g"
Oct 31 12:27:10.338320 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:10.337587    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvtwq\" (UniqueName: \"kubernetes.io/projected/eb20e317-3ebe-4f5c-9b37-1de9ea5f063f-kube-api-access-kvtwq\") pod \"periodic-gathering-87ncb-cnk5g\" (UID: \"eb20e317-3ebe-4f5c-9b37-1de9ea5f063f\") " pod="openshift-insights/periodic-gathering-87ncb-cnk5g"
Oct 31 12:27:10.338320 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:10.337633    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9557a3f0-ae0f-41f9-a9b5-15aeda410a38-crio-socket\") pod \"insights-runtime-extractor-p7phb\" (UID: \"9557a3f0-ae0f-41f9-a9b5-15aeda410a38\") " pod="openshift-insights/insights-runtime-extractor-p7phb"
Oct 31 12:27:10.338320 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:10.337727    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb20e317-3ebe-4f5c-9b37-1de9ea5f063f-trusted-ca-bundle\") pod \"periodic-gathering-87ncb-cnk5g\" (UID: \"eb20e317-3ebe-4f5c-9b37-1de9ea5f063f\") " pod="openshift-insights/periodic-gathering-87ncb-cnk5g"
Oct 31 12:27:10.408917 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:10.408893    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-lffk4\""
Oct 31 12:27:10.438039 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:10.438005    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9557a3f0-ae0f-41f9-a9b5-15aeda410a38-data-volume\") pod \"insights-runtime-extractor-p7phb\" (UID: \"9557a3f0-ae0f-41f9-a9b5-15aeda410a38\") " pod="openshift-insights/insights-runtime-extractor-p7phb"
Oct 31 12:27:10.438198 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:10.438066    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9557a3f0-ae0f-41f9-a9b5-15aeda410a38-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p7phb\" (UID: \"9557a3f0-ae0f-41f9-a9b5-15aeda410a38\") " pod="openshift-insights/insights-runtime-extractor-p7phb"
Oct 31 12:27:10.438198 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:10.438140    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/eb20e317-3ebe-4f5c-9b37-1de9ea5f063f-archives-path\") pod \"periodic-gathering-87ncb-cnk5g\" (UID: \"eb20e317-3ebe-4f5c-9b37-1de9ea5f063f\") " pod="openshift-insights/periodic-gathering-87ncb-cnk5g"
Oct 31 12:27:10.438198 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:10.438162    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jl4xd\" (UniqueName: \"kubernetes.io/projected/9557a3f0-ae0f-41f9-a9b5-15aeda410a38-kube-api-access-jl4xd\") pod \"insights-runtime-extractor-p7phb\" (UID: \"9557a3f0-ae0f-41f9-a9b5-15aeda410a38\") " pod="openshift-insights/insights-runtime-extractor-p7phb"
Oct 31 12:27:10.438371 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:10.438203    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb20e317-3ebe-4f5c-9b37-1de9ea5f063f-service-ca-bundle\") pod \"periodic-gathering-87ncb-cnk5g\" (UID: \"eb20e317-3ebe-4f5c-9b37-1de9ea5f063f\") " pod="openshift-insights/periodic-gathering-87ncb-cnk5g"
Oct 31 12:27:10.438371 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:10.438228    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvtwq\" (UniqueName: \"kubernetes.io/projected/eb20e317-3ebe-4f5c-9b37-1de9ea5f063f-kube-api-access-kvtwq\") pod \"periodic-gathering-87ncb-cnk5g\" (UID: \"eb20e317-3ebe-4f5c-9b37-1de9ea5f063f\") " pod="openshift-insights/periodic-gathering-87ncb-cnk5g"
Oct 31 12:27:10.438371 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:10.438262    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9557a3f0-ae0f-41f9-a9b5-15aeda410a38-crio-socket\") pod \"insights-runtime-extractor-p7phb\" (UID: \"9557a3f0-ae0f-41f9-a9b5-15aeda410a38\") " pod="openshift-insights/insights-runtime-extractor-p7phb"
Oct 31 12:27:10.438371 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:10.438351    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb20e317-3ebe-4f5c-9b37-1de9ea5f063f-trusted-ca-bundle\") pod \"periodic-gathering-87ncb-cnk5g\" (UID: \"eb20e317-3ebe-4f5c-9b37-1de9ea5f063f\") " pod="openshift-insights/periodic-gathering-87ncb-cnk5g"
Oct 31 12:27:10.438772 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:10.438396    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9557a3f0-ae0f-41f9-a9b5-15aeda410a38-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p7phb\" (UID: \"9557a3f0-ae0f-41f9-a9b5-15aeda410a38\") " pod="openshift-insights/insights-runtime-extractor-p7phb"
Oct 31 12:27:10.438772 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:27:10.438480    2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 31 12:27:10.438772 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:27:10.438592    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9557a3f0-ae0f-41f9-a9b5-15aeda410a38-kube-rbac-proxy-cm podName:9557a3f0-ae0f-41f9-a9b5-15aeda410a38 nodeName:}" failed. No retries permitted until 2026-10-31 12:27:10.93854289 +0000 UTC m=+35.831190018 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9557a3f0-ae0f-41f9-a9b5-15aeda410a38-kube-rbac-proxy-cm") pod "insights-runtime-extractor-p7phb" (UID: "9557a3f0-ae0f-41f9-a9b5-15aeda410a38") : configmap "kube-rbac-proxy" not found
Oct 31 12:27:10.439005 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:27:10.438812    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 31 12:27:10.439005 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:27:10.438861    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9557a3f0-ae0f-41f9-a9b5-15aeda410a38-insights-runtime-extractor-tls podName:9557a3f0-ae0f-41f9-a9b5-15aeda410a38 nodeName:}" failed. No retries permitted until 2026-10-31 12:27:10.938846319 +0000 UTC m=+35.831493447 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9557a3f0-ae0f-41f9-a9b5-15aeda410a38-insights-runtime-extractor-tls") pod "insights-runtime-extractor-p7phb" (UID: "9557a3f0-ae0f-41f9-a9b5-15aeda410a38") : secret "insights-runtime-extractor-tls" not found
Oct 31 12:27:10.439005 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:10.438956    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9557a3f0-ae0f-41f9-a9b5-15aeda410a38-data-volume\") pod \"insights-runtime-extractor-p7phb\" (UID: \"9557a3f0-ae0f-41f9-a9b5-15aeda410a38\") " pod="openshift-insights/insights-runtime-extractor-p7phb"
Oct 31 12:27:10.439209 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:10.439063    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9557a3f0-ae0f-41f9-a9b5-15aeda410a38-crio-socket\") pod \"insights-runtime-extractor-p7phb\" (UID: \"9557a3f0-ae0f-41f9-a9b5-15aeda410a38\") " pod="openshift-insights/insights-runtime-extractor-p7phb"
Oct 31 12:27:10.439268 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:10.439218    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/eb20e317-3ebe-4f5c-9b37-1de9ea5f063f-archives-path\") pod \"periodic-gathering-87ncb-cnk5g\" (UID: \"eb20e317-3ebe-4f5c-9b37-1de9ea5f063f\") " pod="openshift-insights/periodic-gathering-87ncb-cnk5g"
Oct 31 12:27:10.439982 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:10.439642    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb20e317-3ebe-4f5c-9b37-1de9ea5f063f-service-ca-bundle\") pod \"periodic-gathering-87ncb-cnk5g\" (UID: \"eb20e317-3ebe-4f5c-9b37-1de9ea5f063f\") " pod="openshift-insights/periodic-gathering-87ncb-cnk5g"
Oct 31 12:27:10.440117 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:10.440091    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb20e317-3ebe-4f5c-9b37-1de9ea5f063f-trusted-ca-bundle\") pod \"periodic-gathering-87ncb-cnk5g\" (UID: \"eb20e317-3ebe-4f5c-9b37-1de9ea5f063f\") " pod="openshift-insights/periodic-gathering-87ncb-cnk5g"
Oct 31 12:27:10.453712 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:10.452757    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-2bmzr"]
Oct 31 12:27:10.459306 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:10.459277    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jl4xd\" (UniqueName: \"kubernetes.io/projected/9557a3f0-ae0f-41f9-a9b5-15aeda410a38-kube-api-access-jl4xd\") pod \"insights-runtime-extractor-p7phb\" (UID: \"9557a3f0-ae0f-41f9-a9b5-15aeda410a38\") " pod="openshift-insights/insights-runtime-extractor-p7phb"
Oct 31 12:27:10.459472 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:10.459448    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvtwq\" (UniqueName: \"kubernetes.io/projected/eb20e317-3ebe-4f5c-9b37-1de9ea5f063f-kube-api-access-kvtwq\") pod \"periodic-gathering-87ncb-cnk5g\" (UID: \"eb20e317-3ebe-4f5c-9b37-1de9ea5f063f\") " pod="openshift-insights/periodic-gathering-87ncb-cnk5g"
Oct 31 12:27:10.466121 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:10.464887    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-tdrmt"]
Oct 31 12:27:10.469401 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:27:10.469337    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a402472_ca33_44ae_b598_96fc8e7a5eb2.slice/crio-8dbe3066e41c634d3607763741f771139a4be4a86fb97f302e275ecb6c7399bd WatchSource:0}: Error finding container 8dbe3066e41c634d3607763741f771139a4be4a86fb97f302e275ecb6c7399bd: Status 404 returned error can't find the container with id 8dbe3066e41c634d3607763741f771139a4be4a86fb97f302e275ecb6c7399bd
Oct 31 12:27:10.695067 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:10.695031    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-87ncb-cnk5g"
Oct 31 12:27:10.943601 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:10.943556    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9557a3f0-ae0f-41f9-a9b5-15aeda410a38-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p7phb\" (UID: \"9557a3f0-ae0f-41f9-a9b5-15aeda410a38\") " pod="openshift-insights/insights-runtime-extractor-p7phb"
Oct 31 12:27:10.944048 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:10.943620    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9557a3f0-ae0f-41f9-a9b5-15aeda410a38-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p7phb\" (UID: \"9557a3f0-ae0f-41f9-a9b5-15aeda410a38\") " pod="openshift-insights/insights-runtime-extractor-p7phb"
Oct 31 12:27:10.944048 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:27:10.943679    2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 31 12:27:10.944048 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:27:10.943748    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9557a3f0-ae0f-41f9-a9b5-15aeda410a38-kube-rbac-proxy-cm podName:9557a3f0-ae0f-41f9-a9b5-15aeda410a38 nodeName:}" failed. No retries permitted until 2026-10-31 12:27:11.943732603 +0000 UTC m=+36.836379732 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9557a3f0-ae0f-41f9-a9b5-15aeda410a38-kube-rbac-proxy-cm") pod "insights-runtime-extractor-p7phb" (UID: "9557a3f0-ae0f-41f9-a9b5-15aeda410a38") : configmap "kube-rbac-proxy" not found
Oct 31 12:27:10.944048 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:27:10.943801    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 31 12:27:10.944048 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:27:10.943874    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9557a3f0-ae0f-41f9-a9b5-15aeda410a38-insights-runtime-extractor-tls podName:9557a3f0-ae0f-41f9-a9b5-15aeda410a38 nodeName:}" failed. No retries permitted until 2026-10-31 12:27:11.943855824 +0000 UTC m=+36.836502953 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9557a3f0-ae0f-41f9-a9b5-15aeda410a38-insights-runtime-extractor-tls") pod "insights-runtime-extractor-p7phb" (UID: "9557a3f0-ae0f-41f9-a9b5-15aeda410a38") : secret "insights-runtime-extractor-tls" not found
Oct 31 12:27:10.965935 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:10.965882    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-vqsv9" event={"ID":"febbc28e-bbcc-4961-bde8-09944d9ca680","Type":"ContainerStarted","Data":"3db3a0ce1da4d3c071e219120fef1b0e1e1c7f5f370abd27b3f588ba9bc84073"}
Oct 31 12:27:10.967611 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:10.967575    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-2bmzr" event={"ID":"a2b450be-ca79-4edf-a1ea-377531a49519","Type":"ContainerStarted","Data":"6aaae0ad5cd357b195d055aaf85b96bed388d3a4f75a1ce9caad08ede506e3c7"}
Oct 31 12:27:10.968915 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:10.968847    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-tdrmt" event={"ID":"7a402472-ca33-44ae-b598-96fc8e7a5eb2","Type":"ContainerStarted","Data":"8dbe3066e41c634d3607763741f771139a4be4a86fb97f302e275ecb6c7399bd"}
Oct 31 12:27:10.971811 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:10.971790    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pcvvm_f032909d-bc39-4c9e-99b7-f3d800842119/console-operator/0.log"
Oct 31 12:27:10.971948 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:10.971824    2576 generic.go:356] "Generic (PLEG): container finished" podID="f032909d-bc39-4c9e-99b7-f3d800842119" containerID="bb311565191af25ddc49de34e1d78a348c1fc59de63bb82795e195728db4edb4" exitCode=255
Oct 31 12:27:10.971948 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:10.971854    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-pcvvm" event={"ID":"f032909d-bc39-4c9e-99b7-f3d800842119","Type":"ContainerDied","Data":"bb311565191af25ddc49de34e1d78a348c1fc59de63bb82795e195728db4edb4"}
Oct 31 12:27:10.972462 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:10.972118    2576 scope.go:122] "RemoveContainer" containerID="bb311565191af25ddc49de34e1d78a348c1fc59de63bb82795e195728db4edb4"
Oct 31 12:27:10.988579 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:10.988546    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-87ncb-cnk5g"]
Oct 31 12:27:10.988717 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:10.988668    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-vqsv9" podStartSLOduration=392.489985884 podStartE2EDuration="6m36.988654695s" totalImagesPullingTime="4.498668811s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:20:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:27:10.98700232 +0000 UTC m=+35.879649473" watchObservedRunningTime="2026-10-31 12:27:10.988654695 +0000 UTC m=+35.881301846"
Oct 31 12:27:10.993388 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:10.993364    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-skkg4"]
Oct 31 12:27:11.014740 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:11.013933    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-skkg4"]
Oct 31 12:27:11.014740 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:11.013978    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-m4fpw"]
Oct 31 12:27:11.014740 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:11.014045    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-skkg4"
Oct 31 12:27:11.020020 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:11.019991    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Oct 31 12:27:11.025521 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:11.024943    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-b9z2w\""
Oct 31 12:27:11.027267 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:11.027244    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Oct 31 12:27:11.029680 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:11.029660    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-m4fpw"]
Oct 31 12:27:11.029805 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:11.029784    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-m4fpw"
Oct 31 12:27:11.044459 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:11.044436    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48lnl\" (UniqueName: \"kubernetes.io/projected/64649bfe-c3e6-469a-a9ff-0db1ca0fadb3-kube-api-access-48lnl\") pod \"migrator-66df5d496b-skkg4\" (UID: \"64649bfe-c3e6-469a-a9ff-0db1ca0fadb3\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-skkg4"
Oct 31 12:27:11.044766 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:11.044746    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpcl9\" (UniqueName: \"kubernetes.io/projected/031551df-64a5-411b-8eb8-7f0bf2d4f170-kube-api-access-wpcl9\") pod \"migrator-66df5d496b-m4fpw\" (UID: \"031551df-64a5-411b-8eb8-7f0bf2d4f170\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-m4fpw"
Oct 31 12:27:11.145891 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:11.145857    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48lnl\" (UniqueName: \"kubernetes.io/projected/64649bfe-c3e6-469a-a9ff-0db1ca0fadb3-kube-api-access-48lnl\") pod \"migrator-66df5d496b-skkg4\" (UID: \"64649bfe-c3e6-469a-a9ff-0db1ca0fadb3\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-skkg4"
Oct 31 12:27:11.146065 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:11.145993    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpcl9\" (UniqueName: \"kubernetes.io/projected/031551df-64a5-411b-8eb8-7f0bf2d4f170-kube-api-access-wpcl9\") pod \"migrator-66df5d496b-m4fpw\" (UID: \"031551df-64a5-411b-8eb8-7f0bf2d4f170\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-m4fpw"
Oct 31 12:27:11.158066 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:11.158032    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48lnl\" (UniqueName: \"kubernetes.io/projected/64649bfe-c3e6-469a-a9ff-0db1ca0fadb3-kube-api-access-48lnl\") pod \"migrator-66df5d496b-skkg4\" (UID: \"64649bfe-c3e6-469a-a9ff-0db1ca0fadb3\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-skkg4"
Oct 31 12:27:11.158195 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:11.158119    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpcl9\" (UniqueName: \"kubernetes.io/projected/031551df-64a5-411b-8eb8-7f0bf2d4f170-kube-api-access-wpcl9\") pod \"migrator-66df5d496b-m4fpw\" (UID: \"031551df-64a5-411b-8eb8-7f0bf2d4f170\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-m4fpw"
Oct 31 12:27:11.208862 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:27:11.208827    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb20e317_3ebe_4f5c_9b37_1de9ea5f063f.slice/crio-8738d804fa86e15f0597349bd890e4bfa1c5c1415beaacbab01aad9c000fbd0a WatchSource:0}: Error finding container 8738d804fa86e15f0597349bd890e4bfa1c5c1415beaacbab01aad9c000fbd0a: Status 404 returned error can't find the container with id 8738d804fa86e15f0597349bd890e4bfa1c5c1415beaacbab01aad9c000fbd0a
Oct 31 12:27:11.344609 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:11.344571    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-skkg4"
Oct 31 12:27:11.366919 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:11.366890    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-m4fpw"
Oct 31 12:27:11.762281 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:11.761761    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-skkg4"]
Oct 31 12:27:11.768078 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:11.768042    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-m4fpw"]
Oct 31 12:27:11.801345 ip-10-0-130-190 kubenswrapper[2576]: W1031 12:27:11.801304    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod031551df_64a5_411b_8eb8_7f0bf2d4f170.slice/crio-c8dd1a20232d5b64644dd25931795726b9f8c36ada12fd4da9078653557feddd WatchSource:0}: Error finding container c8dd1a20232d5b64644dd25931795726b9f8c36ada12fd4da9078653557feddd: Status 404 returned error can't find the container with id c8dd1a20232d5b64644dd25931795726b9f8c36ada12fd4da9078653557feddd
Oct 31 12:27:11.968349 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:11.968249    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9557a3f0-ae0f-41f9-a9b5-15aeda410a38-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p7phb\" (UID: \"9557a3f0-ae0f-41f9-a9b5-15aeda410a38\") " pod="openshift-insights/insights-runtime-extractor-p7phb"
Oct 31 12:27:11.968848 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:27:11.968432    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 31 12:27:11.968848 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:11.968478    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9557a3f0-ae0f-41f9-a9b5-15aeda410a38-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p7phb\" (UID: \"9557a3f0-ae0f-41f9-a9b5-15aeda410a38\") " pod="openshift-insights/insights-runtime-extractor-p7phb"
Oct 31 12:27:11.968848 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:27:11.968552    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9557a3f0-ae0f-41f9-a9b5-15aeda410a38-insights-runtime-extractor-tls podName:9557a3f0-ae0f-41f9-a9b5-15aeda410a38 nodeName:}" failed. No retries permitted until 2026-10-31 12:27:13.968529064 +0000 UTC m=+38.861176216 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9557a3f0-ae0f-41f9-a9b5-15aeda410a38-insights-runtime-extractor-tls") pod "insights-runtime-extractor-p7phb" (UID: "9557a3f0-ae0f-41f9-a9b5-15aeda410a38") : secret "insights-runtime-extractor-tls" not found
Oct 31 12:27:11.968848 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:27:11.968608    2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 31 12:27:11.968848 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:27:11.968658    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9557a3f0-ae0f-41f9-a9b5-15aeda410a38-kube-rbac-proxy-cm podName:9557a3f0-ae0f-41f9-a9b5-15aeda410a38 nodeName:}" failed. No retries permitted until 2026-10-31 12:27:13.968647317 +0000 UTC m=+38.861294449 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9557a3f0-ae0f-41f9-a9b5-15aeda410a38-kube-rbac-proxy-cm") pod "insights-runtime-extractor-p7phb" (UID: "9557a3f0-ae0f-41f9-a9b5-15aeda410a38") : configmap "kube-rbac-proxy" not found
Oct 31 12:27:11.980121 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:11.980084    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-m9xxh" event={"ID":"25691fa3-7017-429a-b94c-c2dde4d29e55","Type":"ContainerStarted","Data":"3acdf6895d66ad573641d4b1a7e303fd16c4735b0ad20e193f3b40b44a887dca"}
Oct 31 12:27:11.983045 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:11.983013    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-tdrmt" event={"ID":"7a402472-ca33-44ae-b598-96fc8e7a5eb2","Type":"ContainerStarted","Data":"b834e7ef8dcc1592b723e272170efd8d9a963f1e8969f51f6e16b6ca6d45e24c"}
Oct 31 12:27:11.983186 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:11.983070    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-tdrmt"
Oct 31 12:27:11.984832 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:11.984787    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-87ncb-cnk5g" event={"ID":"eb20e317-3ebe-4f5c-9b37-1de9ea5f063f","Type":"ContainerStarted","Data":"32de846f50254cf1d3b4006aa0d3d1ce3771e004167002ba1b955de5afb8eebe"}
Oct 31 12:27:11.984929 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:11.984857    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-87ncb-cnk5g" event={"ID":"eb20e317-3ebe-4f5c-9b37-1de9ea5f063f","Type":"ContainerStarted","Data":"8738d804fa86e15f0597349bd890e4bfa1c5c1415beaacbab01aad9c000fbd0a"}
Oct 31 12:27:11.986496 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:11.986474    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pcvvm_f032909d-bc39-4c9e-99b7-f3d800842119/console-operator/1.log"
Oct 31 12:27:11.986892 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:11.986873    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pcvvm_f032909d-bc39-4c9e-99b7-f3d800842119/console-operator/0.log"
Oct 31 12:27:11.986998 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:11.986909    2576 generic.go:356] "Generic (PLEG): container finished" podID="f032909d-bc39-4c9e-99b7-f3d800842119" containerID="c98998f609d234b42991ed336e4bc8d348b0169b157e725e00a2b4bc0c1a44a4" exitCode=255
Oct 31 12:27:11.986998 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:11.986965    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-pcvvm" event={"ID":"f032909d-bc39-4c9e-99b7-f3d800842119","Type":"ContainerDied","Data":"c98998f609d234b42991ed336e4bc8d348b0169b157e725e00a2b4bc0c1a44a4"}
Oct 31 12:27:11.987097 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:11.987008    2576 scope.go:122] "RemoveContainer" containerID="bb311565191af25ddc49de34e1d78a348c1fc59de63bb82795e195728db4edb4"
Oct 31 12:27:11.987194 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:11.987178    2576 scope.go:122] "RemoveContainer" containerID="c98998f609d234b42991ed336e4bc8d348b0169b157e725e00a2b4bc0c1a44a4"
Oct 31 12:27:11.987381 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:27:11.987360    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-pcvvm_openshift-console-operator(f032909d-bc39-4c9e-99b7-f3d800842119)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-pcvvm" podUID="f032909d-bc39-4c9e-99b7-f3d800842119"
Oct 31 12:27:11.988330 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:11.988233    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-skkg4" event={"ID":"64649bfe-c3e6-469a-a9ff-0db1ca0fadb3","Type":"ContainerStarted","Data":"c9ca40edbf0dd49e6bf694363da3d39a56d8bd8ff7baef573fe1d91ab10933b6"}
Oct 31 12:27:11.989961 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:11.989935    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-rl55s" event={"ID":"891b3c47-ff7c-4a7a-a09b-411361a843f7","Type":"ContainerStarted","Data":"36a056be511cf5a6ca6ceb18d6123f79b6fd44e16b76cce301705b5fcc0423c9"}
Oct 31 12:27:11.991171 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:11.991145    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-m4fpw" event={"ID":"031551df-64a5-411b-8eb8-7f0bf2d4f170","Type":"ContainerStarted","Data":"c8dd1a20232d5b64644dd25931795726b9f8c36ada12fd4da9078653557feddd"}
Oct 31 12:27:11.992716 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:11.992691    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-c789s" event={"ID":"99a606a8-546b-4145-8bae-a95061241069","Type":"ContainerStarted","Data":"7ee4f366c61fbdc3c2de10a9399dd08e4a45086da64aadf86ea70e41c05ffafb"}
Oct 31 12:27:11.999830 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:11.999782    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-m9xxh" podStartSLOduration=386.229396908 podStartE2EDuration="6m31.999767979s" totalImagesPullingTime="5.770371071s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:20:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:27:11.999635559 +0000 UTC m=+36.892282710" watchObservedRunningTime="2026-10-31 12:27:11.999767979 +0000 UTC m=+36.892415131"
Oct 31 12:27:12.020388 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:12.020334    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-rl55s" podStartSLOduration=2.238237388 podStartE2EDuration="8.020315814s" totalImagesPullingTime="5.782078426s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:27:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:27:12.019971057 +0000 UTC m=+36.912618209" watchObservedRunningTime="2026-10-31 12:27:12.020315814 +0000 UTC m=+36.912962967"
Oct 31 12:27:12.058629 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:12.057430    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-tdrmt" podStartSLOduration=34.803765003 podStartE2EDuration="36.057417094s" totalImagesPullingTime="1.253652091s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:26:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:27:12.057114055 +0000 UTC m=+36.949761206" watchObservedRunningTime="2026-10-31 12:27:12.057417094 +0000 UTC m=+36.950064240"
Oct 31 12:27:12.058629 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:12.058560    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-87ncb-cnk5g" podStartSLOduration=2.058548806 podStartE2EDuration="2.058548806s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:27:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:27:12.040047402 +0000 UTC m=+36.932694555" watchObservedRunningTime="2026-10-31 12:27:12.058548806 +0000 UTC m=+36.951195954"
Oct 31 12:27:12.079973 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:12.077652    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-c789s" podStartSLOduration=391.298651182 podStartE2EDuration="6m37.077635179s" totalImagesPullingTime="5.778983997s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:20:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:27:12.077430103 +0000 UTC m=+36.970077255" watchObservedRunningTime="2026-10-31 12:27:12.077635179 +0000 UTC m=+36.970282330"
Oct 31 12:27:12.778229 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:12.778188    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/556dda7b-d8e6-4558-b5f3-994b01d6a866-service-ca-bundle\") pod \"router-default-659589798c-776bx\" (UID: \"556dda7b-d8e6-4558-b5f3-994b01d6a866\") " pod="openshift-ingress/router-default-659589798c-776bx"
Oct 31 12:27:12.778229 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:12.778231    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5c8865f9-fdf6-49b9-8b4f-69dbc7787084-metrics-tls\") pod \"dns-default-r6fvt\" (UID: \"5c8865f9-fdf6-49b9-8b4f-69dbc7787084\") " pod="openshift-dns/dns-default-r6fvt"
Oct 31 12:27:12.778436 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:12.778295    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9083c35a-9739-4f3e-8415-bad1e6433d40-registry-tls\") pod \"image-registry-6cfdbfcdc4-2rq84\" (UID: \"9083c35a-9739-4f3e-8415-bad1e6433d40\") " pod="openshift-image-registry/image-registry-6cfdbfcdc4-2rq84"
Oct 31 12:27:12.778436 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:12.778323    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0db116b4-d971-4966-bb05-db28d077b6fe-registry-tls\") pod \"image-registry-6dd647c7b7-9255m\" (UID: \"0db116b4-d971-4966-bb05-db28d077b6fe\") " pod="openshift-image-registry/image-registry-6dd647c7b7-9255m"
Oct 31 12:27:12.778436 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:12.778344    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/dbbd2d70-8911-4aa5-96cf-3a16dcdf246d-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-tcnrw\" (UID: \"dbbd2d70-8911-4aa5-96cf-3a16dcdf246d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-tcnrw"
Oct 31 12:27:12.778436 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:12.778361    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/dda0c26f-8225-4239-bc27-866b66cbb919-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-gt9rn\" (UID: \"dda0c26f-8225-4239-bc27-866b66cbb919\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gt9rn"
Oct 31 12:27:12.778436 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:12.778383    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/556dda7b-d8e6-4558-b5f3-994b01d6a866-metrics-certs\") pod \"router-default-659589798c-776bx\" (UID: \"556dda7b-d8e6-4558-b5f3-994b01d6a866\") " pod="openshift-ingress/router-default-659589798c-776bx"
Oct 31 12:27:12.778436 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:27:12.778404    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/556dda7b-d8e6-4558-b5f3-994b01d6a866-service-ca-bundle podName:556dda7b-d8e6-4558-b5f3-994b01d6a866 nodeName:}" failed. No retries permitted until 2026-10-31 12:27:20.778381563 +0000 UTC m=+45.671028710 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/556dda7b-d8e6-4558-b5f3-994b01d6a866-service-ca-bundle") pod "router-default-659589798c-776bx" (UID: "556dda7b-d8e6-4558-b5f3-994b01d6a866") : configmap references non-existent config key: service-ca.crt
Oct 31 12:27:12.778663 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:27:12.778460    2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 31 12:27:12.778663 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:12.778464    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/50ad7816-358d-461e-a9bd-fadce6246e8f-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-2bfvp\" (UID: \"50ad7816-358d-461e-a9bd-fadce6246e8f\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-2bfvp"
Oct 31 12:27:12.778663 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:27:12.778527    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/556dda7b-d8e6-4558-b5f3-994b01d6a866-metrics-certs podName:556dda7b-d8e6-4558-b5f3-994b01d6a866 nodeName:}" failed. No retries permitted until 2026-10-31 12:27:20.778516384 +0000 UTC m=+45.671163512 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/556dda7b-d8e6-4558-b5f3-994b01d6a866-metrics-certs") pod "router-default-659589798c-776bx" (UID: "556dda7b-d8e6-4558-b5f3-994b01d6a866") : secret "router-metrics-certs-default" not found
Oct 31 12:27:12.778663 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:27:12.778582    2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 31 12:27:12.778663 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:27:12.778602    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 31 12:27:12.778663 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:27:12.778620    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50ad7816-358d-461e-a9bd-fadce6246e8f-networking-console-plugin-cert podName:50ad7816-358d-461e-a9bd-fadce6246e8f nodeName:}" failed. No retries permitted until 2026-10-31 12:27:20.778608618 +0000 UTC m=+45.671255750 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/50ad7816-358d-461e-a9bd-fadce6246e8f-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-2bfvp" (UID: "50ad7816-358d-461e-a9bd-fadce6246e8f") : secret "networking-console-plugin-cert" not found
Oct 31 12:27:12.778663 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:27:12.778637    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5c8865f9-fdf6-49b9-8b4f-69dbc7787084-metrics-tls podName:5c8865f9-fdf6-49b9-8b4f-69dbc7787084 nodeName:}" failed. No retries permitted until 2026-10-31 12:27:20.778627535 +0000 UTC m=+45.671274684 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5c8865f9-fdf6-49b9-8b4f-69dbc7787084-metrics-tls") pod "dns-default-r6fvt" (UID: "5c8865f9-fdf6-49b9-8b4f-69dbc7787084") : secret "dns-default-metrics-tls" not found
Oct 31 12:27:12.778663 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:27:12.778656    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 31 12:27:12.778663 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:27:12.778666    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6cfdbfcdc4-2rq84: secret "image-registry-tls" not found
Oct 31 12:27:12.778922 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:27:12.778690    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9083c35a-9739-4f3e-8415-bad1e6433d40-registry-tls podName:9083c35a-9739-4f3e-8415-bad1e6433d40 nodeName:}" failed. No retries permitted until 2026-10-31 12:27:20.778683005 +0000 UTC m=+45.671330165 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9083c35a-9739-4f3e-8415-bad1e6433d40-registry-tls") pod "image-registry-6cfdbfcdc4-2rq84" (UID: "9083c35a-9739-4f3e-8415-bad1e6433d40") : secret "image-registry-tls" not found
Oct 31 12:27:12.778922 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:27:12.778731    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 31 12:27:12.778922 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:27:12.778737    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6dd647c7b7-9255m: secret "image-registry-tls" not found
Oct 31 12:27:12.778922 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:27:12.778756    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0db116b4-d971-4966-bb05-db28d077b6fe-registry-tls podName:0db116b4-d971-4966-bb05-db28d077b6fe nodeName:}" failed. No retries permitted until 2026-10-31 12:27:20.778749615 +0000 UTC m=+45.671396780 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0db116b4-d971-4966-bb05-db28d077b6fe-registry-tls") pod "image-registry-6dd647c7b7-9255m" (UID: "0db116b4-d971-4966-bb05-db28d077b6fe") : secret "image-registry-tls" not found
Oct 31 12:27:12.778922 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:27:12.778792    2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 31 12:27:12.778922 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:27:12.778817    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dbbd2d70-8911-4aa5-96cf-3a16dcdf246d-samples-operator-tls podName:dbbd2d70-8911-4aa5-96cf-3a16dcdf246d nodeName:}" failed. No retries permitted until 2026-10-31 12:27:20.778807401 +0000 UTC m=+45.671454544 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/dbbd2d70-8911-4aa5-96cf-3a16dcdf246d-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-tcnrw" (UID: "dbbd2d70-8911-4aa5-96cf-3a16dcdf246d") : secret "samples-operator-tls" not found
Oct 31 12:27:12.778922 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:27:12.778857    2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 31 12:27:12.778922 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:27:12.778879    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dda0c26f-8225-4239-bc27-866b66cbb919-cluster-monitoring-operator-tls podName:dda0c26f-8225-4239-bc27-866b66cbb919 nodeName:}" failed. No retries permitted until 2026-10-31 12:27:20.778871872 +0000 UTC m=+45.671519003 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/dda0c26f-8225-4239-bc27-866b66cbb919-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-gt9rn" (UID: "dda0c26f-8225-4239-bc27-866b66cbb919") : secret "cluster-monitoring-operator-tls" not found
Oct 31 12:27:12.834056 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:12.834020    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-lmwkw"]
Oct 31 12:27:12.857969 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:12.856423    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-lmwkw"]
Oct 31 12:27:12.857969 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:12.856596    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-lmwkw"
Oct 31 12:27:12.861258 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:12.861232    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Oct 31 12:27:12.862631 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:12.862570    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Oct 31 12:27:12.862758 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:12.862697    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Oct 31 12:27:12.862966 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:12.862824    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-t2vx8\""
Oct 31 12:27:12.862966 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:12.862935    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Oct 31 12:27:12.981531 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:12.981130    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bm8x9\" (UniqueName: \"kubernetes.io/projected/b9ea9813-55b9-454b-8f10-985533305f85-kube-api-access-bm8x9\") pod \"service-ca-5fffc97bdf-lmwkw\" (UID: \"b9ea9813-55b9-454b-8f10-985533305f85\") " pod="openshift-service-ca/service-ca-5fffc97bdf-lmwkw"
Oct 31 12:27:12.981531 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:12.981225    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/b9ea9813-55b9-454b-8f10-985533305f85-signing-cabundle\") pod \"service-ca-5fffc97bdf-lmwkw\" (UID: \"b9ea9813-55b9-454b-8f10-985533305f85\") " pod="openshift-service-ca/service-ca-5fffc97bdf-lmwkw"
Oct 31 12:27:12.981531 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:12.981340    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/b9ea9813-55b9-454b-8f10-985533305f85-signing-key\") pod \"service-ca-5fffc97bdf-lmwkw\" (UID: \"b9ea9813-55b9-454b-8f10-985533305f85\") " pod="openshift-service-ca/service-ca-5fffc97bdf-lmwkw"
Oct 31 12:27:12.999991 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:12.999961    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pcvvm_f032909d-bc39-4c9e-99b7-f3d800842119/console-operator/1.log"
Oct 31 12:27:13.000809 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:13.000786    2576 scope.go:122] "RemoveContainer" containerID="c98998f609d234b42991ed336e4bc8d348b0169b157e725e00a2b4bc0c1a44a4"
Oct 31 12:27:13.001000 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:27:13.000972    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-pcvvm_openshift-console-operator(f032909d-bc39-4c9e-99b7-f3d800842119)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-pcvvm" podUID="f032909d-bc39-4c9e-99b7-f3d800842119"
Oct 31 12:27:13.082237 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:13.082152    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bm8x9\" (UniqueName: \"kubernetes.io/projected/b9ea9813-55b9-454b-8f10-985533305f85-kube-api-access-bm8x9\") pod \"service-ca-5fffc97bdf-lmwkw\" (UID: \"b9ea9813-55b9-454b-8f10-985533305f85\") " pod="openshift-service-ca/service-ca-5fffc97bdf-lmwkw"
Oct 31 12:27:13.082237 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:13.082221    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/b9ea9813-55b9-454b-8f10-985533305f85-signing-cabundle\") pod \"service-ca-5fffc97bdf-lmwkw\" (UID: \"b9ea9813-55b9-454b-8f10-985533305f85\") " pod="openshift-service-ca/service-ca-5fffc97bdf-lmwkw"
Oct 31 12:27:13.082420 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:13.082345    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/b9ea9813-55b9-454b-8f10-985533305f85-signing-key\") pod \"service-ca-5fffc97bdf-lmwkw\" (UID: \"b9ea9813-55b9-454b-8f10-985533305f85\") " pod="openshift-service-ca/service-ca-5fffc97bdf-lmwkw"
Oct 31 12:27:13.083353 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:13.083320    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/b9ea9813-55b9-454b-8f10-985533305f85-signing-cabundle\") pod \"service-ca-5fffc97bdf-lmwkw\" (UID: \"b9ea9813-55b9-454b-8f10-985533305f85\") " pod="openshift-service-ca/service-ca-5fffc97bdf-lmwkw"
Oct 31 12:27:13.086145 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:13.086119    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/b9ea9813-55b9-454b-8f10-985533305f85-signing-key\") pod \"service-ca-5fffc97bdf-lmwkw\" (UID: \"b9ea9813-55b9-454b-8f10-985533305f85\") " pod="openshift-service-ca/service-ca-5fffc97bdf-lmwkw"
Oct 31 12:27:13.094723 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:13.094698    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bm8x9\" (UniqueName: \"kubernetes.io/projected/b9ea9813-55b9-454b-8f10-985533305f85-kube-api-access-bm8x9\") pod \"service-ca-5fffc97bdf-lmwkw\" (UID: \"b9ea9813-55b9-454b-8f10-985533305f85\") " pod="openshift-service-ca/service-ca-5fffc97bdf-lmwkw"
Oct 31 12:27:13.183255 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:13.182980    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-lmwkw"
Oct 31 12:27:13.484452 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:13.484374    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-lmwkw"]
Oct 31 12:27:13.993272 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:13.993232    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9557a3f0-ae0f-41f9-a9b5-15aeda410a38-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p7phb\" (UID: \"9557a3f0-ae0f-41f9-a9b5-15aeda410a38\") " pod="openshift-insights/insights-runtime-extractor-p7phb"
Oct 31 12:27:13.993697 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:13.993299    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9557a3f0-ae0f-41f9-a9b5-15aeda410a38-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p7phb\" (UID: \"9557a3f0-ae0f-41f9-a9b5-15aeda410a38\") " pod="openshift-insights/insights-runtime-extractor-p7phb"
Oct 31 12:27:13.993697 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:27:13.993385    2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 31 12:27:13.993697 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:27:13.993425    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 31 12:27:13.993697 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:27:13.993469    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9557a3f0-ae0f-41f9-a9b5-15aeda410a38-kube-rbac-proxy-cm podName:9557a3f0-ae0f-41f9-a9b5-15aeda410a38 nodeName:}" failed. No retries permitted until 2026-10-31 12:27:17.993447791 +0000 UTC m=+42.886094924 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9557a3f0-ae0f-41f9-a9b5-15aeda410a38-kube-rbac-proxy-cm") pod "insights-runtime-extractor-p7phb" (UID: "9557a3f0-ae0f-41f9-a9b5-15aeda410a38") : configmap "kube-rbac-proxy" not found
Oct 31 12:27:13.993697 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:27:13.993502    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9557a3f0-ae0f-41f9-a9b5-15aeda410a38-insights-runtime-extractor-tls podName:9557a3f0-ae0f-41f9-a9b5-15aeda410a38 nodeName:}" failed. No retries permitted until 2026-10-31 12:27:17.993480493 +0000 UTC m=+42.886127628 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9557a3f0-ae0f-41f9-a9b5-15aeda410a38-insights-runtime-extractor-tls") pod "insights-runtime-extractor-p7phb" (UID: "9557a3f0-ae0f-41f9-a9b5-15aeda410a38") : secret "insights-runtime-extractor-tls" not found
Oct 31 12:27:14.005682 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:14.005639    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-lmwkw" event={"ID":"b9ea9813-55b9-454b-8f10-985533305f85","Type":"ContainerStarted","Data":"b59c0ff266b8a213c62a1257a4320313a667acab9c455c93a31e103773b87e7c"}
Oct 31 12:27:14.005850 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:14.005708    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-lmwkw" event={"ID":"b9ea9813-55b9-454b-8f10-985533305f85","Type":"ContainerStarted","Data":"f5a0fca700cd53f8f083df04ab0ae9b664259a71e5b410fd4990db2d3ea0b60b"}
Oct 31 12:27:14.024701 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:14.024648    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca/service-ca-5fffc97bdf-lmwkw" podStartSLOduration=2.024601427 podStartE2EDuration="2.024601427s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:27:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:27:14.02340956 +0000 UTC m=+38.916056711" watchObservedRunningTime="2026-10-31 12:27:14.024601427 +0000 UTC m=+38.917248577"
Oct 31 12:27:14.646661 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:14.646633    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-v9ks4_6bbe13aa-df4b-40b0-8595-68fe1166a082/dns-node-resolver/0.log"
Oct 31 12:27:15.011774 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:15.011692    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-skkg4" event={"ID":"64649bfe-c3e6-469a-a9ff-0db1ca0fadb3","Type":"ContainerStarted","Data":"d88403ada24c0b4bb97a6330268dbb99d861cf7342ac46780ff0ff2f62d34d47"}
Oct 31 12:27:15.013651 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:15.013614    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-m4fpw" event={"ID":"031551df-64a5-411b-8eb8-7f0bf2d4f170","Type":"ContainerStarted","Data":"7f4c14ee63a56204dba362920fc10880f081581c495c73312a1cc10c27c6ec90"}
Oct 31 12:27:15.031651 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:15.031589    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-skkg4" podStartSLOduration=2.594759389 podStartE2EDuration="5.031573817s" totalImagesPullingTime="2.436814428s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:27:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:27:15.031443491 +0000 UTC m=+39.924090644" watchObservedRunningTime="2026-10-31 12:27:15.031573817 +0000 UTC m=+39.924220970"
Oct 31 12:27:15.050630 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:15.050572    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-m4fpw" podStartSLOduration=2.615216972 podStartE2EDuration="5.050552447s" totalImagesPullingTime="2.435335475s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:27:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:27:15.048793204 +0000 UTC m=+39.941440356" watchObservedRunningTime="2026-10-31 12:27:15.050552447 +0000 UTC m=+39.943199599"
Oct 31 12:27:15.311626 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:15.311587    2576 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-pcvvm"
Oct 31 12:27:15.311796 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:15.311660    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-pcvvm"
Oct 31 12:27:15.312057 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:15.312040    2576 scope.go:122] "RemoveContainer" containerID="c98998f609d234b42991ed336e4bc8d348b0169b157e725e00a2b4bc0c1a44a4"
Oct 31 12:27:15.312279 ip-10-0-130-190 kubenswrapper[2576]: E1031 12:27:15.312232    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-pcvvm_openshift-console-operator(f032909d-bc39-4c9e-99b7-f3d800842119)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-pcvvm" podUID="f032909d-bc39-4c9e-99b7-f3d800842119"
Oct 31 12:27:15.846382 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:15.846349    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-wdkzp_6c661ce8-9f7b-4509-bd17-2380bcd1c425/node-ca/0.log"
Oct 31 12:27:16.246773 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:16.246703    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-rl55s_891b3c47-ff7c-4a7a-a09b-411361a843f7/serve-healthcheck-canary/0.log"
Oct 31 12:27:16.446946 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:16.446919    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-m4fpw_031551df-64a5-411b-8eb8-7f0bf2d4f170/migrator/0.log"
Oct 31 12:27:16.646649 ip-10-0-130-190 kubenswrapper[2576]: I1031 12:27:16.646620    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-skkg4_64649bfe-c3e6-469a-a9ff-0db1ca0fadb3/migrator/0.log"
Oct 31 12:27:16.834413 ip-10-0-130-190 systemd[1]: Stopping Kubernetes Kubelet...
Oct 31 12:27:16.848941 ip-10-0-130-190 systemd[1]: kubelet.service: Deactivated successfully.
Oct 31 12:27:16.849117 ip-10-0-130-190 systemd[1]: Stopped Kubernetes Kubelet.
Oct 31 12:27:16.849168 ip-10-0-130-190 systemd[1]: kubelet.service: Consumed 3.193s CPU time.
Oct 31 12:27:16.864271 ip-10-0-130-190 systemd[1]: Starting Kubernetes Kubelet...
Oct 31 12:27:16.955758 ip-10-0-130-190 kubenswrapper[5060]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 12:27:16.955758 ip-10-0-130-190 kubenswrapper[5060]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 31 12:27:16.955758 ip-10-0-130-190 kubenswrapper[5060]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 12:27:16.955758 ip-10-0-130-190 kubenswrapper[5060]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 12:27:16.959032 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.958928    5060 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 12:27:16.959032 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.959032    5060 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 12:27:16.959114 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.959038    5060 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 12:27:16.959114 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.959044    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 12:27:16.959114 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.959049    5060 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 12:27:16.959114 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.959054    5060 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 12:27:16.959114 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.959058    5060 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 12:27:16.959114 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.959063    5060 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 12:27:16.959114 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.959068    5060 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 12:27:16.959114 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.959072    5060 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 12:27:16.959114 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.959077    5060 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 12:27:16.959114 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.959082    5060 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 12:27:16.959114 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.959086    5060 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 12:27:16.959114 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.959090    5060 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 12:27:16.959114 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.959094    5060 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 12:27:16.959114 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.959099    5060 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 12:27:16.959114 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.959103    5060 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 12:27:16.959114 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.959107    5060 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 12:27:16.959114 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.959112    5060 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 12:27:16.959114 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.959116    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 12:27:16.959114 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.959120    5060 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 12:27:16.959673 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.959134    5060 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 12:27:16.959673 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.959139    5060 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 12:27:16.959673 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.959143    5060 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 12:27:16.959673 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.959147    5060 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 12:27:16.959673 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.959151    5060 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 12:27:16.959673 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.959156    5060 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 12:27:16.959673 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.959160    5060 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 12:27:16.959673 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.959164    5060 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 12:27:16.959673 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.959168    5060 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 12:27:16.959673 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.959172    5060 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 12:27:16.959673 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.959176    5060 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 12:27:16.959673 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.959181    5060 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 12:27:16.959673 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.959185    5060 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 12:27:16.959673 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.959189    5060 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 12:27:16.959673 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.959196    5060 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 12:27:16.959673 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.959203    5060 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 12:27:16.959673 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.959209    5060 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 12:27:16.959673 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.959214    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 12:27:16.959673 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.959219    5060 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 12:27:16.960138 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.959224    5060 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 12:27:16.960138 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.959228    5060 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 12:27:16.960138 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.959232    5060 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 12:27:16.960138 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.959236    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 12:27:16.960138 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.959240    5060 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 12:27:16.960138 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.959244    5060 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 12:27:16.960138 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.959248    5060 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 12:27:16.960138 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.959253    5060 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 12:27:16.960138 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.959257    5060 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 12:27:16.960138 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.959261    5060 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 12:27:16.960138 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.959265    5060 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 12:27:16.960138 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.959270    5060 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 12:27:16.960138 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.959274    5060 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 12:27:16.960138 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.959278    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 12:27:16.960138 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.959283    5060 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 12:27:16.960138 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.959291    5060 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 12:27:16.960138 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.959296    5060 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 12:27:16.960138 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.959301    5060 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 12:27:16.960138 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.959306    5060 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 12:27:16.960638 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.959310    5060 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 12:27:16.960638 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.959314    5060 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 12:27:16.960638 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.959318    5060 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 12:27:16.960638 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.959323    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 12:27:16.960638 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.959327    5060 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 12:27:16.960638 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.959331    5060 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 12:27:16.960638 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.959335    5060 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 12:27:16.960638 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.959339    5060 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 12:27:16.960638 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.959344    5060 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 12:27:16.960638 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.959348    5060 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 12:27:16.960638 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.959352    5060 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 12:27:16.960638 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.959356    5060 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 12:27:16.960638 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.959360    5060 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 12:27:16.960638 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.959365    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 12:27:16.960638 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.959370    5060 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 12:27:16.960638 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.959374    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 12:27:16.960638 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.959377    5060 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 12:27:16.960638 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.959381    5060 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 12:27:16.960638 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.959385    5060 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 12:27:16.960638 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.959389    5060 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 12:27:16.961165 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.959394    5060 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 12:27:16.961165 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.959398    5060 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 12:27:16.961165 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.959403    5060 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 12:27:16.961165 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.959408    5060 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 12:27:16.961165 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.959414    5060 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 12:27:16.961165 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.959418    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 12:27:16.961165 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.959423    5060 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 12:27:16.961165 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.959427    5060 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 12:27:16.961165 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.959431    5060 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 12:27:16.961165 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.959435    5060 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 12:27:16.961165 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.959440    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 12:27:16.961165 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.959444    5060 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 12:27:16.961165 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.959448    5060 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 12:27:16.961165 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.959452    5060 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 12:27:16.961165 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.959456    5060 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 12:27:16.961165 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.959460    5060 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 12:27:16.961165 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.959465    5060 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 12:27:16.961165 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.959469    5060 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 12:27:16.961165 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.959473    5060 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 12:27:16.961653 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.959477    5060 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 12:27:16.961653 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.959481    5060 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 12:27:16.961653 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.959485    5060 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 12:27:16.961653 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.959504    5060 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 12:27:16.961653 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.959508    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 12:27:16.961653 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.959512    5060 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 12:27:16.961653 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.959518    5060 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 12:27:16.961653 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.959522    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 12:27:16.961653 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.960737    5060 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 12:27:16.961653 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.960747    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 12:27:16.961653 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.960751    5060 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 12:27:16.961653 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.960757    5060 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 12:27:16.961653 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.960762    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 12:27:16.961653 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.960766    5060 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 12:27:16.961653 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.960770    5060 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 12:27:16.961653 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.960775    5060 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 12:27:16.961653 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.960779    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 12:27:16.961653 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.960783    5060 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 12:27:16.961653 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.960788    5060 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 12:27:16.961653 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.960792    5060 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 12:27:16.962174 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.960796    5060 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 12:27:16.962174 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.960800    5060 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 12:27:16.962174 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.960804    5060 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 12:27:16.962174 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.960808    5060 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 12:27:16.962174 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.960813    5060 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 12:27:16.962174 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.960817    5060 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 12:27:16.962174 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.960821    5060 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 12:27:16.962174 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.960825    5060 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 12:27:16.962174 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.960829    5060 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 12:27:16.962174 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.960834    5060 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 12:27:16.962174 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.960838    5060 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 12:27:16.962174 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.960842    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 12:27:16.962174 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.960847    5060 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 12:27:16.962174 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.960851    5060 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 12:27:16.962174 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.960855    5060 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 12:27:16.962174 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.960859    5060 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 12:27:16.962174 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.960863    5060 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 12:27:16.962174 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.960867    5060 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 12:27:16.962174 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.960871    5060 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 12:27:16.962174 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.960876    5060 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 12:27:16.962694 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.960881    5060 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 12:27:16.962694 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.960885    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 12:27:16.962694 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.960890    5060 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 12:27:16.962694 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.960894    5060 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 12:27:16.962694 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.960898    5060 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 12:27:16.962694 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.960902    5060 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 12:27:16.962694 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.960906    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 12:27:16.962694 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.960910    5060 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 12:27:16.962694 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.960914    5060 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 12:27:16.962694 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.960919    5060 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 12:27:16.962694 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.960923    5060 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 12:27:16.962694 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.960927    5060 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 12:27:16.962694 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.960931    5060 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 12:27:16.962694 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.960936    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 12:27:16.962694 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.960940    5060 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 12:27:16.962694 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.960944    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 12:27:16.962694 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.960948    5060 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 12:27:16.962694 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.960952    5060 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 12:27:16.962694 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.960956    5060 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 12:27:16.962694 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.960961    5060 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 12:27:16.963179 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.960965    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 12:27:16.963179 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.960969    5060 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 12:27:16.963179 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.960973    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 12:27:16.963179 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.960977    5060 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 12:27:16.963179 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.960981    5060 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 12:27:16.963179 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.960985    5060 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 12:27:16.963179 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.960990    5060 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 12:27:16.963179 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.960994    5060 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 12:27:16.963179 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.960998    5060 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 12:27:16.963179 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.961003    5060 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 12:27:16.963179 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.961007    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 12:27:16.963179 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.961011    5060 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 12:27:16.963179 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.961015    5060 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 12:27:16.963179 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.961023    5060 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 12:27:16.963179 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.961028    5060 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 12:27:16.963179 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.961032    5060 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 12:27:16.963179 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.961036    5060 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 12:27:16.963179 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.961040    5060 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 12:27:16.963179 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.961044    5060 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 12:27:16.963179 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.961049    5060 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 12:27:16.963731 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.961053    5060 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 12:27:16.963731 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.961057    5060 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 12:27:16.963731 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.961065    5060 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 12:27:16.963731 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.961070    5060 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 12:27:16.963731 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.961074    5060 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 12:27:16.963731 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.961079    5060 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 12:27:16.963731 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.961083    5060 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 12:27:16.963731 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.961087    5060 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 12:27:16.963731 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.961091    5060 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 12:27:16.963731 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.961097    5060 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 12:27:16.963731 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.961103    5060 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 12:27:16.963731 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.961108    5060 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 12:27:16.963731 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.961112    5060 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 12:27:16.963731 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.961116    5060 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 12:27:16.963731 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.961120    5060 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 12:27:16.963731 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.961124    5060 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 12:27:16.963731 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.961128    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 12:27:16.963731 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.961133    5060 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 12:27:16.963731 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.961137    5060 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 12:27:16.964180 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.961141    5060 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 12:27:16.964180 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.961146    5060 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 12:27:16.964180 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.961150    5060 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 12:27:16.964180 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.961154    5060 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 12:27:16.964180 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.961158    5060 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 12:27:16.964180 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.961162    5060 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 12:27:16.964180 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.961167    5060 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 12:27:16.964180 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.961172    5060 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 12:27:16.964180 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.961176    5060 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 12:27:16.964180 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.961180    5060 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 12:27:16.964180 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.961184    5060 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 12:27:16.964180 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.961189    5060 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 12:27:16.964180 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.961193    5060 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 12:27:16.964180 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.961199    5060 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 12:27:16.964180 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.961203    5060 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 12:27:16.964180 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.962004    5060 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 12:27:16.964180 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.962012    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 12:27:16.964180 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.962016    5060 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 12:27:16.964180 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.962021    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 12:27:16.964663 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.962025    5060 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 12:27:16.964663 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.962030    5060 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 12:27:16.964663 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.962034    5060 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 12:27:16.964663 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.962039    5060 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 12:27:16.964663 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.962044    5060 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 12:27:16.964663 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.962049    5060 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 12:27:16.964663 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.962054    5060 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 12:27:16.964663 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.962058    5060 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 12:27:16.964663 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.962062    5060 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 12:27:16.964663 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.962066    5060 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 12:27:16.964663 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.962070    5060 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 12:27:16.964663 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.962075    5060 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 12:27:16.964663 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.962079    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 12:27:16.964663 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.962083    5060 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 12:27:16.964663 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.962087    5060 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 12:27:16.964663 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.962091    5060 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 12:27:16.964663 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.962095    5060 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 12:27:16.964663 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.962100    5060 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 12:27:16.964663 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.962104    5060 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 12:27:16.964663 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.962108    5060 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 12:27:16.965158 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.962119    5060 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 12:27:16.965158 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.962123    5060 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 12:27:16.965158 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.962127    5060 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 12:27:16.965158 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.962131    5060 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 12:27:16.965158 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.962136    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 12:27:16.965158 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.962140    5060 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 12:27:16.965158 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.962144    5060 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 12:27:16.965158 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.962148    5060 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 12:27:16.965158 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.962152    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 12:27:16.965158 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.962157    5060 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 12:27:16.965158 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.962161    5060 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 12:27:16.965158 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.962165    5060 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 12:27:16.965158 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.962170    5060 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 12:27:16.965158 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.962174    5060 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 12:27:16.965158 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.962178    5060 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 12:27:16.965158 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.962182    5060 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 12:27:16.965158 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.962187    5060 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 12:27:16.965158 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.962191    5060 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 12:27:16.965158 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.962195    5060 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 12:27:16.965158 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.962199    5060 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 12:27:16.965665 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.962203    5060 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 12:27:16.965665 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.962207    5060 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 12:27:16.965665 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.962212    5060 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 12:27:16.965665 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.962216    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 12:27:16.965665 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.962220    5060 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 12:27:16.965665 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.962224    5060 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 12:27:16.965665 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.962228    5060 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 12:27:16.965665 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.962233    5060 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 12:27:16.965665 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.962239    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 12:27:16.965665 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.962244    5060 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 12:27:16.965665 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.962248    5060 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 12:27:16.965665 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.962252    5060 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 12:27:16.965665 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.962256    5060 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 12:27:16.965665 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.962267    5060 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 12:27:16.965665 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.962272    5060 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 12:27:16.965665 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.962276    5060 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 12:27:16.965665 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.962281    5060 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 12:27:16.965665 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.962285    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 12:27:16.965665 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.962289    5060 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 12:27:16.966181 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.962293    5060 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 12:27:16.966181 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.962298    5060 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 12:27:16.966181 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.962302    5060 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 12:27:16.966181 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.962306    5060 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 12:27:16.966181 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.962310    5060 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 12:27:16.966181 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.962314    5060 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 12:27:16.966181 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.962319    5060 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 12:27:16.966181 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.962323    5060 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 12:27:16.966181 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.962327    5060 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 12:27:16.966181 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.962331    5060 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 12:27:16.966181 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.962335    5060 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 12:27:16.966181 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.962339    5060 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 12:27:16.966181 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.962344    5060 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 12:27:16.966181 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.962348    5060 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 12:27:16.966181 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.962352    5060 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 12:27:16.966181 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.962356    5060 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 12:27:16.966181 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.962360    5060 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 12:27:16.966181 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.962364    5060 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 12:27:16.966181 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.962369    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 12:27:16.966705 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.962373    5060 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 12:27:16.966705 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.962377    5060 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 12:27:16.966705 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.962382    5060 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 12:27:16.966705 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.962386    5060 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 12:27:16.966705 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.962390    5060 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 12:27:16.966705 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.962394    5060 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 12:27:16.966705 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.962398    5060 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 12:27:16.966705 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.962402    5060 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 12:27:16.966705 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.962408    5060 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 12:27:16.966705 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.962413    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 12:27:16.966705 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.962417    5060 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 12:27:16.966705 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.962421    5060 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 12:27:16.966705 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.962425    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 12:27:16.966705 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.962432    5060 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 12:27:16.966705 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.962437    5060 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 12:27:16.966705 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.962441    5060 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 12:27:16.966705 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.962445    5060 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 12:27:16.966705 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.962449    5060 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 12:27:16.966705 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.962453    5060 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 12:27:16.966705 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.962457    5060 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 12:27:16.967209 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.962465    5060 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 12:27:16.967209 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.962471    5060 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 12:27:16.967209 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.962476    5060 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 12:27:16.967209 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.962481    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 12:27:16.967209 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.962627    5060 flags.go:64] FLAG: --address="0.0.0.0"
Oct 31 12:27:16.967209 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.962642    5060 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 31 12:27:16.967209 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.962652    5060 flags.go:64] FLAG: --anonymous-auth="true"
Oct 31 12:27:16.967209 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.962659    5060 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 31 12:27:16.967209 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.962666    5060 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 31 12:27:16.967209 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.962671    5060 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 31 12:27:16.967209 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.962678    5060 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 31 12:27:16.967209 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.962685    5060 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 31 12:27:16.967209 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.962689    5060 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 31 12:27:16.967209 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.962694    5060 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 31 12:27:16.967209 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.962700    5060 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 31 12:27:16.967209 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.962706    5060 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 31 12:27:16.967209 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.962711    5060 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 31 12:27:16.967209 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.962716    5060 flags.go:64] FLAG: --cgroup-root=""
Oct 31 12:27:16.967209 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.962720    5060 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 31 12:27:16.967209 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.962725    5060 flags.go:64] FLAG: --client-ca-file=""
Oct 31 12:27:16.967209 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.962730    5060 flags.go:64] FLAG: --cloud-provider="external"
Oct 31 12:27:16.967209 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.962736    5060 flags.go:64] FLAG: --cluster-dns="[]"
Oct 31 12:27:16.967209 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.962741    5060 flags.go:64] FLAG: --cluster-domain=""
Oct 31 12:27:16.967787 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.962746    5060 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 31 12:27:16.967787 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.962752    5060 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 31 12:27:16.967787 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.962760    5060 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 31 12:27:16.967787 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.962765    5060 flags.go:64] FLAG: --container-log-max-files="5"
Oct 31 12:27:16.967787 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.962771    5060 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 31 12:27:16.967787 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.962776    5060 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 31 12:27:16.967787 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.962781    5060 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 31 12:27:16.967787 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.962786    5060 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 31 12:27:16.967787 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.962790    5060 flags.go:64] FLAG: --contention-profiling="false"
Oct 31 12:27:16.967787 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.962795    5060 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 31 12:27:16.967787 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.962800    5060 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 31 12:27:16.967787 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.962805    5060 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 31 12:27:16.967787 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.962810    5060 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 31 12:27:16.967787 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.962816    5060 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 31 12:27:16.967787 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.962821    5060 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 31 12:27:16.967787 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.962826    5060 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 31 12:27:16.967787 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.962831    5060 flags.go:64] FLAG: --enable-load-reader="false"
Oct 31 12:27:16.967787 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.962836    5060 flags.go:64] FLAG: --enable-server="true"
Oct 31 12:27:16.967787 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.962843    5060 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 31 12:27:16.967787 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.962850    5060 flags.go:64] FLAG: --event-burst="100"
Oct 31 12:27:16.967787 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.962855    5060 flags.go:64] FLAG: --event-qps="50"
Oct 31 12:27:16.967787 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.962859    5060 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 31 12:27:16.967787 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.962864    5060 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 31 12:27:16.967787 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.962869    5060 flags.go:64] FLAG: --eviction-hard=""
Oct 31 12:27:16.968395 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.962875    5060 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 31 12:27:16.968395 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.962880    5060 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 31 12:27:16.968395 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.962885    5060 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 31 12:27:16.968395 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.962890    5060 flags.go:64] FLAG: --eviction-soft=""
Oct 31 12:27:16.968395 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.962894    5060 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 31 12:27:16.968395 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.962899    5060 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 31 12:27:16.968395 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.962904    5060 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 31 12:27:16.968395 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.962909    5060 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 31 12:27:16.968395 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.962915    5060 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 31 12:27:16.968395 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.962920    5060 flags.go:64] FLAG: --fail-swap-on="true"
Oct 31 12:27:16.968395 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.962926    5060 flags.go:64] FLAG: --feature-gates=""
Oct 31 12:27:16.968395 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.962932    5060 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 31 12:27:16.968395 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.962937    5060 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 31 12:27:16.968395 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.962942    5060 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 31 12:27:16.968395 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.962947    5060 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 31 12:27:16.968395 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.962952    5060 flags.go:64] FLAG: --healthz-port="10248"
Oct 31 12:27:16.968395 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.962957    5060 flags.go:64] FLAG: --help="false"
Oct 31 12:27:16.968395 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.962962    5060 flags.go:64] FLAG: --hostname-override="ip-10-0-130-190.us-east-2.compute.internal"
Oct 31 12:27:16.968395 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.962967    5060 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 31 12:27:16.968395 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.962972    5060 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 31 12:27:16.968395 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.962977    5060 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 31 12:27:16.968395 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.962982    5060 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 31 12:27:16.968395 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.962988    5060 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 31 12:27:16.968978 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.962992    5060 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 31 12:27:16.968978 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.962997    5060 flags.go:64] FLAG: --image-service-endpoint=""
Oct 31 12:27:16.968978 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.963002    5060 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 31 12:27:16.968978 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.963008    5060 flags.go:64] FLAG: --kube-api-burst="100"
Oct 31 12:27:16.968978 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.963013    5060 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 31 12:27:16.968978 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.963018    5060 flags.go:64] FLAG: --kube-api-qps="50"
Oct 31 12:27:16.968978 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.963023    5060 flags.go:64] FLAG: --kube-reserved=""
Oct 31 12:27:16.968978 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.963027    5060 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 31 12:27:16.968978 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.963032    5060 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 31 12:27:16.968978 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.963037    5060 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 31 12:27:16.968978 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.963041    5060 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 31 12:27:16.968978 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.963046    5060 flags.go:64] FLAG: --lock-file=""
Oct 31 12:27:16.968978 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.963051    5060 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 31 12:27:16.968978 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.963056    5060 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 31 12:27:16.968978 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.963060    5060 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 31 12:27:16.968978 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.963069    5060 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 31 12:27:16.968978 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.963074    5060 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 31 12:27:16.968978 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.963079    5060 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 31 12:27:16.968978 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.963083    5060 flags.go:64] FLAG: --logging-format="text"
Oct 31 12:27:16.968978 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.963090    5060 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 31 12:27:16.968978 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.963096    5060 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 31 12:27:16.968978 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.963101    5060 flags.go:64] FLAG: --manifest-url=""
Oct 31 12:27:16.968978 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.963105    5060 flags.go:64] FLAG: --manifest-url-header=""
Oct 31 12:27:16.968978 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.963112    5060 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 31 12:27:16.968978 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.963116    5060 flags.go:64] FLAG: --max-open-files="1000000"
Oct 31 12:27:16.969843 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.963122    5060 flags.go:64] FLAG: --max-pods="110"
Oct 31 12:27:16.969843 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.963127    5060 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 31 12:27:16.969843 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.963132    5060 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 31 12:27:16.969843 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.963137    5060 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 31 12:27:16.969843 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.963142    5060 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 31 12:27:16.969843 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.963146    5060 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 31 12:27:16.969843 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.963152    5060 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 31 12:27:16.969843 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.963157    5060 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 31 12:27:16.969843 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.963167    5060 flags.go:64] FLAG: --node-status-max-images="50"
Oct 31 12:27:16.969843 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.963172    5060 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 31 12:27:16.969843 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.963179    5060 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 31 12:27:16.969843 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.963184    5060 flags.go:64] FLAG: --pod-cidr=""
Oct 31 12:27:16.969843 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.963190    5060 flags.go:64] FLAG: --pod-manifest-path=""
Oct 31 12:27:16.969843 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.963194    5060 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 31 12:27:16.969843 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.963200    5060 flags.go:64] FLAG: --pods-per-core="0"
Oct 31 12:27:16.969843 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.963204    5060 flags.go:64] FLAG: --port="10250"
Oct 31 12:27:16.969843 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.963209    5060 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 31 12:27:16.969843 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.963214    5060 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-0c4d4bff4324dd77c"
Oct 31 12:27:16.969843 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.963219    5060 flags.go:64] FLAG: --qos-reserved=""
Oct 31 12:27:16.969843 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.963224    5060 flags.go:64] FLAG: --read-only-port="10255"
Oct 31 12:27:16.969843 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.963229    5060 flags.go:64] FLAG: --register-node="true"
Oct 31 12:27:16.969843 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.963234    5060 flags.go:64] FLAG: --register-with-taints=""
Oct 31 12:27:16.969843 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.963239    5060 flags.go:64] FLAG: --registry-burst="10"
Oct 31 12:27:16.969843 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.963244    5060 flags.go:64] FLAG: --registry-qps="5"
Oct 31 12:27:16.969843 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.963249    5060 flags.go:64] FLAG: --reserved-cpus=""
Oct 31 12:27:16.969843 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.963255    5060 flags.go:64] FLAG: --reserved-memory=""
Oct 31 12:27:16.970536 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.963263    5060 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 31 12:27:16.970536 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.963268    5060 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 31 12:27:16.970536 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.963273    5060 flags.go:64] FLAG: --rotate-certificates="false"
Oct 31 12:27:16.970536 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.963277    5060 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 31 12:27:16.970536 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.963281    5060 flags.go:64] FLAG: --runonce="false"
Oct 31 12:27:16.970536 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.963286    5060 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 31 12:27:16.970536 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.963291    5060 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 31 12:27:16.970536 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.963295    5060 flags.go:64] FLAG: --seccomp-default="false"
Oct 31 12:27:16.970536 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.963300    5060 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 31 12:27:16.970536 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.963304    5060 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 31 12:27:16.970536 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.963309    5060 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 31 12:27:16.970536 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.963314    5060 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 31 12:27:16.970536 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.963319    5060 flags.go:64] FLAG: --storage-driver-password="root"
Oct 31 12:27:16.970536 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.963323    5060 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 31 12:27:16.970536 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.963328    5060 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 31 12:27:16.970536 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.963332    5060 flags.go:64] FLAG: --storage-driver-user="root"
Oct 31 12:27:16.970536 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.963340    5060 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 31 12:27:16.970536 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.963344    5060 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 31 12:27:16.970536 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.963349    5060 flags.go:64] FLAG: --system-cgroups=""
Oct 31 12:27:16.970536 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.963354    5060 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 31 12:27:16.970536 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.963362    5060 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 31 12:27:16.970536 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.963366    5060 flags.go:64] FLAG: --tls-cert-file=""
Oct 31 12:27:16.970536 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.963371    5060 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 31 12:27:16.970536 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.963377    5060 flags.go:64] FLAG: --tls-min-version=""
Oct 31 12:27:16.970536 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.963382    5060 flags.go:64] FLAG: --tls-private-key-file=""
Oct 31 12:27:16.971151 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.963386    5060 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 31 12:27:16.971151 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.963391    5060 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 31 12:27:16.971151 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.963395    5060 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 31 12:27:16.971151 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.963400    5060 flags.go:64] FLAG: --v="2"
Oct 31 12:27:16.971151 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.963407    5060 flags.go:64] FLAG: --version="false"
Oct 31 12:27:16.971151 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.963413    5060 flags.go:64] FLAG: --vmodule=""
Oct 31 12:27:16.971151 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.963420    5060 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 31 12:27:16.971151 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.963428    5060 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 31 12:27:16.971151 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.963665    5060 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 12:27:16.971151 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.963672    5060 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 12:27:16.971151 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.963676    5060 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 12:27:16.971151 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.963681    5060 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 12:27:16.971151 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.963685    5060 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 12:27:16.971151 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.963689    5060 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 12:27:16.971151 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.963693    5060 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 12:27:16.971151 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.963698    5060 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 12:27:16.971151 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.963702    5060 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 12:27:16.971151 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.963709    5060 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 12:27:16.971151 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.963715    5060 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 12:27:16.971151 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.963720    5060 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 12:27:16.971151 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.963725    5060 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 12:27:16.971686 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.963729    5060 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 12:27:16.971686 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.963734    5060 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 12:27:16.971686 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.963740    5060 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 12:27:16.971686 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.963745    5060 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 12:27:16.971686 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.963750    5060 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 12:27:16.971686 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.963754    5060 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 12:27:16.971686 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.963758    5060 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 12:27:16.971686 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.963763    5060 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 12:27:16.971686 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.963767    5060 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 12:27:16.971686 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.963771    5060 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 12:27:16.971686 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.963775    5060 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 12:27:16.971686 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.963780    5060 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 12:27:16.971686 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.963784    5060 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 12:27:16.971686 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.963788    5060 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 12:27:16.971686 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.963792    5060 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 12:27:16.971686 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.963796    5060 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 12:27:16.971686 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.963800    5060 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 12:27:16.971686 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.963805    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 12:27:16.971686 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.963810    5060 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 12:27:16.971686 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.963815    5060 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 12:27:16.972166 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.963819    5060 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 12:27:16.972166 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.963823    5060 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 12:27:16.972166 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.963827    5060 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 12:27:16.972166 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.963832    5060 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 12:27:16.972166 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.963836    5060 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 12:27:16.972166 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.963840    5060 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 12:27:16.972166 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.963844    5060 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 12:27:16.972166 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.963848    5060 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 12:27:16.972166 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.963852    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 12:27:16.972166 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.963856    5060 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 12:27:16.972166 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.963861    5060 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 12:27:16.972166 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.963865    5060 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 12:27:16.972166 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.963869    5060 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 12:27:16.972166 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.963873    5060 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 12:27:16.972166 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.963880    5060 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 12:27:16.972166 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.963884    5060 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 12:27:16.972166 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.963888    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 12:27:16.972166 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.963892    5060 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 12:27:16.972166 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.963896    5060 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 12:27:16.972166 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.963902    5060 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 12:27:16.972777 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.963906    5060 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 12:27:16.972777 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.963910    5060 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 12:27:16.972777 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.963915    5060 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 12:27:16.972777 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.963919    5060 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 12:27:16.972777 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.963923    5060 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 12:27:16.972777 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.963927    5060 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 12:27:16.972777 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.963931    5060 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 12:27:16.972777 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.963935    5060 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 12:27:16.972777 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.963940    5060 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 12:27:16.972777 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.963944    5060 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 12:27:16.972777 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.963948    5060 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 12:27:16.972777 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.963952    5060 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 12:27:16.972777 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.963957    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 12:27:16.972777 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.963963    5060 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 12:27:16.972777 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.963970    5060 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 12:27:16.972777 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.963974    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 12:27:16.972777 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.963979    5060 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 12:27:16.972777 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.963983    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 12:27:16.972777 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.963987    5060 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 12:27:16.973256 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.963991    5060 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 12:27:16.973256 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.963995    5060 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 12:27:16.973256 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.963999    5060 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 12:27:16.973256 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.964003    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 12:27:16.973256 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.964008    5060 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 12:27:16.973256 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.964012    5060 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 12:27:16.973256 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.964016    5060 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 12:27:16.973256 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.964022    5060 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 12:27:16.973256 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.964026    5060 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 12:27:16.973256 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.964030    5060 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 12:27:16.973256 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.964034    5060 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 12:27:16.973256 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.964038    5060 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 12:27:16.973256 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.964043    5060 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 12:27:16.973256 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.964047    5060 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 12:27:16.973256 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.964051    5060 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 12:27:16.973256 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.964055    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 12:27:16.973256 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.964059    5060 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 12:27:16.973256 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.964063    5060 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 12:27:16.973256 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.964067    5060 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 12:27:16.973256 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.964072    5060 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 12:27:16.973791 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.964076    5060 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 12:27:16.973791 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.964080    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 12:27:16.973791 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.964084    5060 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 12:27:16.973791 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.964089    5060 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 12:27:16.973791 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.964093    5060 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 12:27:16.973791 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.964097    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 12:27:16.973791 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.964102    5060 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 12:27:16.973791 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.964107    5060 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 12:27:16.973791 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.964111    5060 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 12:27:16.973791 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.964115    5060 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 12:27:16.973791 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.964119    5060 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 12:27:16.973791 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.964124    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 12:27:16.973791 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.964128    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 12:27:16.973791 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.964132    5060 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 12:27:16.973791 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.964168    5060 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 12:27:16.974213 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.964221    5060 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 12:27:16.974213 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.964229    5060 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 12:27:16.974213 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.964233    5060 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 12:27:16.974213 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.964238    5060 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 12:27:16.974213 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.964243    5060 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 12:27:16.974213 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.964248    5060 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 12:27:16.974213 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.964252    5060 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 12:27:16.974213 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.964256    5060 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 12:27:16.974213 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.964261    5060 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 12:27:16.974213 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.964265    5060 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 12:27:16.974213 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.964269    5060 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 12:27:16.974213 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.964273    5060 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 12:27:16.974213 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.964277    5060 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 12:27:16.974213 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.964281    5060 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 12:27:16.974213 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.964285    5060 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 12:27:16.974213 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.964289    5060 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 12:27:16.974213 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.964293    5060 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 12:27:16.974213 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.964297    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 12:27:16.974213 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.964302    5060 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 12:27:16.974213 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.964306    5060 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 12:27:16.974930 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.964310    5060 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 12:27:16.974930 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.964314    5060 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 12:27:16.974930 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.964319    5060 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 12:27:16.974930 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.964323    5060 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 12:27:16.974930 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.964327    5060 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 12:27:16.974930 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.964332    5060 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 12:27:16.974930 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.964336    5060 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 12:27:16.974930 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.964340    5060 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 12:27:16.974930 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.964344    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 12:27:16.974930 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.964348    5060 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 12:27:16.974930 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.964353    5060 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 12:27:16.974930 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.964357    5060 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 12:27:16.974930 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.964361    5060 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 12:27:16.974930 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.964365    5060 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 12:27:16.974930 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.964370    5060 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 12:27:16.974930 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.964374    5060 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 12:27:16.974930 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.964379    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 12:27:16.974930 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.964383    5060 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 12:27:16.974930 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.964387    5060 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 12:27:16.974930 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.964393    5060 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 12:27:16.975623 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.964397    5060 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 12:27:16.975623 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.964401    5060 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 12:27:16.975623 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.964405    5060 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 12:27:16.975623 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.964409    5060 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 12:27:16.975623 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.964414    5060 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 12:27:16.975623 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.964418    5060 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 12:27:16.975623 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.964422    5060 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 12:27:16.975623 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.964426    5060 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 12:27:16.975623 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.964430    5060 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 12:27:16.975623 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.964435    5060 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 12:27:16.975623 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.964441    5060 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 12:27:16.975623 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.964445    5060 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 12:27:16.975623 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.964449    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 12:27:16.975623 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.964455    5060 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 12:27:16.975623 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.964461    5060 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 12:27:16.975623 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.964467    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 12:27:16.975623 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.964472    5060 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 12:27:16.975623 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.964478    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 12:27:16.975623 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.964483    5060 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 12:27:16.976469 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.964508    5060 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 12:27:16.976469 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.964513    5060 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 12:27:16.976469 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.964517    5060 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 12:27:16.976469 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.964521    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 12:27:16.976469 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.964526    5060 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 12:27:16.976469 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.964530    5060 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 12:27:16.976469 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.964534    5060 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 12:27:16.976469 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.964539    5060 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 12:27:16.976469 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.964543    5060 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 12:27:16.976469 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.964550    5060 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 12:27:16.976469 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.964554    5060 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 12:27:16.976469 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.964558    5060 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 12:27:16.976469 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.964562    5060 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 12:27:16.976469 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.964567    5060 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 12:27:16.976469 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.964571    5060 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 12:27:16.976469 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.964575    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 12:27:16.976469 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.964579    5060 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 12:27:16.976469 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.964583    5060 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 12:27:16.976469 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.964587    5060 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 12:27:16.976469 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.964591    5060 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 12:27:16.977206 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.964595    5060 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 12:27:16.977206 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.964600    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 12:27:16.977206 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.964604    5060 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 12:27:16.977206 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.964610    5060 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 12:27:16.977206 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.964614    5060 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 12:27:16.977206 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.964619    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 12:27:16.977206 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.964624    5060 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 12:27:16.977206 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.964628    5060 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 12:27:16.977206 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.964632    5060 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 12:27:16.977206 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.964636    5060 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 12:27:16.977206 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.964641    5060 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 12:27:16.977206 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.964646    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 12:27:16.977206 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.964650    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 12:27:16.977206 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.964655    5060 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 12:27:16.977206 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.964660    5060 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 12:27:16.977206 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.964664    5060 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 12:27:16.977206 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.964668    5060 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 12:27:16.977206 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.964672    5060 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 12:27:16.977206 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.964677    5060 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 12:27:16.977900 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.964681    5060 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 12:27:16.977900 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.964685    5060 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 12:27:16.977900 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.964690    5060 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 12:27:16.977900 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.964694    5060 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 12:27:16.977900 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.964702    5060 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 12:27:16.977900 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.964708    5060 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 12:27:16.977900 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.964713    5060 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 12:27:16.977900 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.964717    5060 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 12:27:16.977900 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.968998    5060 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 31 12:27:16.977900 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.969018    5060 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 31 12:27:16.977900 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.969045    5060 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 31 12:27:16.977900 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.969052    5060 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 31 12:27:16.977900 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.969224    5060 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 12:27:16.977900 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.969232    5060 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 12:27:16.977900 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.969238    5060 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 12:27:16.977900 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.969243    5060 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 12:27:16.977900 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.969248    5060 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 12:27:16.977900 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.969253    5060 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 12:27:16.977900 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.969258    5060 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 12:27:16.978386 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.969262    5060 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 12:27:16.978386 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.969267    5060 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 12:27:16.978386 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.969277    5060 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 12:27:16.978386 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.969281    5060 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 12:27:16.978386 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.969286    5060 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 12:27:16.978386 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.969290    5060 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 12:27:16.978386 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.969295    5060 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 12:27:16.978386 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.969299    5060 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 12:27:16.978386 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.969304    5060 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 12:27:16.978386 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.969308    5060 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 12:27:16.978386 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.969313    5060 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 12:27:16.978386 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.969317    5060 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 12:27:16.978386 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.969322    5060 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 12:27:16.978386 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.969326    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 12:27:16.978386 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.969331    5060 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 12:27:16.978386 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.969340    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 12:27:16.978386 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.969344    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 12:27:16.978386 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.969348    5060 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 12:27:16.978386 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.969353    5060 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 12:27:16.978386 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.969357    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 12:27:16.978867 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.969362    5060 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 12:27:16.978867 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.969366    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 12:27:16.978867 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.969372    5060 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 12:27:16.978867 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.969376    5060 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 12:27:16.978867 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.969382    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 12:27:16.978867 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.969386    5060 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 12:27:16.978867 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.969390    5060 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 12:27:16.978867 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.969395    5060 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 12:27:16.978867 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.969404    5060 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 12:27:16.978867 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.969409    5060 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 12:27:16.978867 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.969413    5060 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 12:27:16.978867 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.969418    5060 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 12:27:16.978867 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.969422    5060 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 12:27:16.978867 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.969427    5060 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 12:27:16.978867 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.969431    5060 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 12:27:16.978867 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.969436    5060 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 12:27:16.978867 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.969440    5060 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 12:27:16.978867 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.969444    5060 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 12:27:16.978867 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.969451    5060 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 12:27:16.979317 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.969458    5060 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 12:27:16.979317 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.969463    5060 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 12:27:16.979317 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.969475    5060 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 12:27:16.979317 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.969481    5060 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 12:27:16.979317 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.969509    5060 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 12:27:16.979317 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.969516    5060 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 12:27:16.979317 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.969521    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 12:27:16.979317 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.969526    5060 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 12:27:16.979317 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.969530    5060 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 12:27:16.979317 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.969535    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 12:27:16.979317 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.969539    5060 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 12:27:16.979317 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.969543    5060 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 12:27:16.979317 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.969548    5060 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 12:27:16.979317 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.969552    5060 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 12:27:16.979317 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.969562    5060 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 12:27:16.979317 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.969567    5060 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 12:27:16.979317 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.969573    5060 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 12:27:16.979317 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.969578    5060 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 12:27:16.979317 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.969582    5060 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 12:27:16.979795 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.969586    5060 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 12:27:16.979795 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.969590    5060 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 12:27:16.979795 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.969595    5060 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 12:27:16.979795 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.969600    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 12:27:16.979795 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.969604    5060 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 12:27:16.979795 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.969608    5060 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 12:27:16.979795 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.969613    5060 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 12:27:16.979795 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.969617    5060 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 12:27:16.979795 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.969627    5060 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 12:27:16.979795 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.969631    5060 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 12:27:16.979795 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.969636    5060 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 12:27:16.979795 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.969641    5060 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 12:27:16.979795 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.969645    5060 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 12:27:16.979795 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.969650    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 12:27:16.979795 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.969654    5060 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 12:27:16.979795 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.969659    5060 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 12:27:16.979795 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.969663    5060 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 12:27:16.979795 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.969668    5060 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 12:27:16.979795 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.969672    5060 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 12:27:16.979795 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.969677    5060 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 12:27:16.980309 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.969686    5060 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 12:27:16.980309 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.969690    5060 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 12:27:16.980309 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.969694    5060 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 12:27:16.980309 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.969699    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 12:27:16.980309 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.969704    5060 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 12:27:16.980309 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.969708    5060 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 12:27:16.980309 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.969713    5060 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 12:27:16.980309 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.969717    5060 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 12:27:16.980309 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.969722    5060 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 12:27:16.980309 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.969726    5060 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 12:27:16.980309 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.969774    5060 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 12:27:16.980309 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.969795    5060 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 12:27:16.980309 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.969800    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 12:27:16.980309 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.969806    5060 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 12:27:16.980309 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.969811    5060 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 12:27:16.980309 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.969817    5060 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 12:27:16.980309 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.969994    5060 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 12:27:16.980309 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970001    5060 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 12:27:16.980309 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970004    5060 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 12:27:16.980309 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970008    5060 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 12:27:16.980807 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970011    5060 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 12:27:16.980807 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.970048    5060 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 12:27:16.980807 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970207    5060 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 12:27:16.980807 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970214    5060 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 12:27:16.980807 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970217    5060 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 12:27:16.980807 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970220    5060 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 12:27:16.980807 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970224    5060 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 12:27:16.980807 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970226    5060 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 12:27:16.980807 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970230    5060 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 12:27:16.980807 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970233    5060 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 12:27:16.980807 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970236    5060 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 12:27:16.980807 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970239    5060 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 12:27:16.980807 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970242    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 12:27:16.980807 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970245    5060 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 12:27:16.980807 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970248    5060 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 12:27:16.980807 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970250    5060 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 12:27:16.981241 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970253    5060 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 12:27:16.981241 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970256    5060 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 12:27:16.981241 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970258    5060 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 12:27:16.981241 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970261    5060 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 12:27:16.981241 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970264    5060 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 12:27:16.981241 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970266    5060 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 12:27:16.981241 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970269    5060 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 12:27:16.981241 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970272    5060 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 12:27:16.981241 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970275    5060 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 12:27:16.981241 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970277    5060 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 12:27:16.981241 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970281    5060 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 12:27:16.981241 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970283    5060 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 12:27:16.981241 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970286    5060 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 12:27:16.981241 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970289    5060 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 12:27:16.981241 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970292    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 12:27:16.981241 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970294    5060 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 12:27:16.981241 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970297    5060 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 12:27:16.981241 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970299    5060 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 12:27:16.981241 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970302    5060 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 12:27:16.981241 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970305    5060 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 12:27:16.981795 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970307    5060 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 12:27:16.981795 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970310    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 12:27:16.981795 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970312    5060 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 12:27:16.981795 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970315    5060 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 12:27:16.981795 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970318    5060 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 12:27:16.981795 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970321    5060 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 12:27:16.981795 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970324    5060 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 12:27:16.981795 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970326    5060 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 12:27:16.981795 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970329    5060 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 12:27:16.981795 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970331    5060 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 12:27:16.981795 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970334    5060 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 12:27:16.981795 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970336    5060 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 12:27:16.981795 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970339    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 12:27:16.981795 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970341    5060 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 12:27:16.981795 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970344    5060 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 12:27:16.981795 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970348    5060 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 12:27:16.981795 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970352    5060 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 12:27:16.981795 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970355    5060 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 12:27:16.981795 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970357    5060 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 12:27:16.981795 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970361    5060 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 12:27:16.982275 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970364    5060 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 12:27:16.982275 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970367    5060 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 12:27:16.982275 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970371    5060 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 12:27:16.982275 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970375    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 12:27:16.982275 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970379    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 12:27:16.982275 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970382    5060 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 12:27:16.982275 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970385    5060 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 12:27:16.982275 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970388    5060 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 12:27:16.982275 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970391    5060 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 12:27:16.982275 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970394    5060 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 12:27:16.982275 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970397    5060 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 12:27:16.982275 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970399    5060 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 12:27:16.982275 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970402    5060 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 12:27:16.982275 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970404    5060 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 12:27:16.982275 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970407    5060 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 12:27:16.982275 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970409    5060 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 12:27:16.982275 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970412    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 12:27:16.982275 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970415    5060 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 12:27:16.982275 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970417    5060 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 12:27:16.982784 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970420    5060 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 12:27:16.982784 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970423    5060 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 12:27:16.982784 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970425    5060 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 12:27:16.982784 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970428    5060 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 12:27:16.982784 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970431    5060 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 12:27:16.982784 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970433    5060 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 12:27:16.982784 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970436    5060 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 12:27:16.982784 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970439    5060 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 12:27:16.982784 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970442    5060 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 12:27:16.982784 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970444    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 12:27:16.982784 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970447    5060 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 12:27:16.982784 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970450    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 12:27:16.982784 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970453    5060 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 12:27:16.982784 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970455    5060 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 12:27:16.982784 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970458    5060 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 12:27:16.982784 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970460    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 12:27:16.982784 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970463    5060 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 12:27:16.982784 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970465    5060 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 12:27:16.982784 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970468    5060 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 12:27:16.983290 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970470    5060 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 12:27:16.983290 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970473    5060 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 12:27:16.983290 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970475    5060 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 12:27:16.983290 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970478    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 12:27:16.983290 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970480    5060 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 12:27:16.983290 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970483    5060 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 12:27:16.983290 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970485    5060 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 12:27:16.983290 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970508    5060 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 12:27:16.983290 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970511    5060 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 12:27:16.983290 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970513    5060 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 12:27:16.983290 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970516    5060 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 12:27:16.983290 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970519    5060 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 12:27:16.983290 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970522    5060 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 12:27:16.983290 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970525    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 12:27:16.983290 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.970549    5060 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 12:27:16.983290 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970581    5060 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 12:27:16.983814 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970585    5060 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 12:27:16.983814 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970588    5060 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 12:27:16.983814 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970591    5060 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 12:27:16.983814 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970594    5060 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 12:27:16.983814 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970597    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 12:27:16.983814 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970599    5060 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 12:27:16.983814 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970602    5060 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 12:27:16.983814 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970606    5060 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 12:27:16.983814 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970610    5060 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 12:27:16.983814 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970612    5060 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 12:27:16.983814 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970615    5060 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 12:27:16.983814 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970618    5060 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 12:27:16.983814 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970621    5060 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 12:27:16.983814 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970624    5060 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 12:27:16.983814 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970628    5060 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 12:27:16.983814 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970632    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 12:27:16.983814 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970635    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 12:27:16.983814 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970637    5060 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 12:27:16.983814 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970641    5060 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 12:27:16.984441 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970644    5060 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 12:27:16.984441 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970647    5060 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 12:27:16.984441 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970649    5060 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 12:27:16.984441 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970652    5060 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 12:27:16.984441 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970655    5060 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 12:27:16.984441 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970657    5060 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 12:27:16.984441 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970660    5060 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 12:27:16.984441 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970662    5060 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 12:27:16.984441 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970665    5060 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 12:27:16.984441 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970667    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 12:27:16.984441 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970670    5060 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 12:27:16.984441 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970673    5060 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 12:27:16.984441 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970675    5060 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 12:27:16.984441 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970678    5060 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 12:27:16.984441 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970681    5060 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 12:27:16.984441 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970684    5060 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 12:27:16.984441 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970687    5060 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 12:27:16.984441 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970692    5060 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 12:27:16.984441 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970695    5060 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 12:27:16.985318 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970697    5060 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 12:27:16.985318 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970700    5060 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 12:27:16.985318 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970703    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 12:27:16.985318 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970705    5060 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 12:27:16.985318 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970708    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 12:27:16.985318 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970711    5060 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 12:27:16.985318 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970714    5060 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 12:27:16.985318 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970716    5060 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 12:27:16.985318 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970719    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 12:27:16.985318 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970721    5060 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 12:27:16.985318 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970724    5060 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 12:27:16.985318 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970727    5060 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 12:27:16.985318 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970729    5060 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 12:27:16.985318 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970732    5060 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 12:27:16.985318 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970734    5060 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 12:27:16.985318 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970737    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 12:27:16.985318 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970740    5060 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 12:27:16.985318 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970743    5060 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 12:27:16.985318 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970745    5060 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 12:27:16.986005 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970748    5060 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 12:27:16.986005 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970751    5060 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 12:27:16.986005 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970753    5060 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 12:27:16.986005 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970756    5060 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 12:27:16.986005 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970758    5060 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 12:27:16.986005 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970761    5060 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 12:27:16.986005 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970764    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 12:27:16.986005 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970767    5060 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 12:27:16.986005 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970770    5060 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 12:27:16.986005 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970773    5060 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 12:27:16.986005 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970775    5060 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 12:27:16.986005 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970779    5060 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 12:27:16.986005 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970782    5060 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 12:27:16.986005 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970785    5060 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 12:27:16.986005 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970788    5060 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 12:27:16.986005 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970790    5060 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 12:27:16.986005 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970793    5060 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 12:27:16.986005 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970796    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 12:27:16.986005 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970799    5060 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 12:27:16.986005 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970802    5060 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 12:27:16.986891 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970804    5060 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 12:27:16.986891 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970807    5060 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 12:27:16.986891 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970810    5060 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 12:27:16.986891 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970812    5060 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 12:27:16.986891 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970815    5060 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 12:27:16.986891 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970817    5060 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 12:27:16.986891 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970820    5060 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 12:27:16.986891 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970822    5060 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 12:27:16.986891 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970825    5060 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 12:27:16.986891 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970828    5060 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 12:27:16.986891 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970830    5060 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 12:27:16.986891 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970833    5060 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 12:27:16.986891 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970835    5060 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 12:27:16.986891 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970838    5060 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 12:27:16.986891 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970840    5060 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 12:27:16.986891 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970843    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 12:27:16.986891 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970845    5060 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 12:27:16.986891 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970848    5060 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 12:27:16.986891 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970851    5060 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 12:27:16.986891 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970854    5060 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 12:27:16.987370 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970857    5060 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 12:27:16.987370 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970860    5060 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 12:27:16.987370 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970862    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 12:27:16.987370 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970866    5060 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 12:27:16.987370 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970869    5060 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 12:27:16.987370 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970872    5060 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 12:27:16.987370 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970875    5060 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 12:27:16.987370 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:16.970878    5060 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 12:27:16.987370 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.970944    5060 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 31 12:27:16.987370 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.971909    5060 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Oct 31 12:27:16.987370 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.971974    5060 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Oct 31 12:27:16.987370 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.972159    5060 server.go:1014] "Starting client certificate rotation"
Oct 31 12:27:16.987370 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.972319    5060 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 31 12:27:16.987370 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.972390    5060 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-30 12:21:35 +0000 UTC" deadline="2028-06-16 17:58:31.525315144 +0000 UTC"
Oct 31 12:27:16.987370 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.972430    5060 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14261h31m14.552888458s"
Oct 31 12:27:16.987370 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.972634    5060 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 31 12:27:16.987788 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.973656    5060 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 31 12:27:16.987788 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.974820    5060 log.go:25] "Validated CRI v1 runtime API"
Oct 31 12:27:16.987788 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.976944    5060 log.go:25] "Validated CRI v1 image API"
Oct 31 12:27:16.987788 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.977634    5060 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 31 12:27:16.987788 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.979638    5060 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 ea297aca-9120-4875-9123-81f10ff3a7a9:/dev/nvme0n1p3 eaa7c2b6-cdc6-44ca-9bdd-fdaa17fd7979:/dev/nvme0n1p4]
Oct 31 12:27:16.987939 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.979658    5060 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1712b1a89fc033bb7af133fed6fccaed65abbd63124e20f0e3ea940d30aa7113/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1712b1a89fc033bb7af133fed6fccaed65abbd63124e20f0e3ea940d30aa7113/userdata/shm major:0 minor:74 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1fe0c2cef9e318b723913d1aafd071a4c4a50766fab690a4b526ef633e216e36/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1fe0c2cef9e318b723913d1aafd071a4c4a50766fab690a4b526ef633e216e36/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/24f74c6f47a4176d951ed9f1e88a53b5b9928f802c402c0074106d93796b3218/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/24f74c6f47a4176d951ed9f1e88a53b5b9928f802c402c0074106d93796b3218/userdata/shm major:0 minor:243 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3cd295581fbb43bd6a480a27b300dbdfe231373fcec52e0d8ceeb87c35bd6500/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3cd295581fbb43bd6a480a27b300dbdfe231373fcec52e0d8ceeb87c35bd6500/userdata/shm major:0 minor:95 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/432c0f75ce4f14ee6e21e05374e78fc51fdb2f101584925475c6900c584e1409/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/432c0f75ce4f14ee6e21e05374e78fc51fdb2f101584925475c6900c584e1409/userdata/shm major:0 minor:232 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/432c216b017de6fb2dc9ee38001b85f4604b2715e988e0d643b46af056985f7d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/432c216b017de6fb2dc9ee38001b85f4604b2715e988e0d643b46af056985f7d/userdata/shm major:0 minor:246 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5a4af82d83c4e2d729012b35f36754f633c7d8edae532b658e217176aeed8416/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5a4af82d83c4e2d729012b35f36754f633c7d8edae532b658e217176aeed8416/userdata/shm major:0 minor:230 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/68516ca2184b4f7cae9b513be83b9d11699718f173a9dfde5a4ffb4a1031cddc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/68516ca2184b4f7cae9b513be83b9d11699718f173a9dfde5a4ffb4a1031cddc/userdata/shm major:0 minor:240 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6aaae0ad5cd357b195d055aaf85b96bed388d3a4f75a1ce9caad08ede506e3c7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6aaae0ad5cd357b195d055aaf85b96bed388d3a4f75a1ce9caad08ede506e3c7/userdata/shm major:0 minor:301 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8738d804fa86e15f0597349bd890e4bfa1c5c1415beaacbab01aad9c000fbd0a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8738d804fa86e15f0597349bd890e4bfa1c5c1415beaacbab01aad9c000fbd0a/userdata/shm major:0 minor:314 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8a59f3fca9f665e623be85a15c0d77fe56925fd62aae37db535bb9a23c596a6f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8a59f3fca9f665e623be85a15c0d77fe56925fd62aae37db535bb9a23c596a6f/userdata/shm major:0 minor:228 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8a79b2069dde730f773d0ac904f6e23f6db008b97e8fd24680a8e7fa053f2e9f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8a79b2069dde730f773d0ac904f6e23f6db008b97e8fd24680a8e7fa053f2e9f/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8dbe3066e41c634d3607763741f771139a4be4a86fb97f302e275ecb6c7399bd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8dbe3066e41c634d3607763741f771139a4be4a86fb97f302e275ecb6c7399bd/userdata/shm major:0 minor:296 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9ec8c5086bd55eeaa1cf3cd8ee3146133e8915c24281d20fe7de74ac684c8429/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9ec8c5086bd55eeaa1cf3cd8ee3146133e8915c24281d20fe7de74ac684c8429/userdata/shm major:0 minor:236 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a130b0e336cef0401531b357c6118d6c3ec61ce95a7b67c9eda4cf43d7a4f443/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a130b0e336cef0401531b357c6118d6c3ec61ce95a7b67c9eda4cf43d7a4f443/userdata/shm major:0 minor:238 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b530060702c06b4b43a5a8cdf318bc791c379fc3c905612530366b47f66d02c8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b530060702c06b4b43a5a8cdf318bc791c379fc3c905612530366b47f66d02c8/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b539c38438f2cfd785f2a2a6e023750667ea93e764840b11ad42f359c3687b08/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b539c38438f2cfd785f2a2a6e023750667ea93e764840b11ad42f359c3687b08/userdata/shm major:0 minor:51 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b61159ca029793099370ff0779dda563c7d5600e24e1415f4ca6047a637d4ed7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b61159ca029793099370ff0779dda563c7d5600e24e1415f4ca6047a637d4ed7/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c1849c803f134d969c39443fe4e01b9f3a600259e4ff5749f6d3ca25a2a168b1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c1849c803f134d969c39443fe4e01b9f3a600259e4ff5749f6d3ca25a2a168b1/userdata/shm major:0 minor:234 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c4cb00c97f37c42bcd3b91b32e6ec987fd6fd61186b49b744d6301563f405c70/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c4cb00c97f37c42bcd3b91b32e6ec987fd6fd61186b49b744d6301563f405c70/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c8dd1a20232d5b64644dd25931795726b9f8c36ada12fd4da9078653557feddd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c8dd1a20232d5b64644dd25931795726b9f8c36ada12fd4da9078653557feddd/userdata/shm major:0 minor:323 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c9ca40edbf0dd49e6bf694363da3d39a56d8bd8ff7baef573fe1d91ab10933b6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c9ca40edbf0dd49e6bf694363da3d39a56d8bd8ff7baef573fe1d91ab10933b6/userdata/shm major:0 minor:322 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d14f2152e247518ab96f882b2d1c789e2985ca7b210789682e82d1d99b01a2ea/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d14f2152e247518ab96f882b2d1c789e2985ca7b210789682e82d1d99b01a2ea/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/dd140492a16d3d96e416a11f11f04bef073fc5e086c5ae9eeeb1a019605577c1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/dd140492a16d3d96e416a11f11f04bef073fc5e086c5ae9eeeb1a019605577c1/userdata/shm major:0 minor:242 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e4538bb23cd43573e0f29c86571dcd40acc1137dc3413f283057119ec8a4a608/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e4538bb23cd43573e0f29c86571dcd40acc1137dc3413f283057119ec8a4a608/userdata/shm major:0 minor:88 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f5a0fca700cd53f8f083df04ab0ae9b664259a71e5b410fd4990db2d3ea0b60b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f5a0fca700cd53f8f083df04ab0ae9b664259a71e5b410fd4990db2d3ea0b60b/userdata/shm major:0 minor:362 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/031551df-64a5-411b-8eb8-7f0bf2d4f170/volumes/kubernetes.io~projected/kube-api-access-wpcl9:{mountpoint:/var/lib/kubelet/pods/031551df-64a5-411b-8eb8-7f0bf2d4f170/volumes/kubernetes.io~projected/kube-api-access-wpcl9 major:0 minor:319 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/07714f20-a70e-49e8-8e51-d03a8e00005a/volumes/kubernetes.io~projected/kube-api-access-hqmhd:{mountpoint:/var/lib/kubelet/pods/07714f20-a70e-49e8-8e51-d03a8e00005a/volumes/kubernetes.io~projected/kube-api-access-hqmhd major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0db116b4-d971-4966-bb05-db28d077b6fe/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/0db116b4-d971-4966-bb05-db28d077b6fe/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0db116b4-d971-4966-bb05-db28d077b6fe/volumes/kubernetes.io~projected/kube-api-access-nlgsp:{mountpoint:/var/lib/kubelet/pods/0db116b4-d971-4966-bb05-db28d077b6fe/volumes/kubernetes.io~projected/kube-api-access-nlgsp major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0db116b4-d971-4966-bb05-db28d077b6fe/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/0db116b4-d971-4966-bb05-db28d077b6fe/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0db116b4-d971-4966-bb05-db28d077b6fe/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/0db116b4-d971-4966-bb05-db28d077b6fe/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/25691fa3-7017-429a-b94c-c2dde4d29e55/volumes/kubernetes.io~projected/kube-api-access-gvqpw:{mountpoint:/var/lib/kubelet/pods/25691fa3-7017-429a-b94c-c2dde4d29e55/volumes/kubernetes.io~projected/kube-api-access-gvqpw major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3bb2c04e-55a2-4206-9530-7298f4469379/volumes/kubernetes.io~projected/kube-api-access-xv9qt:{mountpoint:/var/lib/kubelet/pods/3bb2c04e-55a2-4206-9530-7298f4469379/volumes/kubernetes.io~projected/kube-api-access-xv9qt major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee/volumes/kubernetes.io~empty-dir/tmp major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee/volumes/kubernetes.io~projected/kube-api-access-t5nx4:{mountpoint:/var/lib/kubelet/pods/51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee/volumes/kubernetes.io~projected/kube-api-access-t5nx4 major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/556dda7b-d8e6-4558-b5f3-994b01d6a866/volumes/kubernetes.io~projected/kube-api-access-9g26m:{mountpoint:/var/lib/kubelet/pods/556dda7b-d8e6-4558-b5f3-994b01d6a866/volumes/kubernetes.io~projected/kube-api-access-9g26m major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/556dda7b-d8e6-4558-b5f3-994b01d6a866/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/556dda7b-d8e6-4558-b5f3-994b01d6a866/volumes/kubernetes.io~secret/default-certificate major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/556dda7b-d8e6-4558-b5f3-994b01d6a866/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/556dda7b-d8e6-4558-b5f3-994b01d6a866/volumes/kubernetes.io~secret/stats-auth major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5c8865f9-fdf6-49b9-8b4f-69dbc7787084/volumes/kubernetes.io~projected/kube-api-access-mhm56:{mountpoint:/var/lib/kubelet/pods/5c8865f9-fdf6-49b9-8b4f-69dbc7787084/volumes/kubernetes.io~projected/kube-api-access-mhm56 major:0 minor:220 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/613125d4-fbb4-4b43-9c47-24e039bd4697/volumes/kubernetes.io~projected/kube-api-access-mvzh5:{mountpoint:/var/lib/kubelet/pods/613125d4-fbb4-4b43-9c47-24e039bd4697/volumes/kubernetes.io~projected/kube-api-access-mvzh5 major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/64649bfe-c3e6-469a-a9ff-0db1ca0fadb3/volumes/kubernetes.io~projected/kube-api-access-48lnl:{mountpoint:/var/lib/kubelet/pods/64649bfe-c3e6-469a-a9ff-0db1ca0fadb3/volumes/kubernetes.io~projected/kube-api-access-48lnl major:0 minor:318 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6b3e77ff-4445-42fa-a7c3-f35d442c276d/volumes/kubernetes.io~projected/kube-api-access-vknw2:{mountpoint:/var/lib/kubelet/pods/6b3e77ff-4445-42fa-a7c3-f35d442c276d/volumes/kubernetes.io~projected/kube-api-access-vknw2 major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6bbe13aa-df4b-40b0-8595-68fe1166a082/volumes/kubernetes.io~projected/kube-api-access-7j2rp:{mountpoint:/var/lib/kubelet/pods/6bbe13aa-df4b-40b0-8595-68fe1166a082/volumes/kubernetes.io~projected/kube-api-access-7j2rp major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6c661ce8-9f7b-4509-bd17-2380bcd1c425/volumes/kubernetes.io~projected/kube-api-access-nx5hk:{mountpoint:/var/lib/kubelet/pods/6c661ce8-9f7b-4509-bd17-2380bcd1c425/volumes/kubernetes.io~projected/kube-api-access-nx5hk major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7a402472-ca33-44ae-b598-96fc8e7a5eb2/volumes/kubernetes.io~projected/kube-api-access-2tf6n:{mountpoint:/var/lib/kubelet/pods/7a402472-ca33-44ae-b598-96fc8e7a5eb2/volumes/kubernetes.io~projected/kube-api-access-2tf6n major:0 minor:293 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/891b3c47-ff7c-4a7a-a09b-411361a843f7/volumes/kubernetes.io~projected/kube-api-access-mrpck:{mountpoint:/var/lib/kubelet/pods/891b3c47-ff7c-4a7a-a09b-411361a843f7/volumes/kubernetes.io~projected/kube-api-access-mrpck major:0 minor:226 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/891b3c47-ff7c-4a7a-a09b-411361a843f7/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/891b3c47-ff7c-4a7a-a09b-411361a843f7/volumes/kubernetes.io~secret/cert major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9083c35a-9739-4f3e-8415-bad1e6433d40/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/9083c35a-9739-4f3e-8415-bad1e6433d40/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9083c35a-9739-4f3e-8415-bad1e6433d40/volumes/kubernetes.io~projected/kube-api-access-5xwrq:{mountpoint:/var/lib/kubelet/pods/9083c35a-9739-4f3e-8415-bad1e6433d40/volumes/kubernetes.io~projected/kube-api-access-5xwrq major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9083c35a-9739-4f3e-8415-bad1e6433d40/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/9083c35a-9739-4f3e-8415-bad1e6433d40/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9083c35a-9739-4f3e-8415-bad1e6433d40/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/9083c35a-9739-4f3e-8415-bad1e6433d40/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9557a3f0-ae0f-41f9-a9b5-15aeda410a38/volumes/kubernetes.io~projected/kube-api-access-jl4xd:{mountpoint:/var/lib/kubelet/pods/9557a3f0-ae0f-41f9-a9b5-15aeda410a38/volumes/kubernetes.io~projected/kube-api-access-jl4xd major:0 minor:308 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/99a606a8-546b-4145-8bae-a95061241069/volumes/kubernetes.io~projected/kube-api-access-6t4dc:{mountpoint:/var/lib/kubelet/pods/99a606a8-546b-4145-8bae-a95061241069/volumes/kubernetes.io~projected/kube-api-access-6t4dc major:0 minor:221 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/99a606a8-546b-4145-8bae-a95061241069/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/99a606a8-546b-4145-8bae-a95061241069/volumes/kubernetes.io~secret/serving-cert major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a1c3175b-e32e-420e-877f-71bfe64d9c6c/volumes/kubernetes.io~projected/kube-api-access-nwj8z:{mountpoint:/var/lib/kubelet/pods/a1c3175b-e32e-420e-877f-71bfe64d9c6c/volumes/kubernetes.io~projected/kube-api-access-nwj8z major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a1e578b0-2e77-4f52-8a44-379b6059c3d9/volumes/kubernetes.io~projected/kube-api-access-dsg9j:{mountpoint:/var/lib/kubelet/pods/a1e578b0-2e77-4f52-8a44-379b6059c3d9/volumes/kubernetes.io~projected/kube-api-access-dsg9j major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a2b450be-ca79-4edf-a1ea-377531a49519/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/a2b450be-ca79-4edf-a1ea-377531a49519/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:288 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b28ee2bb-0387-4d2d-8ca7-3fef91619c6b/volumes/kubernetes.io~projected/kube-api-access-8mnlt:{mountpoint:/var/lib/kubelet/pods/b28ee2bb-0387-4d2d-8ca7-3fef91619c6b/volumes/kubernetes.io~projected/kube-api-access-8mnlt major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b5014b57-9b83-4fc9-b0ed-55793b2b3a98/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/b5014b57-9b83-4fc9-b0ed-55793b2b3a98/volumes/kubernetes.io~secret/agent-certs major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b9ea9813-55b9-454b-8f10-985533305f85/volumes/kubernetes.io~projected/kube-api-access-bm8x9:{mountpoint:/var/lib/kubelet/pods/b9ea9813-55b9-454b-8f10-985533305f85/volumes/kubernetes.io~projected/kube-api-access-bm8x9 major:0 minor:361 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b9ea9813-55b9-454b-8f10-985533305f85/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/b9ea9813-55b9-454b-8f10-985533305f85/volumes/kubernetes.io~secret/signing-key major:0 minor:340 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d43a4e16-3834-4d31-bed4-42fc0aaaaf17/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/d43a4e16-3834-4d31-bed4-42fc0aaaaf17/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d43a4e16-3834-4d31-bed4-42fc0aaaaf17/volumes/kubernetes.io~projected/kube-api-access-hzcgz:{mountpoint:/var/lib/kubelet/pods/d43a4e16-3834-4d31-bed4-42fc0aaaaf17/volumes/kubernetes.io~projected/kube-api-access-hzcgz major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d43a4e16-3834-4d31-bed4-42fc0aaaaf17/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/d43a4e16-3834-4d31-bed4-42fc0aaaaf17/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dbbd2d70-8911-4aa5-96cf-3a16dcdf246d/volumes/kubernetes.io~projected/kube-api-access-bs5p9:{mountpoint:/var/lib/kubelet/pods/dbbd2d70-8911-4aa5-96cf-3a16dcdf246d/volumes/kubernetes.io~projected/kube-api-access-bs5p9 major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dda0c26f-8225-4239-bc27-866b66cbb919/volumes/kubernetes.io~projected/kube-api-access-jw4wt:{mountpoint:/var/lib/kubelet/pods/dda0c26f-8225-4239-bc27-866b66cbb919/volumes/kubernetes.io~projected/kube-api-access-jw4wt major:0 minor:222 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ddb04b2e-2724-44e9-948a-877a90d8a071/volumes/kubernetes.io~projected/kube-api-access-5wq6d:{mountpoint:/var/lib/kubelet/pods/ddb04b2e-2724-44e9-948a-877a90d8a071/volumes/kubernetes.io~projected/kube-api-access-5wq6d major:0 minor:227 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ddb04b2e-2724-44e9-948a-877a90d8a071/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/ddb04b2e-2724-44e9-948a-877a90d8a071/volumes/kubernetes.io~secret/serving-cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e64bd69d-946f-42ad-94ae-b87b7de3a581/volumes/kubernetes.io~projected/kube-api-access-gqk48:{mountpoint:/var/lib/kubelet/pods/e64bd69d-946f-42ad-94ae-b87b7de3a581/volumes/kubernetes.io~projected/kube-api-access-gqk48 major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/eb20e317-3ebe-4f5c-9b37-1de9ea5f063f/volumes/kubernetes.io~projected/kube-api-access-kvtwq:{mountpoint:/var/lib/kubelet/pods/eb20e317-3ebe-4f5c-9b37-1de9ea5f063f/volumes/kubernetes.io~projected/kube-api-access-kvtwq major:0 minor:309 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f032909d-bc39-4c9e-99b7-f3d800842119/volumes/kubernetes.io~projected/kube-api-access-9rblt:{mountpoint:/var/lib/kubelet/pods/f032909d-bc39-4c9e-99b7-f3d800842119/volumes/kubernetes.io~projected/kube-api-access-9rblt major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f032909d-bc39-4c9e-99b7-f3d800842119/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/f032909d-bc39-4c9e-99b7-f3d800842119/volumes/kubernetes.io~secret/serving-cert major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/febbc28e-bbcc-4961-bde8-09944d9ca680/volumes/kubernetes.io~projected/kube-api-access-jsrnd:{mountpoint:/var/lib/kubelet/pods/febbc28e-bbcc-4961-bde8-09944d9ca680/volumes/kubernetes.io~projected/kube-api-access-jsrnd major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/febbc28e-bbcc-4961-bde8-09944d9ca680/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/febbc28e-bbcc-4961-bde8-09944d9ca680/volumes/kubernetes.io~secret/serving-cert major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ff424288-19ca-4db0-9f16-fed9fb58404a/volumes/kubernetes.io~projected/kube-api-access-zk92l:{mountpoint:/var/lib/kubelet/pods/ff424288-19ca-4db0-9f16-fed9fb58404a/volumes/kubernetes.io~projected/kube-api-access-zk92l major:0 minor:216 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/3e60253d4bb79e3e88cf67c57782071ecf9c2700c20a614820f0d18a80f6c685/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/f5d5385ab0d1705c6602a85d1e9ace3c4d0cd317a761009ab70f36a290b27aa6/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/4e75beb0d29a95a1a2865e68de585cf24261e2f1ef4f7a7410d681842239555d/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/9632a1fb035f6dcc6aaa83bd4472feb48338a5ec93102a1c1e92cffd8b3ba112/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/961df6a3751c8336341958484a367f61295d69ed4794b8895ca26a27c2146c8f/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/e872c76757c3236fe70f808b948f95dc6f8066e4be5a8f46dcb2342350ede0bc/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/d3083515dc7cf3830f7eb88ea2e3dbfc76d1105faf6ad20826b95a7acc49edec/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/93bf4324f257073542266473a27422fd4fb9c055a5d27a2252fbd109bef0b0a0/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-136:{mountpoint:/var/lib/containers/storage/overlay/5d217f39d6c0966afb68017acd6ce56a71ee57074ab7c1b523ce2adae8af219b/merged major:0 minor:136 fsType:overlay blockSize:0} overlay_0-138:{mountpoint:/var/lib/containers/storage/overlay/b382cad7e18731694da2baa9fc1c26614eec6ccffa822fe33762e76f84de460b/merged major:0 minor:138 fsType:overlay blockSize:0} overlay_0-141:{mountpoint:/var/lib/containers/storage/overlay/9c5dd18100c3081ae94d00616fa14599d4912bba3622616fdddd11b66becd266/merged major:0 minor:141 fsType:overlay blockSize:0} overlay_0-143:{mountpoint:/var/lib/containers/storage/overlay/bb8ca4fbdbf8fc19c3fecd9975790c590918135bcbb2eb75684077a70d244686/merged major:0 minor:143 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/63156ff5a7fbdbdf8a7ddac8dfb98061cba14a88f8ecc0a8b161be9f9f5507fe/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/47400e7a35c37e39887125f7461b3fbc8dab5d6f045a780e61a7b40eb5b78a14/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/8cd04df6203f32587c10a3d66b2a184656598e2fd377d2dcab730fa361376650/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-169:{mountpoint:/var/lib/containers/storage/overlay/99e5f5fbb84bef05df616ee517ac22bdf13f493bb151dfdd6956fdd9ca40df64/merged major:0 minor:169 fsType:overlay blockSize:0} overlay_0-174:{mountpoint:/var/lib/containers/storage/overlay/3c4fd5132a8739834b68040320920f556f63cedc0393195c9a27e018bb2c7fc9/merged major:0 minor:174 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/786684e8b4e25ce1ac7db1c12d450a2755d28eede2b66a77ecc1e5b224d09560/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/5ab5bf20b8d40f2cf3b336f65557c93c828886158773efd5dcd3406447376ec9/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/01f98c1405ef4a8ed6c57d231a418f1534f13d50aa69dd2e7f391e96e9b97262/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/e1f1298f2f67c82132929543f1eaf8d233fe1d7ca5b8a72782efbd2cb9f2b08f/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/0cda66740da83643fe3c66646c23b96b09b80a2a1129ce7e0c37add031271a52/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/548eda8894cd0b0d42a382321ee09d3e9ff0135a87f6128b362d3e4f20f843d5/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/d22091e2b8d76e3dc88fe433b823fb7f749eb889cb4b99763aed58534c2f3a89/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/3be6e26e0b1b10011cdd4dff42821d9d06e318c475f9fe5ddc88839030aeb9b1/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/cb0a436f580a2bf5f9a08664650e5d6420b8293bb4ddb6f59bba189277f899c0/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/74b06d0086c944ee1edc4106897fb6e5bb0d53987b7d0a7e70aca9378e8c7c9b/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/dfa6612647033c1ee32a8bab4361ebfff188a19ebcfbb40e3d1884fe2ac0674b/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/1da0f14e3cd5a7877a733637e86afcbbf32f4d0dfd526e015e88ea63dc971b27/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-264:{mountpoint:/var/lib/containers/storage/overlay/e9c38ac4fb6386b1d878d306caf5eef99e834d9a16584f773f29a41daddf9921/merged major:0 minor:264 fsType:overlay blockSize:0} overlay_0-270:{mountpoint:/var/lib/containers/storage/overlay/fefca901ea066512624738313c7f99353279d04690ce069bc636a628b074a965/merged major:0 minor:270 fsType:overlay blockSize:0} overlay_0-276:{mountpoint:/var/lib/containers/storage/overlay/b78766884e263310c6209ca677ac058f22e3aec6a8d24c4780efe9757b06f104/merged major:0 minor:276 fsType:overlay blockSize:0} overlay_0-278:{mountpoint:/var/lib/containers/storage/overlay/3a0af71c64b06a8231cc77ab65f17a3ccda34b8b13fe116663cccb3b15348891/merged major:0 minor:278 fsType:overlay blockSize:0} overlay_0-280:{mountpoint:/var/lib/containers/storage/overlay/428d8ac4620a451641de1a7966d67deda8a59261cfcc8ebe804ec8c052b730f0/merged major:0 minor:280 fsType:overlay blockSize:0} overlay_0-282:{mountpoint:/var/lib/containers/storage/overlay/ba1278d11acb08d0a318af423cdde0a93210bf541d1a7ec80c82bb067dfbab32/merged major:0 minor:282 fsType:overlay blockSize:0} overlay_0-294:{mountpoint:/var/lib/containers/storage/overlay/bc65a849d6b30caeb9078e7035e35d4c28600a3c99fbcde6a1ab85b759312836/merged major:0 minor:294 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/b0f50db59714a641592ac6b305bbaee3fc3f66979a83ecce989329ccc45463a0/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-306:{mountpoint:/var/lib/containers/storage/overlay/a365685a223729d40cbf8f02d9d42a164e136613c8f777097e8573e329431dc2/merged major:0 minor:306 fsType:overlay blockSize:0} overlay_0-310:{mountpoint:/var/lib/containers/storage/overlay/c0e53bf0283a8d5794311fd4eca8bbcad8988218f506ba9b51fc7d9ef0b5523a/merged major:0 minor:310 fsType:overlay blockSize:0} overlay_0-320:{mountpoint:/var/lib/containers/storage/overlay/5a638c839658a958bccb300477d6f7a20a5dbffc51f8465d157aad6349be5d76/merged major:0 minor:320 fsType:overlay blockSize:0} overlay_0-326:{mountpoint:/var/lib/containers/storage/overlay/724eee3c14de869897abb2a4eaeb7d884d9c960659ad44201d85ee89ec5e45a2/merged major:0 minor:326 fsType:overlay blockSize:0} overlay_0-328:{mountpoint:/var/lib/containers/storage/overlay/ada7eb2bfdb4eef80aef5915330dfea3604498764316e328245c509eafeac94d/merged major:0 minor:328 fsType:overlay blockSize:0} overlay_0-330:{mountpoint:/var/lib/containers/storage/overlay/3064842dbf97e35af0768be760d95a6adaaa46e59529a84824cdeffbf4ff412e/merged major:0 minor:330 fsType:overlay blockSize:0} overlay_0-332:{mountpoint:/var/lib/containers/storage/overlay/74a043f339898783620a62e03346934b39c764c5bfb0e3fc4b191c6617dc70bc/merged major:0 minor:332 fsType:overlay blockSize:0} overlay_0-334:{mountpoint:/var/lib/containers/storage/overlay/eb314ce9a91230021bdee4505a818ac2db141af433eae15b13079e72f7ad5ab1/merged major:0 minor:334 fsType:overlay blockSize:0} overlay_0-336:{mountpoint:/var/lib/containers/storage/overlay/98d4c15ddb0a1f3b557fd5a9c3c52fef382b57ff53ecb0e8c80534b55a165c72/merged major:0 minor:336 fsType:overlay blockSize:0} overlay_0-338:{mountpoint:/var/lib/containers/storage/overlay/f6c39f371ce5c583efef899372fff871b19e304f28f3c9f601a9c1f372ec1cc3/merged major:0 minor:338 fsType:overlay blockSize:0} overlay_0-364:{mountpoint:/var/lib/containers/storage/overlay/14fdec3a7f961e5fbe2ac2f8a2be65417aa59ae760a8ef8eb263493cabc479e9/merged major:0 minor:364 fsType:overlay blockSize:0} overlay_0-366:{mountpoint:/var/lib/containers/storage/overlay/876997db283b063f2da6a98f669963be5045b1e2bd2d3511caf7b6394126c02c/merged major:0 minor:366 fsType:overlay blockSize:0} overlay_0-368:{mountpoint:/var/lib/containers/storage/overlay/ab3b61dfc730f506dc53deab11b2a767fdb3c9401b0800974d62c0e090690877/merged major:0 minor:368 fsType:overlay blockSize:0} overlay_0-374:{mountpoint:/var/lib/containers/storage/overlay/21d132ce4322ad30d5ca8d31690cd18722a5c9c3ac3b2c67319cd37cfe658cb5/merged major:0 minor:374 fsType:overlay blockSize:0} overlay_0-376:{mountpoint:/var/lib/containers/storage/overlay/12b3fddda68e3cd88eb943e9a9f13c6432527bf2800b52af4599c0f86b1bbf95/merged major:0 minor:376 fsType:overlay blockSize:0} overlay_0-49:{mountpoint:/var/lib/containers/storage/overlay/28c466c9ee633a1324a31103176b256e564ff02ae110e88dc4fc47af58d35e07/merged major:0 minor:49 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/d1c7c9dcdd7bc2cd042c5064960c91e46d37dea170a81781a012bd61d0aa95b9/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/ee702c7bb370288b61698f328aef0522ec4913738f4fc9542d3630ed4f72a29d/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/5dd88ecf12602e808fb8160c4e2be66fd7409d795b95070f1927a1fb0b172ab8/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/9ca568d94450517b48db781a341621a84a30f774c56a14c4391c8d6334d2898a/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/498022f79bfde6bf12322e10504d7bd9c250e8d4790b3ae8cd6595094be02f60/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/6180c3e3023cb0895e92786428c816fc754342ece5a56a14c075af0b17fdc24d/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-89:{mountpoint:/var/lib/containers/storage/overlay/25532e1c52749f39ca65e5dbebf42bb0378adeef3da476bf14e66bdb7505d0d4/merged major:0 minor:89 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/426e16e8ec566285aa6f0429375d4c469f2fb1629a723663a6cf0e9400cf10ee/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-97:{mountpoint:/var/lib/containers/storage/overlay/84fd3baea4c7f8e266389037c64a4b9eb65f53f2a19b4eaaa11d20110fa73fc4/merged major:0 minor:97 fsType:overlay blockSize:0}]
Oct 31 12:27:16.989420 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.988995    5060 manager.go:217] Machine: {Timestamp:2026-10-31 12:27:16.988524934 +0000 UTC m=+0.091592638 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101494 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec21afcc06851de4c41dc10297111780 SystemUUID:ec21afcc-0685-1de4-c41d-c10297111780 BootID:2f054b22-1f8d-49a7-a430-cfe17c0f04f1 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3cd295581fbb43bd6a480a27b300dbdfe231373fcec52e0d8ceeb87c35bd6500/userdata/shm DeviceMajor:0 DeviceMinor:95 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-328 DeviceMajor:0 DeviceMinor:328 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-332 DeviceMajor:0 DeviceMinor:332 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-326 DeviceMajor:0 DeviceMinor:326 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9083c35a-9739-4f3e-8415-bad1e6433d40/volumes/kubernetes.io~projected/kube-api-access-5xwrq DeviceMajor:0 DeviceMinor:215 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/6c661ce8-9f7b-4509-bd17-2380bcd1c425/volumes/kubernetes.io~projected/kube-api-access-nx5hk DeviceMajor:0 DeviceMinor:63 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-97 DeviceMajor:0 DeviceMinor:97 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-376 DeviceMajor:0 DeviceMinor:376 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1712b1a89fc033bb7af133fed6fccaed65abbd63124e20f0e3ea940d30aa7113/userdata/shm DeviceMajor:0 DeviceMinor:74 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/556dda7b-d8e6-4558-b5f3-994b01d6a866/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:208 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/dda0c26f-8225-4239-bc27-866b66cbb919/volumes/kubernetes.io~projected/kube-api-access-jw4wt DeviceMajor:0 DeviceMinor:222 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-264 DeviceMajor:0 DeviceMinor:264 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6bbe13aa-df4b-40b0-8595-68fe1166a082/volumes/kubernetes.io~projected/kube-api-access-7j2rp DeviceMajor:0 DeviceMinor:59 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b530060702c06b4b43a5a8cdf318bc791c379fc3c905612530366b47f66d02c8/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-338 DeviceMajor:0 DeviceMinor:338 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/432c0f75ce4f14ee6e21e05374e78fc51fdb2f101584925475c6900c584e1409/userdata/shm DeviceMajor:0 DeviceMinor:232 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-270 DeviceMajor:0 DeviceMinor:270 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a2b450be-ca79-4edf-a1ea-377531a49519/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:288 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8dbe3066e41c634d3607763741f771139a4be4a86fb97f302e275ecb6c7399bd/userdata/shm DeviceMajor:0 DeviceMinor:296 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/07714f20-a70e-49e8-8e51-d03a8e00005a/volumes/kubernetes.io~projected/kube-api-access-hqmhd DeviceMajor:0 DeviceMinor:218 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a1c3175b-e32e-420e-877f-71bfe64d9c6c/volumes/kubernetes.io~projected/kube-api-access-nwj8z DeviceMajor:0 DeviceMinor:58 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0db116b4-d971-4966-bb05-db28d077b6fe/volumes/kubernetes.io~projected/kube-api-access-nlgsp DeviceMajor:0 DeviceMinor:225 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/6b3e77ff-4445-42fa-a7c3-f35d442c276d/volumes/kubernetes.io~projected/kube-api-access-vknw2 DeviceMajor:0 DeviceMinor:66 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9083c35a-9739-4f3e-8415-bad1e6433d40/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:197 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0db116b4-d971-4966-bb05-db28d077b6fe/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:204 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ff424288-19ca-4db0-9f16-fed9fb58404a/volumes/kubernetes.io~projected/kube-api-access-zk92l DeviceMajor:0 DeviceMinor:216 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-310 DeviceMajor:0 DeviceMinor:310 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5c8865f9-fdf6-49b9-8b4f-69dbc7787084/volumes/kubernetes.io~projected/kube-api-access-mhm56 DeviceMajor:0 DeviceMinor:220 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-294 DeviceMajor:0 DeviceMinor:294 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e64bd69d-946f-42ad-94ae-b87b7de3a581/volumes/kubernetes.io~projected/kube-api-access-gqk48 DeviceMajor:0 DeviceMinor:65 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1fe0c2cef9e318b723913d1aafd071a4c4a50766fab690a4b526ef633e216e36/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-136 DeviceMajor:0 DeviceMinor:136 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-141 DeviceMajor:0 DeviceMinor:141 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d43a4e16-3834-4d31-bed4-42fc0aaaaf17/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/febbc28e-bbcc-4961-bde8-09944d9ca680/volumes/kubernetes.io~projected/kube-api-access-jsrnd DeviceMajor:0 DeviceMinor:210 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-330 DeviceMajor:0 DeviceMinor:330 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/891b3c47-ff7c-4a7a-a09b-411361a843f7/volumes/kubernetes.io~projected/kube-api-access-mrpck DeviceMajor:0 DeviceMinor:226 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-278 DeviceMajor:0 DeviceMinor:278 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:57 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f032909d-bc39-4c9e-99b7-f3d800842119/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:209 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/25691fa3-7017-429a-b94c-c2dde4d29e55/volumes/kubernetes.io~projected/kube-api-access-gvqpw DeviceMajor:0 DeviceMinor:214 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0db116b4-d971-4966-bb05-db28d077b6fe/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:224 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-280 DeviceMajor:0 DeviceMinor:280 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7a402472-ca33-44ae-b598-96fc8e7a5eb2/volumes/kubernetes.io~projected/kube-api-access-2tf6n DeviceMajor:0 DeviceMinor:293 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee/volumes/kubernetes.io~projected/kube-api-access-t5nx4 DeviceMajor:0 DeviceMinor:60 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/99a606a8-546b-4145-8bae-a95061241069/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:203 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-374 DeviceMajor:0 DeviceMinor:374 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/891b3c47-ff7c-4a7a-a09b-411361a843f7/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:201 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8a59f3fca9f665e623be85a15c0d77fe56925fd62aae37db535bb9a23c596a6f/userdata/shm DeviceMajor:0 DeviceMinor:228 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/031551df-64a5-411b-8eb8-7f0bf2d4f170/volumes/kubernetes.io~projected/kube-api-access-wpcl9 DeviceMajor:0 DeviceMinor:319 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-320 DeviceMajor:0 DeviceMinor:320 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f5a0fca700cd53f8f083df04ab0ae9b664259a71e5b410fd4990db2d3ea0b60b/userdata/shm DeviceMajor:0 DeviceMinor:362 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/432c216b017de6fb2dc9ee38001b85f4604b2715e988e0d643b46af056985f7d/userdata/shm DeviceMajor:0 DeviceMinor:246 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/556dda7b-d8e6-4558-b5f3-994b01d6a866/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:206 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f032909d-bc39-4c9e-99b7-f3d800842119/volumes/kubernetes.io~projected/kube-api-access-9rblt DeviceMajor:0 DeviceMinor:217 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/556dda7b-d8e6-4558-b5f3-994b01d6a866/volumes/kubernetes.io~projected/kube-api-access-9g26m DeviceMajor:0 DeviceMinor:223 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5a4af82d83c4e2d729012b35f36754f633c7d8edae532b658e217176aeed8416/userdata/shm DeviceMajor:0 DeviceMinor:230 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a130b0e336cef0401531b357c6118d6c3ec61ce95a7b67c9eda4cf43d7a4f443/userdata/shm DeviceMajor:0 DeviceMinor:238 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/24f74c6f47a4176d951ed9f1e88a53b5b9928f802c402c0074106d93796b3218/userdata/shm DeviceMajor:0 DeviceMinor:243 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-276 DeviceMajor:0 DeviceMinor:276 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-334 DeviceMajor:0 DeviceMinor:334 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ddb04b2e-2724-44e9-948a-877a90d8a071/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:45 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/613125d4-fbb4-4b43-9c47-24e039bd4697/volumes/kubernetes.io~projected/kube-api-access-mvzh5 DeviceMajor:0 DeviceMinor:212 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b9ea9813-55b9-454b-8f10-985533305f85/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:340 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-366 DeviceMajor:0 DeviceMinor:366 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d43a4e16-3834-4d31-bed4-42fc0aaaaf17/volumes/kubernetes.io~projected/kube-api-access-hzcgz DeviceMajor:0 DeviceMinor:61 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e4538bb23cd43573e0f29c86571dcd40acc1137dc3413f283057119ec8a4a608/userdata/shm DeviceMajor:0 DeviceMinor:88 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/dd140492a16d3d96e416a11f11f04bef073fc5e086c5ae9eeeb1a019605577c1/userdata/shm DeviceMajor:0 DeviceMinor:242 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-336 DeviceMajor:0 DeviceMinor:336 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-174 DeviceMajor:0 DeviceMinor:174 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6aaae0ad5cd357b195d055aaf85b96bed388d3a4f75a1ce9caad08ede506e3c7/userdata/shm DeviceMajor:0 DeviceMinor:301 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:55 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b5014b57-9b83-4fc9-b0ed-55793b2b3a98/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:64 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8a79b2069dde730f773d0ac904f6e23f6db008b97e8fd24680a8e7fa053f2e9f/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9083c35a-9739-4f3e-8415-bad1e6433d40/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:202 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-143 DeviceMajor:0 DeviceMinor:143 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-169 DeviceMajor:0 DeviceMinor:169 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0db116b4-d971-4966-bb05-db28d077b6fe/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:207 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9083c35a-9739-4f3e-8415-bad1e6433d40/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:211 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/68516ca2184b4f7cae9b513be83b9d11699718f173a9dfde5a4ffb4a1031cddc/userdata/shm DeviceMajor:0 DeviceMinor:240 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c8dd1a20232d5b64644dd25931795726b9f8c36ada12fd4da9078653557feddd/userdata/shm DeviceMajor:0 DeviceMinor:323 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d14f2152e247518ab96f882b2d1c789e2985ca7b210789682e82d1d99b01a2ea/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-89 DeviceMajor:0 DeviceMinor:89 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-364 DeviceMajor:0 DeviceMinor:364 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-138 DeviceMajor:0 DeviceMinor:138 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c1849c803f134d969c39443fe4e01b9f3a600259e4ff5749f6d3ca25a2a168b1/userdata/shm DeviceMajor:0 DeviceMinor:234 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8738d804fa86e15f0597349bd890e4bfa1c5c1415beaacbab01aad9c000fbd0a/userdata/shm DeviceMajor:0 DeviceMinor:314 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b539c38438f2cfd785f2a2a6e023750667ea93e764840b11ad42f359c3687b08/userdata/shm DeviceMajor:0 DeviceMinor:51 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d43a4e16-3834-4d31-bed4-42fc0aaaaf17/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:56 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a1e578b0-2e77-4f52-8a44-379b6059c3d9/volumes/kubernetes.io~projected/kube-api-access-dsg9j DeviceMajor:0 DeviceMinor:67 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/eb20e317-3ebe-4f5c-9b37-1de9ea5f063f/volumes/kubernetes.io~projected/kube-api-access-kvtwq DeviceMajor:0 DeviceMinor:309 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b9ea9813-55b9-454b-8f10-985533305f85/volumes/kubernetes.io~projected/kube-api-access-bm8x9 DeviceMajor:0 DeviceMinor:361 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ddb04b2e-2724-44e9-948a-877a90d8a071/volumes/kubernetes.io~projected/kube-api-access-5wq6d DeviceMajor:0 DeviceMinor:227 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/dbbd2d70-8911-4aa5-96cf-3a16dcdf246d/volumes/kubernetes.io~projected/kube-api-access-bs5p9 DeviceMajor:0 DeviceMinor:219 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9ec8c5086bd55eeaa1cf3cd8ee3146133e8915c24281d20fe7de74ac684c8429/userdata/shm DeviceMajor:0 DeviceMinor:236 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/64649bfe-c3e6-469a-a9ff-0db1ca0fadb3/volumes/kubernetes.io~projected/kube-api-access-48lnl DeviceMajor:0 DeviceMinor:318 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c9ca40edbf0dd49e6bf694363da3d39a56d8bd8ff7baef573fe1d91ab10933b6/userdata/shm DeviceMajor:0 DeviceMinor:322 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3bb2c04e-55a2-4206-9530-7298f4469379/volumes/kubernetes.io~projected/kube-api-access-xv9qt DeviceMajor:0 DeviceMinor:62 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/febbc28e-bbcc-4961-bde8-09944d9ca680/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:205 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/99a606a8-546b-4145-8bae-a95061241069/volumes/kubernetes.io~projected/kube-api-access-6t4dc DeviceMajor:0 DeviceMinor:221 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-306 DeviceMajor:0 DeviceMinor:306 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9557a3f0-ae0f-41f9-a9b5-15aeda410a38/volumes/kubernetes.io~projected/kube-api-access-jl4xd DeviceMajor:0 DeviceMinor:308 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-368 DeviceMajor:0 DeviceMinor:368 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b61159ca029793099370ff0779dda563c7d5600e24e1415f4ca6047a637d4ed7/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-49 DeviceMajor:0 DeviceMinor:49 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c4cb00c97f37c42bcd3b91b32e6ec987fd6fd61186b49b744d6301563f405c70/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-282 DeviceMajor:0 DeviceMinor:282 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b28ee2bb-0387-4d2d-8ca7-3fef91619c6b/volumes/kubernetes.io~projected/kube-api-access-8mnlt DeviceMajor:0 DeviceMinor:213 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:24f74c6f47a4176 MacAddress:e6:76:5e:a3:86:ae Speed:10000 Mtu:8901} {Name:432c0f75ce4f14e MacAddress:02:60:60:3b:23:a7 Speed:10000 Mtu:8901} {Name:432c216b017de6f MacAddress:b6:1f:2d:a3:1a:4a Speed:10000 Mtu:8901} {Name:5a4af82d83c4e2d MacAddress:d6:77:21:f1:41:f0 Speed:10000 Mtu:8901} {Name:68516ca2184b4f7 MacAddress:62:f1:62:96:b7:0c Speed:10000 Mtu:8901} {Name:6aaae0ad5cd357b MacAddress:ea:ff:98:f5:f5:a0 Speed:10000 Mtu:8901} {Name:8738d804fa86e15 MacAddress:d6:5e:60:ad:37:c6 Speed:10000 Mtu:8901} {Name:8a59f3fca9f665e MacAddress:3a:fc:e8:a0:d4:8b Speed:10000 Mtu:8901} {Name:8dbe3066e41c634 MacAddress:2e:f5:9e:33:6c:eb Speed:10000 Mtu:8901} {Name:9ec8c5086bd55ee MacAddress:32:2a:65:c5:ef:32 Speed:10000 Mtu:8901} {Name:a130b0e336cef04 MacAddress:ba:8f:84:5b:09:74 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:f2:f3:ef:db Speed:0 Mtu:9001} {Name:br-int MacAddress:4e:80:80:0b:eb:dc Speed:0 Mtu:8901} {Name:c1849c803f134d9 MacAddress:d2:8b:15:36:58:ec Speed:10000 Mtu:8901} {Name:c8dd1a20232d5b6 MacAddress:9e:56:0b:7e:e9:03 Speed:10000 Mtu:8901} {Name:c9ca40edbf0dd49 MacAddress:0e:9d:f6:72:9c:62 Speed:10000 Mtu:8901} {Name:dd140492a16d3d9 MacAddress:d2:9b:b2:56:f4:e6 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:f2:f3:ef:db Speed:0 Mtu:9001} {Name:f5a0fca700cd53f MacAddress:b6:37:49:30:91:40 Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:a6:48:43:66:06:ee Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:7a:5e:81:5a:f1:15 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 31 12:27:16.989668 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.989425    5060 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 31 12:27:16.989668 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.989471    5060 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 31 12:27:16.989793 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.989768    5060 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 31 12:27:16.989925 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.989794    5060 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-190.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 31 12:27:16.989968 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.989927    5060 topology_manager.go:143] "Creating topology manager with none policy"
Oct 31 12:27:16.989968 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.989935    5060 container_manager_linux.go:311] "Creating device plugin manager"
Oct 31 12:27:16.989968 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.989947    5060 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 31 12:27:16.989968 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.989966    5060 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 31 12:27:16.990066 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.989986    5060 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 31 12:27:16.990147 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.990138    5060 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 31 12:27:16.990245 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.990236    5060 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 31 12:27:16.990279 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.990272    5060 kubelet.go:493] "Attempting to sync node with API server"
Oct 31 12:27:16.990310 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.990283    5060 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 31 12:27:16.990310 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.990293    5060 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 31 12:27:16.990310 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.990302    5060 kubelet.go:405] "Adding apiserver pod source"
Oct 31 12:27:16.990310 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.990309    5060 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 31 12:27:16.991086 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.991071    5060 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 31 12:27:16.991086 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.991083    5060 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 31 12:27:16.991605 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.991536    5060 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 31 12:27:16.995527 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.995511    5060 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 31 12:27:16.995527 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.995525    5060 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 31 12:27:16.995647 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.995637    5060 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Oct 31 12:27:16.995859 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.995847    5060 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 31 12:27:16.995894 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.995879    5060 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 31 12:27:16.996011 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.995999    5060 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 31 12:27:16.996059 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.996020    5060 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 31 12:27:16.996059 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.996033    5060 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 31 12:27:16.996059 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.996046    5060 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 31 12:27:16.996059 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.996054    5060 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 31 12:27:16.996187 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.996063    5060 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 31 12:27:16.996187 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.996091    5060 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 31 12:27:16.996187 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.996097    5060 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 31 12:27:16.996187 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.996104    5060 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 31 12:27:16.996187 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.996110    5060 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 31 12:27:16.996187 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.996128    5060 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 31 12:27:16.996187 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.996144    5060 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 31 12:27:16.996187 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.996170    5060 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 31 12:27:16.996187 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.996176    5060 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 31 12:27:16.999754 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.999739    5060 server.go:1263] "Started kubelet"
Oct 31 12:27:16.999831 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.999782    5060 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 31 12:27:17.000265 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:16.999980    5060 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 31 12:27:17.000343 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.000262    5060 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 31 12:27:17.000378 ip-10-0-130-190 systemd[1]: Started Kubernetes Kubelet.
Oct 31 12:27:17.003155 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.000636    5060 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 31 12:27:17.003155 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.002097    5060 server.go:317] "Adding debug handlers to kubelet server"
Oct 31 12:27:17.007815 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.007795    5060 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 31 12:27:17.008184 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.008158    5060 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 31 12:27:17.008978 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:27:17.008686    5060 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-190.us-east-2.compute.internal\" not found"
Oct 31 12:27:17.008978 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.008703    5060 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 31 12:27:17.008978 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.008718    5060 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 31 12:27:17.008978 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.008772    5060 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 12:21:37 +0000 UTC" deadline="2028-05-09 12:27:04.598084716 +0000 UTC"
Oct 31 12:27:17.008978 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.008800    5060 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13343h59m47.589288709s"
Oct 31 12:27:17.008978 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.008824    5060 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 31 12:27:17.011507 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.011461    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9083c35a-9739-4f3e-8415-bad1e6433d40" volumeName="kubernetes.io/secret/9083c35a-9739-4f3e-8415-bad1e6433d40-installation-pull-secrets" seLinuxMountContext=""
Oct 31 12:27:17.011507 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.011507    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dda0c26f-8225-4239-bc27-866b66cbb919" volumeName="kubernetes.io/projected/dda0c26f-8225-4239-bc27-866b66cbb919-kube-api-access-jw4wt" seLinuxMountContext=""
Oct 31 12:27:17.011659 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.011518    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ddb04b2e-2724-44e9-948a-877a90d8a071" volumeName="kubernetes.io/empty-dir/ddb04b2e-2724-44e9-948a-877a90d8a071-snapshots" seLinuxMountContext=""
Oct 31 12:27:17.011659 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.011527    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ddb04b2e-2724-44e9-948a-877a90d8a071" volumeName="kubernetes.io/projected/ddb04b2e-2724-44e9-948a-877a90d8a071-kube-api-access-5wq6d" seLinuxMountContext=""
Oct 31 12:27:17.011659 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.011536    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eb20e317-3ebe-4f5c-9b37-1de9ea5f063f" volumeName="kubernetes.io/projected/eb20e317-3ebe-4f5c-9b37-1de9ea5f063f-kube-api-access-kvtwq" seLinuxMountContext=""
Oct 31 12:27:17.011659 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.011544    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3bb2c04e-55a2-4206-9530-7298f4469379" volumeName="kubernetes.io/projected/3bb2c04e-55a2-4206-9530-7298f4469379-kube-api-access-xv9qt" seLinuxMountContext=""
Oct 31 12:27:17.011659 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.011551    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b9ea9813-55b9-454b-8f10-985533305f85" volumeName="kubernetes.io/projected/b9ea9813-55b9-454b-8f10-985533305f85-kube-api-access-bm8x9" seLinuxMountContext=""
Oct 31 12:27:17.011659 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.011558    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ddb04b2e-2724-44e9-948a-877a90d8a071" volumeName="kubernetes.io/configmap/ddb04b2e-2724-44e9-948a-877a90d8a071-trusted-ca-bundle" seLinuxMountContext=""
Oct 31 12:27:17.011659 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.011567    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f032909d-bc39-4c9e-99b7-f3d800842119" volumeName="kubernetes.io/configmap/f032909d-bc39-4c9e-99b7-f3d800842119-config" seLinuxMountContext=""
Oct 31 12:27:17.011659 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.011574    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ff424288-19ca-4db0-9f16-fed9fb58404a" volumeName="kubernetes.io/projected/ff424288-19ca-4db0-9f16-fed9fb58404a-kube-api-access-zk92l" seLinuxMountContext=""
Oct 31 12:27:17.011659 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.011581    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5c8865f9-fdf6-49b9-8b4f-69dbc7787084" volumeName="kubernetes.io/empty-dir/5c8865f9-fdf6-49b9-8b4f-69dbc7787084-tmp-dir" seLinuxMountContext=""
Oct 31 12:27:17.011659 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.011588    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="613125d4-fbb4-4b43-9c47-24e039bd4697" volumeName="kubernetes.io/projected/613125d4-fbb4-4b43-9c47-24e039bd4697-kube-api-access-mvzh5" seLinuxMountContext=""
Oct 31 12:27:17.011659 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.011597    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9083c35a-9739-4f3e-8415-bad1e6433d40" volumeName="kubernetes.io/configmap/9083c35a-9739-4f3e-8415-bad1e6433d40-registry-certificates" seLinuxMountContext=""
Oct 31 12:27:17.011659 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.011607    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a1c3175b-e32e-420e-877f-71bfe64d9c6c" volumeName="kubernetes.io/configmap/a1c3175b-e32e-420e-877f-71bfe64d9c6c-cni-binary-copy" seLinuxMountContext=""
Oct 31 12:27:17.011659 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.011623    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b9ea9813-55b9-454b-8f10-985533305f85" volumeName="kubernetes.io/configmap/b9ea9813-55b9-454b-8f10-985533305f85-signing-cabundle" seLinuxMountContext=""
Oct 31 12:27:17.011659 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.011633    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b9ea9813-55b9-454b-8f10-985533305f85" volumeName="kubernetes.io/secret/b9ea9813-55b9-454b-8f10-985533305f85-signing-key" seLinuxMountContext=""
Oct 31 12:27:17.012323 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.011645    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b5014b57-9b83-4fc9-b0ed-55793b2b3a98" volumeName="kubernetes.io/configmap/b5014b57-9b83-4fc9-b0ed-55793b2b3a98-konnectivity-ca" seLinuxMountContext=""
Oct 31 12:27:17.012323 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.011705    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b5014b57-9b83-4fc9-b0ed-55793b2b3a98" volumeName="kubernetes.io/secret/b5014b57-9b83-4fc9-b0ed-55793b2b3a98-agent-certs" seLinuxMountContext=""
Oct 31 12:27:17.012323 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.011719    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="64649bfe-c3e6-469a-a9ff-0db1ca0fadb3" volumeName="kubernetes.io/projected/64649bfe-c3e6-469a-a9ff-0db1ca0fadb3-kube-api-access-48lnl" seLinuxMountContext=""
Oct 31 12:27:17.012323 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.011732    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0db116b4-d971-4966-bb05-db28d077b6fe" volumeName="kubernetes.io/projected/0db116b4-d971-4966-bb05-db28d077b6fe-kube-api-access-nlgsp" seLinuxMountContext=""
Oct 31 12:27:17.012323 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.011742    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="25691fa3-7017-429a-b94c-c2dde4d29e55" volumeName="kubernetes.io/projected/25691fa3-7017-429a-b94c-c2dde4d29e55-kube-api-access-gvqpw" seLinuxMountContext=""
Oct 31 12:27:17.012323 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.011753    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eb20e317-3ebe-4f5c-9b37-1de9ea5f063f" volumeName="kubernetes.io/configmap/eb20e317-3ebe-4f5c-9b37-1de9ea5f063f-service-ca-bundle" seLinuxMountContext=""
Oct 31 12:27:17.012323 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.011765    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eb20e317-3ebe-4f5c-9b37-1de9ea5f063f" volumeName="kubernetes.io/configmap/eb20e317-3ebe-4f5c-9b37-1de9ea5f063f-trusted-ca-bundle" seLinuxMountContext=""
Oct 31 12:27:17.012323 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.011775    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="031551df-64a5-411b-8eb8-7f0bf2d4f170" volumeName="kubernetes.io/projected/031551df-64a5-411b-8eb8-7f0bf2d4f170-kube-api-access-wpcl9" seLinuxMountContext=""
Oct 31 12:27:17.012323 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.011792    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee" volumeName="kubernetes.io/empty-dir/51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee-tmp" seLinuxMountContext=""
Oct 31 12:27:17.012323 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.011804    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6bbe13aa-df4b-40b0-8595-68fe1166a082" volumeName="kubernetes.io/empty-dir/6bbe13aa-df4b-40b0-8595-68fe1166a082-tmp-dir" seLinuxMountContext=""
Oct 31 12:27:17.012323 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.011826    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="99a606a8-546b-4145-8bae-a95061241069" volumeName="kubernetes.io/projected/99a606a8-546b-4145-8bae-a95061241069-kube-api-access-6t4dc" seLinuxMountContext=""
Oct 31 12:27:17.012323 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.011839    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="febbc28e-bbcc-4961-bde8-09944d9ca680" volumeName="kubernetes.io/configmap/febbc28e-bbcc-4961-bde8-09944d9ca680-config" seLinuxMountContext=""
Oct 31 12:27:17.012323 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.011869    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="febbc28e-bbcc-4961-bde8-09944d9ca680" volumeName="kubernetes.io/projected/febbc28e-bbcc-4961-bde8-09944d9ca680-kube-api-access-jsrnd" seLinuxMountContext=""
Oct 31 12:27:17.012323 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.011884    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="891b3c47-ff7c-4a7a-a09b-411361a843f7" volumeName="kubernetes.io/secret/891b3c47-ff7c-4a7a-a09b-411361a843f7-cert" seLinuxMountContext=""
Oct 31 12:27:17.012323 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.011896    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="99a606a8-546b-4145-8bae-a95061241069" volumeName="kubernetes.io/secret/99a606a8-546b-4145-8bae-a95061241069-serving-cert" seLinuxMountContext=""
Oct 31 12:27:17.012323 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.011906    5060 factory.go:55] Registering systemd factory
Oct 31 12:27:17.012323 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.011916    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a1c3175b-e32e-420e-877f-71bfe64d9c6c" volumeName="kubernetes.io/configmap/a1c3175b-e32e-420e-877f-71bfe64d9c6c-whereabouts-flatfile-configmap" seLinuxMountContext=""
Oct 31 12:27:17.012323 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.011923    5060 factory.go:223] Registration of the systemd container factory successfully
Oct 31 12:27:17.012323 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.011928    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a1c3175b-e32e-420e-877f-71bfe64d9c6c" volumeName="kubernetes.io/projected/a1c3175b-e32e-420e-877f-71bfe64d9c6c-kube-api-access-nwj8z" seLinuxMountContext=""
Oct 31 12:27:17.012323 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.011941    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d43a4e16-3834-4d31-bed4-42fc0aaaaf17" volumeName="kubernetes.io/projected/d43a4e16-3834-4d31-bed4-42fc0aaaaf17-kube-api-access-hzcgz" seLinuxMountContext=""
Oct 31 12:27:17.012323 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.011954    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eb20e317-3ebe-4f5c-9b37-1de9ea5f063f" volumeName="kubernetes.io/empty-dir/eb20e317-3ebe-4f5c-9b37-1de9ea5f063f-archives-path" seLinuxMountContext=""
Oct 31 12:27:17.012323 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.011968    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee" volumeName="kubernetes.io/empty-dir/51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee-etc-tuned" seLinuxMountContext=""
Oct 31 12:27:17.012323 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.011986    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="febbc28e-bbcc-4961-bde8-09944d9ca680" volumeName="kubernetes.io/secret/febbc28e-bbcc-4961-bde8-09944d9ca680-serving-cert" seLinuxMountContext=""
Oct 31 12:27:17.012323 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.012009    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="556dda7b-d8e6-4558-b5f3-994b01d6a866" volumeName="kubernetes.io/secret/556dda7b-d8e6-4558-b5f3-994b01d6a866-default-certificate" seLinuxMountContext=""
Oct 31 12:27:17.012323 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.012022    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6b3e77ff-4445-42fa-a7c3-f35d442c276d" volumeName="kubernetes.io/configmap/6b3e77ff-4445-42fa-a7c3-f35d442c276d-multus-daemon-config" seLinuxMountContext=""
Oct 31 12:27:17.012323 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.012033    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3bb2c04e-55a2-4206-9530-7298f4469379" volumeName="kubernetes.io/configmap/3bb2c04e-55a2-4206-9530-7298f4469379-iptables-alerter-script" seLinuxMountContext=""
Oct 31 12:27:17.012323 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.012044    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0db116b4-d971-4966-bb05-db28d077b6fe" volumeName="kubernetes.io/configmap/0db116b4-d971-4966-bb05-db28d077b6fe-trusted-ca" seLinuxMountContext=""
Oct 31 12:27:17.012323 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.012073    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="556dda7b-d8e6-4558-b5f3-994b01d6a866" volumeName="kubernetes.io/secret/556dda7b-d8e6-4558-b5f3-994b01d6a866-stats-auth" seLinuxMountContext=""
Oct 31 12:27:17.012323 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.012087    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6c661ce8-9f7b-4509-bd17-2380bcd1c425" volumeName="kubernetes.io/projected/6c661ce8-9f7b-4509-bd17-2380bcd1c425-kube-api-access-nx5hk" seLinuxMountContext=""
Oct 31 12:27:17.012323 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.012101    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d43a4e16-3834-4d31-bed4-42fc0aaaaf17" volumeName="kubernetes.io/configmap/d43a4e16-3834-4d31-bed4-42fc0aaaaf17-ovnkube-script-lib" seLinuxMountContext=""
Oct 31 12:27:17.012323 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.012115    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dda0c26f-8225-4239-bc27-866b66cbb919" volumeName="kubernetes.io/configmap/dda0c26f-8225-4239-bc27-866b66cbb919-telemetry-config" seLinuxMountContext=""
Oct 31 12:27:17.012323 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.012125    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="07714f20-a70e-49e8-8e51-d03a8e00005a" volumeName="kubernetes.io/projected/07714f20-a70e-49e8-8e51-d03a8e00005a-kube-api-access-hqmhd" seLinuxMountContext=""
Oct 31 12:27:17.012323 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.012139    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6b3e77ff-4445-42fa-a7c3-f35d442c276d" volumeName="kubernetes.io/projected/6b3e77ff-4445-42fa-a7c3-f35d442c276d-kube-api-access-vknw2" seLinuxMountContext=""
Oct 31 12:27:17.012323 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.012143    5060 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 12:27:17.012323 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.012150    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9083c35a-9739-4f3e-8415-bad1e6433d40" volumeName="kubernetes.io/empty-dir/9083c35a-9739-4f3e-8415-bad1e6433d40-ca-trust-extracted" seLinuxMountContext=""
Oct 31 12:27:17.012323 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.012175    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dbbd2d70-8911-4aa5-96cf-3a16dcdf246d" volumeName="kubernetes.io/configmap/dbbd2d70-8911-4aa5-96cf-3a16dcdf246d-samples-operator-config" seLinuxMountContext=""
Oct 31 12:27:17.012323 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.012201    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dbbd2d70-8911-4aa5-96cf-3a16dcdf246d" volumeName="kubernetes.io/projected/dbbd2d70-8911-4aa5-96cf-3a16dcdf246d-kube-api-access-bs5p9" seLinuxMountContext=""
Oct 31 12:27:17.012323 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.012214    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ddb04b2e-2724-44e9-948a-877a90d8a071" volumeName="kubernetes.io/empty-dir/ddb04b2e-2724-44e9-948a-877a90d8a071-tmp" seLinuxMountContext=""
Oct 31 12:27:17.012323 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.012226    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0db116b4-d971-4966-bb05-db28d077b6fe" volumeName="kubernetes.io/configmap/0db116b4-d971-4966-bb05-db28d077b6fe-registry-certificates" seLinuxMountContext=""
Oct 31 12:27:17.012323 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.012249    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0db116b4-d971-4966-bb05-db28d077b6fe" volumeName="kubernetes.io/secret/0db116b4-d971-4966-bb05-db28d077b6fe-image-registry-private-configuration" seLinuxMountContext=""
Oct 31 12:27:17.012323 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.012264    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="99a606a8-546b-4145-8bae-a95061241069" volumeName="kubernetes.io/configmap/99a606a8-546b-4145-8bae-a95061241069-config" seLinuxMountContext=""
Oct 31 12:27:17.012323 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.012293    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="556dda7b-d8e6-4558-b5f3-994b01d6a866" volumeName="kubernetes.io/projected/556dda7b-d8e6-4558-b5f3-994b01d6a866-kube-api-access-9g26m" seLinuxMountContext=""
Oct 31 12:27:17.012323 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.012313    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9083c35a-9739-4f3e-8415-bad1e6433d40" volumeName="kubernetes.io/configmap/9083c35a-9739-4f3e-8415-bad1e6433d40-trusted-ca" seLinuxMountContext=""
Oct 31 12:27:17.012323 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.012353    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a1e578b0-2e77-4f52-8a44-379b6059c3d9" volumeName="kubernetes.io/projected/a1e578b0-2e77-4f52-8a44-379b6059c3d9-kube-api-access-dsg9j" seLinuxMountContext=""
Oct 31 12:27:17.014617 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.012366    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d43a4e16-3834-4d31-bed4-42fc0aaaaf17" volumeName="kubernetes.io/configmap/d43a4e16-3834-4d31-bed4-42fc0aaaaf17-ovnkube-config" seLinuxMountContext=""
Oct 31 12:27:17.014617 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.012388    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ddb04b2e-2724-44e9-948a-877a90d8a071" volumeName="kubernetes.io/secret/ddb04b2e-2724-44e9-948a-877a90d8a071-serving-cert" seLinuxMountContext=""
Oct 31 12:27:17.014617 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.012401    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a1c3175b-e32e-420e-877f-71bfe64d9c6c" volumeName="kubernetes.io/configmap/a1c3175b-e32e-420e-877f-71bfe64d9c6c-cni-sysctl-allowlist" seLinuxMountContext=""
Oct 31 12:27:17.014617 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.012413    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5c8865f9-fdf6-49b9-8b4f-69dbc7787084" volumeName="kubernetes.io/projected/5c8865f9-fdf6-49b9-8b4f-69dbc7787084-kube-api-access-mhm56" seLinuxMountContext=""
Oct 31 12:27:17.014617 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.012435    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0db116b4-d971-4966-bb05-db28d077b6fe" volumeName="kubernetes.io/projected/0db116b4-d971-4966-bb05-db28d077b6fe-bound-sa-token" seLinuxMountContext=""
Oct 31 12:27:17.014617 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.012451    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0db116b4-d971-4966-bb05-db28d077b6fe" volumeName="kubernetes.io/secret/0db116b4-d971-4966-bb05-db28d077b6fe-installation-pull-secrets" seLinuxMountContext=""
Oct 31 12:27:17.014617 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.012461    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6c661ce8-9f7b-4509-bd17-2380bcd1c425" volumeName="kubernetes.io/configmap/6c661ce8-9f7b-4509-bd17-2380bcd1c425-serviceca" seLinuxMountContext=""
Oct 31 12:27:17.014617 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.012474    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7a402472-ca33-44ae-b598-96fc8e7a5eb2" volumeName="kubernetes.io/projected/7a402472-ca33-44ae-b598-96fc8e7a5eb2-kube-api-access-2tf6n" seLinuxMountContext=""
Oct 31 12:27:17.014617 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.012484    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6b3e77ff-4445-42fa-a7c3-f35d442c276d" volumeName="kubernetes.io/configmap/6b3e77ff-4445-42fa-a7c3-f35d442c276d-cni-binary-copy" seLinuxMountContext=""
Oct 31 12:27:17.014617 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.012523    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="891b3c47-ff7c-4a7a-a09b-411361a843f7" volumeName="kubernetes.io/projected/891b3c47-ff7c-4a7a-a09b-411361a843f7-kube-api-access-mrpck" seLinuxMountContext=""
Oct 31 12:27:17.014617 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.012534    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9083c35a-9739-4f3e-8415-bad1e6433d40" volumeName="kubernetes.io/projected/9083c35a-9739-4f3e-8415-bad1e6433d40-bound-sa-token" seLinuxMountContext=""
Oct 31 12:27:17.014617 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.012546    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9083c35a-9739-4f3e-8415-bad1e6433d40" volumeName="kubernetes.io/projected/9083c35a-9739-4f3e-8415-bad1e6433d40-kube-api-access-5xwrq" seLinuxMountContext=""
Oct 31 12:27:17.014617 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.012558    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9083c35a-9739-4f3e-8415-bad1e6433d40" volumeName="kubernetes.io/secret/9083c35a-9739-4f3e-8415-bad1e6433d40-image-registry-private-configuration" seLinuxMountContext=""
Oct 31 12:27:17.014617 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.012573    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9557a3f0-ae0f-41f9-a9b5-15aeda410a38" volumeName="kubernetes.io/projected/9557a3f0-ae0f-41f9-a9b5-15aeda410a38-kube-api-access-jl4xd" seLinuxMountContext=""
Oct 31 12:27:17.014617 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.012584    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d43a4e16-3834-4d31-bed4-42fc0aaaaf17" volumeName="kubernetes.io/configmap/d43a4e16-3834-4d31-bed4-42fc0aaaaf17-env-overrides" seLinuxMountContext=""
Oct 31 12:27:17.014617 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.012592    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f032909d-bc39-4c9e-99b7-f3d800842119" volumeName="kubernetes.io/secret/f032909d-bc39-4c9e-99b7-f3d800842119-serving-cert" seLinuxMountContext=""
Oct 31 12:27:17.014617 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.012600    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9557a3f0-ae0f-41f9-a9b5-15aeda410a38" volumeName="kubernetes.io/empty-dir/9557a3f0-ae0f-41f9-a9b5-15aeda410a38-data-volume" seLinuxMountContext=""
Oct 31 12:27:17.014617 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.012605    5060 factory.go:153] Registering CRI-O factory
Oct 31 12:27:17.014617 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.012610    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a2b450be-ca79-4edf-a1ea-377531a49519" volumeName="kubernetes.io/secret/a2b450be-ca79-4edf-a1ea-377531a49519-original-pull-secret" seLinuxMountContext=""
Oct 31 12:27:17.014617 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.012618    5060 factory.go:223] Registration of the crio container factory successfully
Oct 31 12:27:17.014617 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.012621    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d43a4e16-3834-4d31-bed4-42fc0aaaaf17" volumeName="kubernetes.io/secret/d43a4e16-3834-4d31-bed4-42fc0aaaaf17-ovn-node-metrics-cert" seLinuxMountContext=""
Oct 31 12:27:17.014617 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.012634    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e64bd69d-946f-42ad-94ae-b87b7de3a581" volumeName="kubernetes.io/projected/e64bd69d-946f-42ad-94ae-b87b7de3a581-kube-api-access-gqk48" seLinuxMountContext=""
Oct 31 12:27:17.014617 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.012643    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6bbe13aa-df4b-40b0-8595-68fe1166a082" volumeName="kubernetes.io/projected/6bbe13aa-df4b-40b0-8595-68fe1166a082-kube-api-access-7j2rp" seLinuxMountContext=""
Oct 31 12:27:17.014617 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.012651    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5c8865f9-fdf6-49b9-8b4f-69dbc7787084" volumeName="kubernetes.io/configmap/5c8865f9-fdf6-49b9-8b4f-69dbc7787084-config-volume" seLinuxMountContext=""
Oct 31 12:27:17.014617 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.012658    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0db116b4-d971-4966-bb05-db28d077b6fe" volumeName="kubernetes.io/empty-dir/0db116b4-d971-4966-bb05-db28d077b6fe-ca-trust-extracted" seLinuxMountContext=""
Oct 31 12:27:17.014617 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.012665    5060 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 31 12:27:17.014617 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.012666    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b28ee2bb-0387-4d2d-8ca7-3fef91619c6b" volumeName="kubernetes.io/projected/b28ee2bb-0387-4d2d-8ca7-3fef91619c6b-kube-api-access-8mnlt" seLinuxMountContext=""
Oct 31 12:27:17.014617 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.012686    5060 factory.go:103] Registering Raw factory
Oct 31 12:27:17.014617 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.012703    5060 manager.go:1199] Started watching for new ooms in manager
Oct 31 12:27:17.014617 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.012684    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ddb04b2e-2724-44e9-948a-877a90d8a071" volumeName="kubernetes.io/configmap/ddb04b2e-2724-44e9-948a-877a90d8a071-service-ca-bundle" seLinuxMountContext=""
Oct 31 12:27:17.014617 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.012875    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f032909d-bc39-4c9e-99b7-f3d800842119" volumeName="kubernetes.io/configmap/f032909d-bc39-4c9e-99b7-f3d800842119-trusted-ca" seLinuxMountContext=""
Oct 31 12:27:17.014617 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.012897    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f032909d-bc39-4c9e-99b7-f3d800842119" volumeName="kubernetes.io/projected/f032909d-bc39-4c9e-99b7-f3d800842119-kube-api-access-9rblt" seLinuxMountContext=""
Oct 31 12:27:17.014617 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.012911    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="50ad7816-358d-461e-a9bd-fadce6246e8f" volumeName="kubernetes.io/configmap/50ad7816-358d-461e-a9bd-fadce6246e8f-nginx-conf" seLinuxMountContext=""
Oct 31 12:27:17.014617 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.012937    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee" volumeName="kubernetes.io/projected/51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee-kube-api-access-t5nx4" seLinuxMountContext=""
Oct 31 12:27:17.014617 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.012953    5060 reconstruct.go:97] "Volume reconstruction finished"
Oct 31 12:27:17.014617 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.012961    5060 reconciler.go:29] "Reconciler: start to sync state"
Oct 31 12:27:17.014617 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.013083    5060 manager.go:319] Starting recovery of all containers
Oct 31 12:27:17.023997 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.019723    5060 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 12:27:17.023997 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.020868    5060 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Oct 31 12:27:17.023997 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.023100    5060 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 12:27:17.030734 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.030711    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pcvvm_f032909d-bc39-4c9e-99b7-f3d800842119/console-operator/1.log"
Oct 31 12:27:17.031035 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.031018    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pcvvm_f032909d-bc39-4c9e-99b7-f3d800842119/console-operator/1.log"
Oct 31 12:27:17.038126 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.038091    5060 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 31 12:27:17.039730 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.039705    5060 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 31 12:27:17.039730 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.039728    5060 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 31 12:27:17.039875 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.039749    5060 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 31 12:27:17.039875 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:27:17.039794    5060 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Oct 31 12:27:17.043471 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.043452    5060 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 12:27:17.045345 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.045317    5060 generic.go:356] "Generic (PLEG): container finished" podID="928a0065cb02eec7c5308f7ecf1e7e6c" containerID="e8f994fa7a328558aca930a66289edc6895311b96cd49b2b7ce9af1967439f32" exitCode=0
Oct 31 12:27:17.066401 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.066372    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pcvvm_f032909d-bc39-4c9e-99b7-f3d800842119/console-operator/1.log"
Oct 31 12:27:17.066591 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.066408    5060 generic.go:356] "Generic (PLEG): container finished" podID="f032909d-bc39-4c9e-99b7-f3d800842119" containerID="c98998f609d234b42991ed336e4bc8d348b0169b157e725e00a2b4bc0c1a44a4" exitCode=255
Oct 31 12:27:17.082041 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.082012    5060 generic.go:356] "Generic (PLEG): container finished" podID="a1c3175b-e32e-420e-877f-71bfe64d9c6c" containerID="39c01b342d9309b09ba86218154e82594fafe0cebfa7ecc46f44bda8d2167341" exitCode=0
Oct 31 12:27:17.082041 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.082036    5060 generic.go:356] "Generic (PLEG): container finished" podID="a1c3175b-e32e-420e-877f-71bfe64d9c6c" containerID="44ccdd536d82848d2351206309ee00963dc164980658786d52f9f51997905e40" exitCode=0
Oct 31 12:27:17.082041 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.082044    5060 generic.go:356] "Generic (PLEG): container finished" podID="a1c3175b-e32e-420e-877f-71bfe64d9c6c" containerID="1a72cc05db4951f01ee9048390ecbb98a21b223630b33c9fcb888f790ed78755" exitCode=0
Oct 31 12:27:17.082041 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.082050    5060 generic.go:356] "Generic (PLEG): container finished" podID="a1c3175b-e32e-420e-877f-71bfe64d9c6c" containerID="e09b86dbee27e9abd088acc93d8bff8dcf71d5ec422aae1a04d82c64bdef0d3d" exitCode=0
Oct 31 12:27:17.082367 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.082055    5060 generic.go:356] "Generic (PLEG): container finished" podID="a1c3175b-e32e-420e-877f-71bfe64d9c6c" containerID="5277aadaa97b46ce60e04e87b5b25f8b8ffaef3c16a182a3798a6c0523fb2caf" exitCode=0
Oct 31 12:27:17.082367 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.082061    5060 generic.go:356] "Generic (PLEG): container finished" podID="a1c3175b-e32e-420e-877f-71bfe64d9c6c" containerID="bb15aebbbc18ac0274c0f0e477b19cf85a26e0215d4399f57bb1f8e8750991bd" exitCode=0
Oct 31 12:27:17.115210 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.115151    5060 manager.go:324] Recovery completed
Oct 31 12:27:17.135532 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.135502    5060 cpu_manager.go:226] "Starting" policy="none"
Oct 31 12:27:17.135532 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.135526    5060 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 31 12:27:17.135683 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.135550    5060 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 31 12:27:17.135683 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.135677    5060 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Oct 31 12:27:17.135778 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.135689    5060 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Oct 31 12:27:17.135778 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.135706    5060 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Oct 31 12:27:17.135778 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.135712    5060 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Oct 31 12:27:17.135778 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.135717    5060 policy_none.go:50] "Start"
Oct 31 12:27:17.135778 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.135726    5060 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 31 12:27:17.135778 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.135734    5060 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 31 12:27:17.140139 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:27:17.140118    5060 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet"
Oct 31 12:27:17.153240 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.153217    5060 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Oct 31 12:27:17.153240 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.153238    5060 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Oct 31 12:27:17.153240 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.153245    5060 policy_none.go:44] "Start"
Oct 31 12:27:17.157719 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.157687    5060 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 31 12:27:17.160462 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.160443    5060 manager.go:341] "Starting Device Plugin manager"
Oct 31 12:27:17.161033 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:27:17.160473    5060 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 31 12:27:17.161033 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.161034    5060 server.go:87] "Starting device plugin registration server"
Oct 31 12:27:17.161297 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.161286    5060 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 31 12:27:17.161363 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.161299    5060 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 31 12:27:17.161464 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.161407    5060 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 31 12:27:17.162518 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.161541    5060 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 31 12:27:17.162595 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.162543    5060 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 31 12:27:17.162595 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.162551    5060 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 31 12:27:17.162897 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.162791    5060 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-31T12:27:17.162518334Z","UUID":"a334c5fe-36e8-44ea-9f28-667a9d8a5808","Handler":null,"Name":"","Endpoint":""}
Oct 31 12:27:17.164460 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.164439    5060 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 31 12:27:17.164564 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.164480    5060 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 31 12:27:17.166261 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:27:17.166236    5060 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 31 12:27:17.261553 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.261511    5060 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 12:27:17.262843 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.262818    5060 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-190.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 12:27:17.262988 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.262856    5060 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-190.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 12:27:17.262988 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.262871    5060 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-190.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 12:27:17.262988 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.262924    5060 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-130-190.us-east-2.compute.internal"
Oct 31 12:27:17.276692 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.276658    5060 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-130-190.us-east-2.compute.internal"
Oct 31 12:27:17.276794 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.276749    5060 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-130-190.us-east-2.compute.internal"
Oct 31 12:27:17.340581 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.340529    5060 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-190.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-130-190.us-east-2.compute.internal"]
Oct 31 12:27:17.341119 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.341087    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-190.us-east-2.compute.internal" event={"ID":"928a0065cb02eec7c5308f7ecf1e7e6c","Type":"ContainerStarted","Data":"c2e7f76aa5d274e97fce9326472fe173770b9da4acf41adfb8ecba48d8232731"}
Oct 31 12:27:17.341180 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.341127    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-190.us-east-2.compute.internal" event={"ID":"928a0065cb02eec7c5308f7ecf1e7e6c","Type":"ContainerDied","Data":"e8f994fa7a328558aca930a66289edc6895311b96cd49b2b7ce9af1967439f32"}
Oct 31 12:27:17.341223 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.341213    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-190.us-east-2.compute.internal" event={"ID":"928a0065cb02eec7c5308f7ecf1e7e6c","Type":"ContainerStarted","Data":"b61159ca029793099370ff0779dda563c7d5600e24e1415f4ca6047a637d4ed7"}
Oct 31 12:27:17.341264 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.341254    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-190.us-east-2.compute.internal" event={"ID":"7fb532212935ad861267bbe741c1366b","Type":"ContainerStarted","Data":"dfda9c75e01d4df0bb0b1f645cc9fd65c59496e8697e211e22f1e8daccc12d03"}
Oct 31 12:27:17.341303 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.341268    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-190.us-east-2.compute.internal" event={"ID":"7fb532212935ad861267bbe741c1366b","Type":"ContainerStarted","Data":"b539c38438f2cfd785f2a2a6e023750667ea93e764840b11ad42f359c3687b08"}
Oct 31 12:27:17.341351 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.341334    5060 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-190.us-east-2.compute.internal"
Oct 31 12:27:17.341458 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.341447    5060 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-190.us-east-2.compute.internal"
Oct 31 12:27:17.356175 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.356148    5060 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 12:27:17.356294 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:27:17.356208    5060 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-130-190.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-190.us-east-2.compute.internal"
Oct 31 12:27:17.356741 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.356719    5060 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 12:27:17.356811 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:27:17.356765    5060 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-130-190.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-190.us-east-2.compute.internal"
Oct 31 12:27:17.423143 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.423051    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/928a0065cb02eec7c5308f7ecf1e7e6c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-190.us-east-2.compute.internal\" (UID: \"928a0065cb02eec7c5308f7ecf1e7e6c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-190.us-east-2.compute.internal"
Oct 31 12:27:17.423143 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.423092    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7fb532212935ad861267bbe741c1366b-config\") pod \"kube-apiserver-proxy-ip-10-0-130-190.us-east-2.compute.internal\" (UID: \"7fb532212935ad861267bbe741c1366b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-190.us-east-2.compute.internal"
Oct 31 12:27:17.423330 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.423173    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/928a0065cb02eec7c5308f7ecf1e7e6c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-190.us-east-2.compute.internal\" (UID: \"928a0065cb02eec7c5308f7ecf1e7e6c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-190.us-east-2.compute.internal"
Oct 31 12:27:17.523808 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.523769    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/928a0065cb02eec7c5308f7ecf1e7e6c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-190.us-east-2.compute.internal\" (UID: \"928a0065cb02eec7c5308f7ecf1e7e6c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-190.us-east-2.compute.internal"
Oct 31 12:27:17.523959 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.523811    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/928a0065cb02eec7c5308f7ecf1e7e6c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-190.us-east-2.compute.internal\" (UID: \"928a0065cb02eec7c5308f7ecf1e7e6c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-190.us-east-2.compute.internal"
Oct 31 12:27:17.523959 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.523840    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7fb532212935ad861267bbe741c1366b-config\") pod \"kube-apiserver-proxy-ip-10-0-130-190.us-east-2.compute.internal\" (UID: \"7fb532212935ad861267bbe741c1366b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-190.us-east-2.compute.internal"
Oct 31 12:27:17.523959 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.523881    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/928a0065cb02eec7c5308f7ecf1e7e6c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-190.us-east-2.compute.internal\" (UID: \"928a0065cb02eec7c5308f7ecf1e7e6c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-190.us-east-2.compute.internal"
Oct 31 12:27:17.523959 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.523902    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/928a0065cb02eec7c5308f7ecf1e7e6c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-190.us-east-2.compute.internal\" (UID: \"928a0065cb02eec7c5308f7ecf1e7e6c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-190.us-east-2.compute.internal"
Oct 31 12:27:17.523959 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.523882    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7fb532212935ad861267bbe741c1366b-config\") pod \"kube-apiserver-proxy-ip-10-0-130-190.us-east-2.compute.internal\" (UID: \"7fb532212935ad861267bbe741c1366b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-190.us-east-2.compute.internal"
Oct 31 12:27:17.990630 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:17.990597    5060 apiserver.go:52] "Watching apiserver"
Oct 31 12:27:18.006217 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.006189    5060 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 31 12:27:18.006927 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.006901    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/node-ca-wdkzp","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-hf9hr","openshift-ingress-canary/ingress-canary-rl55s","openshift-multus/multus-additional-cni-plugins-44zj2","openshift-cluster-node-tuning-operator/tuned-qkt2l","openshift-image-registry/image-registry-6dd647c7b7-9255m","openshift-service-ca-operator/service-ca-operator-544d749c8c-c789s","openshift-dns/dns-default-r6fvt","openshift-kube-storage-version-migrator/migrator-66df5d496b-m4fpw","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-190.us-east-2.compute.internal","openshift-network-console/networking-console-plugin-86c478bb88-2bfvp","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jmlvc","openshift-insights/insights-operator-7c4986d449-z879s","openshift-network-diagnostics/network-check-source-6f78d59f99-m9xxh","kube-system/kas-connection-checker-685895c7c9-rrc8k","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-tcnrw","openshift-insights/periodic-gathering-87ncb-cnk5g","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-vqsv9","openshift-multus/network-metrics-daemon-94kh9","openshift-kube-storage-version-migrator/migrator-66df5d496b-skkg4","openshift-insights/insights-runtime-extractor-p7phb","openshift-ovn-kubernetes/ovnkube-node-86758","kube-system/kube-apiserver-proxy-ip-10-0-130-190.us-east-2.compute.internal","openshift-console-operator/console-operator-7cd56b8dbc-pcvvm","openshift-network-diagnostics/network-check-target-tdrmt","openshift-service-ca/service-ca-5fffc97bdf-lmwkw","openshift-monitoring/cluster-monitoring-operator-884576c57-gt9rn","openshift-multus/multus-ccps4","openshift-image-registry/image-registry-6cfdbfcdc4-2rq84","openshift-network-operator/iptables-alerter-plbnz","kube-system/kas-connection-checker-685895c7c9-vvzpg","kube-system/konnectivity-agent-8jlvq","kube-system/global-pull-secret-syncer-2bmzr","kube-system/kas-connection-checker-685895c7c9-k64bz","openshift-ingress/router-default-659589798c-776bx","openshift-dns/node-resolver-v9ks4"]
Oct 31 12:27:18.008374 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.008349    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gt9rn"
Oct 31 12:27:18.010122 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.010104    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-tcnrw"
Oct 31 12:27:18.010513 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.010474    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-659589798c-776bx"
Oct 31 12:27:18.011276 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.011259    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6cfdbfcdc4-2rq84"
Oct 31 12:27:18.011651 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.011630    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-2bfvp"
Oct 31 12:27:18.012166 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.012146    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\""
Oct 31 12:27:18.012166 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.012146    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\""
Oct 31 12:27:18.012386 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.012369    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Oct 31 12:27:18.014076 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.014059    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Oct 31 12:27:18.014202 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.014187    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Oct 31 12:27:18.014375 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.014359    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-94kh9"
Oct 31 12:27:18.014424 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.014399    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Oct 31 12:27:18.015159 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.015140    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Oct 31 12:27:18.015394 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.015362    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Oct 31 12:27:18.015480 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.015423    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Oct 31 12:27:18.019190 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.019168    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Oct 31 12:27:18.019330 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.019272    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Oct 31 12:27:18.019402 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.019373    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Oct 31 12:27:18.019464 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.019452    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Oct 31 12:27:18.019560 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.019546    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Oct 31 12:27:18.019640 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.019629    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Oct 31 12:27:18.019753 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.019738    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 31 12:27:18.019906 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.019882    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 31 12:27:18.019986 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.019965    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Oct 31 12:27:18.020073 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.020059    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Oct 31 12:27:18.020266 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.020249    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 31 12:27:18.020347 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.020266    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 31 12:27:18.020414 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.020352    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Oct 31 12:27:18.020900 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.020884    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Oct 31 12:27:18.021040 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.021025    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Oct 31 12:27:18.021128 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.021116    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Oct 31 12:27:18.021216 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.021203    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\""
Oct 31 12:27:18.023569 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.023550    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6dd647c7b7-9255m"
Oct 31 12:27:18.024199 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.024185    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-r6fvt"
Oct 31 12:27:18.025475 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.025461    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-p7phb"
Oct 31 12:27:18.026637 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.026614    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ddb04b2e-2724-44e9-948a-877a90d8a071-tmp\") pod \"insights-operator-7c4986d449-z879s\" (UID: \"ddb04b2e-2724-44e9-948a-877a90d8a071\") " pod="openshift-insights/insights-operator-7c4986d449-z879s"
Oct 31 12:27:18.026706 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.026653    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ddb04b2e-2724-44e9-948a-877a90d8a071-serving-cert\") pod \"insights-operator-7c4986d449-z879s\" (UID: \"ddb04b2e-2724-44e9-948a-877a90d8a071\") " pod="openshift-insights/insights-operator-7c4986d449-z879s"
Oct 31 12:27:18.026706 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.026683    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvzh5\" (UniqueName: \"kubernetes.io/projected/613125d4-fbb4-4b43-9c47-24e039bd4697-kube-api-access-mvzh5\") pod \"kas-connection-checker-685895c7c9-k64bz\" (UID: \"613125d4-fbb4-4b43-9c47-24e039bd4697\") " pod="kube-system/kas-connection-checker-685895c7c9-k64bz"
Oct 31 12:27:18.026795 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.026725    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/febbc28e-bbcc-4961-bde8-09944d9ca680-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-vqsv9\" (UID: \"febbc28e-bbcc-4961-bde8-09944d9ca680\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-vqsv9"
Oct 31 12:27:18.026795 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.026740    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ddb04b2e-2724-44e9-948a-877a90d8a071-tmp\") pod \"insights-operator-7c4986d449-z879s\" (UID: \"ddb04b2e-2724-44e9-948a-877a90d8a071\") " pod="openshift-insights/insights-operator-7c4986d449-z879s"
Oct 31 12:27:18.026795 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.026769    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/dda0c26f-8225-4239-bc27-866b66cbb919-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-gt9rn\" (UID: \"dda0c26f-8225-4239-bc27-866b66cbb919\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gt9rn"
Oct 31 12:27:18.026928 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.026798    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqmhd\" (UniqueName: \"kubernetes.io/projected/07714f20-a70e-49e8-8e51-d03a8e00005a-kube-api-access-hqmhd\") pod \"kas-connection-checker-685895c7c9-vvzpg\" (UID: \"07714f20-a70e-49e8-8e51-d03a8e00005a\") " pod="kube-system/kas-connection-checker-685895c7c9-vvzpg"
Oct 31 12:27:18.026928 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.026821    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mnlt\" (UniqueName: \"kubernetes.io/projected/b28ee2bb-0387-4d2d-8ca7-3fef91619c6b-kube-api-access-8mnlt\") pod \"volume-data-source-validator-58c9f5f46d-hf9hr\" (UID: \"b28ee2bb-0387-4d2d-8ca7-3fef91619c6b\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-hf9hr"
Oct 31 12:27:18.026928 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.026844    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ddb04b2e-2724-44e9-948a-877a90d8a071-service-ca-bundle\") pod \"insights-operator-7c4986d449-z879s\" (UID: \"ddb04b2e-2724-44e9-948a-877a90d8a071\") " pod="openshift-insights/insights-operator-7c4986d449-z879s"
Oct 31 12:27:18.026928 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.026869    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zk92l\" (UniqueName: \"kubernetes.io/projected/ff424288-19ca-4db0-9f16-fed9fb58404a-kube-api-access-zk92l\") pod \"kas-connection-checker-685895c7c9-rrc8k\" (UID: \"ff424288-19ca-4db0-9f16-fed9fb58404a\") " pod="kube-system/kas-connection-checker-685895c7c9-rrc8k"
Oct 31 12:27:18.026928 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.026894    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/dda0c26f-8225-4239-bc27-866b66cbb919-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-gt9rn\" (UID: \"dda0c26f-8225-4239-bc27-866b66cbb919\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gt9rn"
Oct 31 12:27:18.026928 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.026920    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsrnd\" (UniqueName: \"kubernetes.io/projected/febbc28e-bbcc-4961-bde8-09944d9ca680-kube-api-access-jsrnd\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-vqsv9\" (UID: \"febbc28e-bbcc-4961-bde8-09944d9ca680\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-vqsv9"
Oct 31 12:27:18.027259 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.026944    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/ddb04b2e-2724-44e9-948a-877a90d8a071-snapshots\") pod \"insights-operator-7c4986d449-z879s\" (UID: \"ddb04b2e-2724-44e9-948a-877a90d8a071\") " pod="openshift-insights/insights-operator-7c4986d449-z879s"
Oct 31 12:27:18.027259 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.026968    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ddb04b2e-2724-44e9-948a-877a90d8a071-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-z879s\" (UID: \"ddb04b2e-2724-44e9-948a-877a90d8a071\") " pod="openshift-insights/insights-operator-7c4986d449-z879s"
Oct 31 12:27:18.027259 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.026992    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wq6d\" (UniqueName: \"kubernetes.io/projected/ddb04b2e-2724-44e9-948a-877a90d8a071-kube-api-access-5wq6d\") pod \"insights-operator-7c4986d449-z879s\" (UID: \"ddb04b2e-2724-44e9-948a-877a90d8a071\") " pod="openshift-insights/insights-operator-7c4986d449-z879s"
Oct 31 12:27:18.027259 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.027018    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/febbc28e-bbcc-4961-bde8-09944d9ca680-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-vqsv9\" (UID: \"febbc28e-bbcc-4961-bde8-09944d9ca680\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-vqsv9"
Oct 31 12:27:18.027259 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.026942    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/febbc28e-bbcc-4961-bde8-09944d9ca680-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-vqsv9\" (UID: \"febbc28e-bbcc-4961-bde8-09944d9ca680\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-vqsv9"
Oct 31 12:27:18.027259 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.027058    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jw4wt\" (UniqueName: \"kubernetes.io/projected/dda0c26f-8225-4239-bc27-866b66cbb919-kube-api-access-jw4wt\") pod \"cluster-monitoring-operator-884576c57-gt9rn\" (UID: \"dda0c26f-8225-4239-bc27-866b66cbb919\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gt9rn"
Oct 31 12:27:18.027259 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.027254    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ddb04b2e-2724-44e9-948a-877a90d8a071-serving-cert\") pod \"insights-operator-7c4986d449-z879s\" (UID: \"ddb04b2e-2724-44e9-948a-877a90d8a071\") " pod="openshift-insights/insights-operator-7c4986d449-z879s"
Oct 31 12:27:18.027585 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.027266    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Oct 31 12:27:18.027585 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.027425    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/ddb04b2e-2724-44e9-948a-877a90d8a071-snapshots\") pod \"insights-operator-7c4986d449-z879s\" (UID: \"ddb04b2e-2724-44e9-948a-877a90d8a071\") " pod="openshift-insights/insights-operator-7c4986d449-z879s"
Oct 31 12:27:18.027585 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.027570    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/dda0c26f-8225-4239-bc27-866b66cbb919-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-gt9rn\" (UID: \"dda0c26f-8225-4239-bc27-866b66cbb919\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gt9rn"
Oct 31 12:27:18.027713 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.027687    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/febbc28e-bbcc-4961-bde8-09944d9ca680-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-vqsv9\" (UID: \"febbc28e-bbcc-4961-bde8-09944d9ca680\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-vqsv9"
Oct 31 12:27:18.027781 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.027762    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ddb04b2e-2724-44e9-948a-877a90d8a071-service-ca-bundle\") pod \"insights-operator-7c4986d449-z879s\" (UID: \"ddb04b2e-2724-44e9-948a-877a90d8a071\") " pod="openshift-insights/insights-operator-7c4986d449-z879s"
Oct 31 12:27:18.032535 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.032516    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Oct 31 12:27:18.032998 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.032984    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Oct 31 12:27:18.037135 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.037101    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvzh5\" (UniqueName: \"kubernetes.io/projected/613125d4-fbb4-4b43-9c47-24e039bd4697-kube-api-access-mvzh5\") pod \"kas-connection-checker-685895c7c9-k64bz\" (UID: \"613125d4-fbb4-4b43-9c47-24e039bd4697\") " pod="kube-system/kas-connection-checker-685895c7c9-k64bz"
Oct 31 12:27:18.037135 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.037113    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqmhd\" (UniqueName: \"kubernetes.io/projected/07714f20-a70e-49e8-8e51-d03a8e00005a-kube-api-access-hqmhd\") pod \"kas-connection-checker-685895c7c9-vvzpg\" (UID: \"07714f20-a70e-49e8-8e51-d03a8e00005a\") " pod="kube-system/kas-connection-checker-685895c7c9-vvzpg"
Oct 31 12:27:18.037135 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.037123    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsrnd\" (UniqueName: \"kubernetes.io/projected/febbc28e-bbcc-4961-bde8-09944d9ca680-kube-api-access-jsrnd\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-vqsv9\" (UID: \"febbc28e-bbcc-4961-bde8-09944d9ca680\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-vqsv9"
Oct 31 12:27:18.037135 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.037100    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zk92l\" (UniqueName: \"kubernetes.io/projected/ff424288-19ca-4db0-9f16-fed9fb58404a-kube-api-access-zk92l\") pod \"kas-connection-checker-685895c7c9-rrc8k\" (UID: \"ff424288-19ca-4db0-9f16-fed9fb58404a\") " pod="kube-system/kas-connection-checker-685895c7c9-rrc8k"
Oct 31 12:27:18.037399 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.037340    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mnlt\" (UniqueName: \"kubernetes.io/projected/b28ee2bb-0387-4d2d-8ca7-3fef91619c6b-kube-api-access-8mnlt\") pod \"volume-data-source-validator-58c9f5f46d-hf9hr\" (UID: \"b28ee2bb-0387-4d2d-8ca7-3fef91619c6b\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-hf9hr"
Oct 31 12:27:18.037615 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.037596    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jw4wt\" (UniqueName: \"kubernetes.io/projected/dda0c26f-8225-4239-bc27-866b66cbb919-kube-api-access-jw4wt\") pod \"cluster-monitoring-operator-884576c57-gt9rn\" (UID: \"dda0c26f-8225-4239-bc27-866b66cbb919\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gt9rn"
Oct 31 12:27:18.037842 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.037803    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wq6d\" (UniqueName: \"kubernetes.io/projected/ddb04b2e-2724-44e9-948a-877a90d8a071-kube-api-access-5wq6d\") pod \"insights-operator-7c4986d449-z879s\" (UID: \"ddb04b2e-2724-44e9-948a-877a90d8a071\") " pod="openshift-insights/insights-operator-7c4986d449-z879s"
Oct 31 12:27:18.037877 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.037858    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ddb04b2e-2724-44e9-948a-877a90d8a071-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-z879s\" (UID: \"ddb04b2e-2724-44e9-948a-877a90d8a071\") " pod="openshift-insights/insights-operator-7c4986d449-z879s"
Oct 31 12:27:18.099649 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.099621    5060 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-190.us-east-2.compute.internal"
Oct 31 12:27:18.099787 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.099701    5060 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-190.us-east-2.compute.internal"
Oct 31 12:27:18.112263 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.112230    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 31 12:27:18.112263 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.112236    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 31 12:27:18.112541 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.112324    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 31 12:27:18.112541 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.112364    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Oct 31 12:27:18.112541 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.112409    5060 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 12:27:18.112541 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:27:18.112458    5060 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-130-190.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-190.us-east-2.compute.internal"
Oct 31 12:27:18.112541 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.112460    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Oct 31 12:27:18.112541 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.112532    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Oct 31 12:27:18.112833 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.112589    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 31 12:27:18.112833 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.112641    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 31 12:27:18.112833 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.112650    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 31 12:27:18.112833 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.112655    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 31 12:27:18.112833 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.112650    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 31 12:27:18.112833 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.112692    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 31 12:27:18.112833 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.112713    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 31 12:27:18.112833 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.112715    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 31 12:27:18.113141 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.113108    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 31 12:27:18.113141 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.113121    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 31 12:27:18.113141 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.113123    5060 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 31 12:27:18.113302 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.113193    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 31 12:27:18.113302 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.113195    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 31 12:27:18.113302 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.113199    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 31 12:27:18.113302 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.113216    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 31 12:27:18.113302 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.113212    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 31 12:27:18.113302 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.113230    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 31 12:27:18.113483 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.113373    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 31 12:27:18.113483 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.113373    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 31 12:27:18.113483 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.113427    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Oct 31 12:27:18.113483 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.113464    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Oct 31 12:27:18.113670 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.113517    5060 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 12:27:18.113670 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:27:18.113556    5060 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-130-190.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-190.us-east-2.compute.internal"
Oct 31 12:27:18.113670 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.113563    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 31 12:27:18.113670 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.113589    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 31 12:27:18.113670 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.113580    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Oct 31 12:27:18.113670 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.113596    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 31 12:27:18.114342 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.114319    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 31 12:27:18.114342 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.114336    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 31 12:27:18.114454 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.114351    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 31 12:27:18.114454 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.114451    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 31 12:27:18.114545 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.114470    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Oct 31 12:27:18.114545 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.114486    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 31 12:27:18.114545 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.114506    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 31 12:27:18.114545 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.114502    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Oct 31 12:27:18.114826 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.114815    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 31 12:27:18.116027 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.116015    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 31 12:27:18.119054 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.119038    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Oct 31 12:27:18.119921 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.119906    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-vq8lr\""
Oct 31 12:27:18.120534 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.120456    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 31 12:27:18.120534 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.120461    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 31 12:27:18.120534 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.120458    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Oct 31 12:27:18.120534 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.120524    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-s7td2\""
Oct 31 12:27:18.120713 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.120537    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-ng46x\""
Oct 31 12:27:18.120713 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.120457    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-df646\""
Oct 31 12:27:18.120713 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.120459    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-9d2j7\""
Oct 31 12:27:18.120713 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.120472    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Oct 31 12:27:18.121403 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.121362    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 31 12:27:18.121515 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.121424    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-c6kzr\""
Oct 31 12:27:18.121515 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.121505    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 31 12:27:18.121668 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.121644    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Oct 31 12:27:18.121668 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.121656    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Oct 31 12:27:18.121668 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.121651    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Oct 31 12:27:18.122737 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.122720    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Oct 31 12:27:18.122808 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.122745    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 31 12:27:18.122808 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.122768    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-hzb2l\""
Oct 31 12:27:18.122808 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.122775    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 31 12:27:18.122808 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.122775    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-ght96\""
Oct 31 12:27:18.122808 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.122785    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-kldqp\""
Oct 31 12:27:18.123098 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.122840    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-t2vx8\""
Oct 31 12:27:18.123098 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.122870    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-nzzrg\""
Oct 31 12:27:18.123098 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.122872    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-mwkk2\""
Oct 31 12:27:18.123098 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.122911    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 31 12:27:18.123098 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.122918    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 31 12:27:18.123098 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.122926    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-b9z2w\""
Oct 31 12:27:18.123098 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.122928    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Oct 31 12:27:18.123098 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.122934    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-8ljk6\""
Oct 31 12:27:18.123098 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.122966    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-m5kv7\""
Oct 31 12:27:18.123098 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.122973    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-bbb6f\""
Oct 31 12:27:18.123098 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.122967    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Oct 31 12:27:18.123098 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.122982    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-ghjl2\""
Oct 31 12:27:18.123098 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.122966    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-x5rvd\""
Oct 31 12:27:18.123098 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.122992    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-lffk4\""
Oct 31 12:27:18.123098 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.123005    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-vcqch\""
Oct 31 12:27:18.127517 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.127486    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f032909d-bc39-4c9e-99b7-f3d800842119-config\") pod \"console-operator-7cd56b8dbc-pcvvm\" (UID: \"f032909d-bc39-4c9e-99b7-f3d800842119\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-pcvvm"
Oct 31 12:27:18.127664 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.127524    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f032909d-bc39-4c9e-99b7-f3d800842119-trusted-ca\") pod \"console-operator-7cd56b8dbc-pcvvm\" (UID: \"f032909d-bc39-4c9e-99b7-f3d800842119\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-pcvvm"
Oct 31 12:27:18.127664 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.127540    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee-etc-tuned\") pod \"tuned-qkt2l\" (UID: \"51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee\") " pod="openshift-cluster-node-tuning-operator/tuned-qkt2l"
Oct 31 12:27:18.127664 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.127557    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nx5hk\" (UniqueName: \"kubernetes.io/projected/6c661ce8-9f7b-4509-bd17-2380bcd1c425-kube-api-access-nx5hk\") pod \"node-ca-wdkzp\" (UID: \"6c661ce8-9f7b-4509-bd17-2380bcd1c425\") " pod="openshift-image-registry/node-ca-wdkzp"
Oct 31 12:27:18.127664 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.127572    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/891b3c47-ff7c-4a7a-a09b-411361a843f7-cert\") pod \"ingress-canary-rl55s\" (UID: \"891b3c47-ff7c-4a7a-a09b-411361a843f7\") " pod="openshift-ingress-canary/ingress-canary-rl55s"
Oct 31 12:27:18.127894 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.127658    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee-etc-tuned\") pod \"tuned-qkt2l\" (UID: \"51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee\") " pod="openshift-cluster-node-tuning-operator/tuned-qkt2l"
Oct 31 12:27:18.127894 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.127686    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d43a4e16-3834-4d31-bed4-42fc0aaaaf17-run-systemd\") pod \"ovnkube-node-86758\" (UID: \"d43a4e16-3834-4d31-bed4-42fc0aaaaf17\") " pod="openshift-ovn-kubernetes/ovnkube-node-86758"
Oct 31 12:27:18.127894 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.127724    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5c8865f9-fdf6-49b9-8b4f-69dbc7787084-config-volume\") pod \"dns-default-r6fvt\" (UID: \"5c8865f9-fdf6-49b9-8b4f-69dbc7787084\") " pod="openshift-dns/dns-default-r6fvt"
Oct 31 12:27:18.127894 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.127752    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee-var-lib-kubelet\") pod \"tuned-qkt2l\" (UID: \"51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee\") " pod="openshift-cluster-node-tuning-operator/tuned-qkt2l"
Oct 31 12:27:18.127894 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.127762    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f032909d-bc39-4c9e-99b7-f3d800842119-config\") pod \"console-operator-7cd56b8dbc-pcvvm\" (UID: \"f032909d-bc39-4c9e-99b7-f3d800842119\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-pcvvm"
Oct 31 12:27:18.127894 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.127788    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/891b3c47-ff7c-4a7a-a09b-411361a843f7-cert\") pod \"ingress-canary-rl55s\" (UID: \"891b3c47-ff7c-4a7a-a09b-411361a843f7\") " pod="openshift-ingress-canary/ingress-canary-rl55s"
Oct 31 12:27:18.127894 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.127801    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bs5p9\" (UniqueName: \"kubernetes.io/projected/dbbd2d70-8911-4aa5-96cf-3a16dcdf246d-kube-api-access-bs5p9\") pod \"cluster-samples-operator-6ddb6c8b94-tcnrw\" (UID: \"dbbd2d70-8911-4aa5-96cf-3a16dcdf246d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-tcnrw"
Oct 31 12:27:18.127894 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.127863    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d43a4e16-3834-4d31-bed4-42fc0aaaaf17-run-ovn\") pod \"ovnkube-node-86758\" (UID: \"d43a4e16-3834-4d31-bed4-42fc0aaaaf17\") " pod="openshift-ovn-kubernetes/ovnkube-node-86758"
Oct 31 12:27:18.127894 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.127873    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f032909d-bc39-4c9e-99b7-f3d800842119-trusted-ca\") pod \"console-operator-7cd56b8dbc-pcvvm\" (UID: \"f032909d-bc39-4c9e-99b7-f3d800842119\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-pcvvm"
Oct 31 12:27:18.128260 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.127916    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/556dda7b-d8e6-4558-b5f3-994b01d6a866-stats-auth\") pod \"router-default-659589798c-776bx\" (UID: \"556dda7b-d8e6-4558-b5f3-994b01d6a866\") " pod="openshift-ingress/router-default-659589798c-776bx"
Oct 31 12:27:18.128260 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.127938    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6bbe13aa-df4b-40b0-8595-68fe1166a082-hosts-file\") pod \"node-resolver-v9ks4\" (UID: \"6bbe13aa-df4b-40b0-8595-68fe1166a082\") " pod="openshift-dns/node-resolver-v9ks4"
Oct 31 12:27:18.128260 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.127943    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5c8865f9-fdf6-49b9-8b4f-69dbc7787084-config-volume\") pod \"dns-default-r6fvt\" (UID: \"5c8865f9-fdf6-49b9-8b4f-69dbc7787084\") " pod="openshift-dns/dns-default-r6fvt"
Oct 31 12:27:18.128260 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.127982    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6bbe13aa-df4b-40b0-8595-68fe1166a082-tmp-dir\") pod \"node-resolver-v9ks4\" (UID: \"6bbe13aa-df4b-40b0-8595-68fe1166a082\") " pod="openshift-dns/node-resolver-v9ks4"
Oct 31 12:27:18.128260 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.128062    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6bbe13aa-df4b-40b0-8595-68fe1166a082-tmp-dir\") pod \"node-resolver-v9ks4\" (UID: \"6bbe13aa-df4b-40b0-8595-68fe1166a082\") " pod="openshift-dns/node-resolver-v9ks4"
Oct 31 12:27:18.128260 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.128093    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rblt\" (UniqueName: \"kubernetes.io/projected/f032909d-bc39-4c9e-99b7-f3d800842119-kube-api-access-9rblt\") pod \"console-operator-7cd56b8dbc-pcvvm\" (UID: \"f032909d-bc39-4c9e-99b7-f3d800842119\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-pcvvm"
Oct 31 12:27:18.128260 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.128178    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/556dda7b-d8e6-4558-b5f3-994b01d6a866-stats-auth\") pod \"router-default-659589798c-776bx\" (UID: \"556dda7b-d8e6-4558-b5f3-994b01d6a866\") " pod="openshift-ingress/router-default-659589798c-776bx"
Oct 31 12:27:18.128260 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.128182    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e64bd69d-946f-42ad-94ae-b87b7de3a581-metrics-certs\") pod \"network-metrics-daemon-94kh9\" (UID: \"e64bd69d-946f-42ad-94ae-b87b7de3a581\") " pod="openshift-multus/network-metrics-daemon-94kh9"
Oct 31 12:27:18.128260 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.128217    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d43a4e16-3834-4d31-bed4-42fc0aaaaf17-node-log\") pod \"ovnkube-node-86758\" (UID: \"d43a4e16-3834-4d31-bed4-42fc0aaaaf17\") " pod="openshift-ovn-kubernetes/ovnkube-node-86758"
Oct 31 12:27:18.128260 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.128238    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a1e578b0-2e77-4f52-8a44-379b6059c3d9-sys-fs\") pod \"aws-ebs-csi-driver-node-jmlvc\" (UID: \"a1e578b0-2e77-4f52-8a44-379b6059c3d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jmlvc"
Oct 31 12:27:18.128260 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.128259    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d43a4e16-3834-4d31-bed4-42fc0aaaaf17-host-run-netns\") pod \"ovnkube-node-86758\" (UID: \"d43a4e16-3834-4d31-bed4-42fc0aaaaf17\") " pod="openshift-ovn-kubernetes/ovnkube-node-86758"
Oct 31 12:27:18.128675 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.128290    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a1c3175b-e32e-420e-877f-71bfe64d9c6c-cnibin\") pod \"multus-additional-cni-plugins-44zj2\" (UID: \"a1c3175b-e32e-420e-877f-71bfe64d9c6c\") " pod="openshift-multus/multus-additional-cni-plugins-44zj2"
Oct 31 12:27:18.128675 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.128342    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6b3e77ff-4445-42fa-a7c3-f35d442c276d-cnibin\") pod \"multus-ccps4\" (UID: \"6b3e77ff-4445-42fa-a7c3-f35d442c276d\") " pod="openshift-multus/multus-ccps4"
Oct 31 12:27:18.128675 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.128385    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d43a4e16-3834-4d31-bed4-42fc0aaaaf17-host-cni-bin\") pod \"ovnkube-node-86758\" (UID: \"d43a4e16-3834-4d31-bed4-42fc0aaaaf17\") " pod="openshift-ovn-kubernetes/ovnkube-node-86758"
Oct 31 12:27:18.128675 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.128413    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a1e578b0-2e77-4f52-8a44-379b6059c3d9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jmlvc\" (UID: \"a1e578b0-2e77-4f52-8a44-379b6059c3d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jmlvc"
Oct 31 12:27:18.128675 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.128451    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvqpw\" (UniqueName: \"kubernetes.io/projected/25691fa3-7017-429a-b94c-c2dde4d29e55-kube-api-access-gvqpw\") pod \"network-check-source-6f78d59f99-m9xxh\" (UID: \"25691fa3-7017-429a-b94c-c2dde4d29e55\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-m9xxh"
Oct 31 12:27:18.128675 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.128545    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0db116b4-d971-4966-bb05-db28d077b6fe-ca-trust-extracted\") pod \"image-registry-6dd647c7b7-9255m\" (UID: \"0db116b4-d971-4966-bb05-db28d077b6fe\") " pod="openshift-image-registry/image-registry-6dd647c7b7-9255m"
Oct 31 12:27:18.128675 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.128590    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee-etc-systemd\") pod \"tuned-qkt2l\" (UID: \"51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee\") " pod="openshift-cluster-node-tuning-operator/tuned-qkt2l"
Oct 31 12:27:18.128675 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.128620    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzcgz\" (UniqueName: \"kubernetes.io/projected/d43a4e16-3834-4d31-bed4-42fc0aaaaf17-kube-api-access-hzcgz\") pod \"ovnkube-node-86758\" (UID: \"d43a4e16-3834-4d31-bed4-42fc0aaaaf17\") " pod="openshift-ovn-kubernetes/ovnkube-node-86758"
Oct 31 12:27:18.128675 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.128664    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0db116b4-d971-4966-bb05-db28d077b6fe-ca-trust-extracted\") pod \"image-registry-6dd647c7b7-9255m\" (UID: \"0db116b4-d971-4966-bb05-db28d077b6fe\") " pod="openshift-image-registry/image-registry-6dd647c7b7-9255m"
Oct 31 12:27:18.128948 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.128680    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6b3e77ff-4445-42fa-a7c3-f35d442c276d-multus-daemon-config\") pod \"multus-ccps4\" (UID: \"6b3e77ff-4445-42fa-a7c3-f35d442c276d\") " pod="openshift-multus/multus-ccps4"
Oct 31 12:27:18.128948 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.128755    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6c661ce8-9f7b-4509-bd17-2380bcd1c425-host\") pod \"node-ca-wdkzp\" (UID: \"6c661ce8-9f7b-4509-bd17-2380bcd1c425\") " pod="openshift-image-registry/node-ca-wdkzp"
Oct 31 12:27:18.128948 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.128782    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tf6n\" (UniqueName: \"kubernetes.io/projected/7a402472-ca33-44ae-b598-96fc8e7a5eb2-kube-api-access-2tf6n\") pod \"network-check-target-tdrmt\" (UID: \"7a402472-ca33-44ae-b598-96fc8e7a5eb2\") " pod="openshift-network-diagnostics/network-check-target-tdrmt"
Oct 31 12:27:18.128948 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.128900    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d43a4e16-3834-4d31-bed4-42fc0aaaaf17-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-86758\" (UID: \"d43a4e16-3834-4d31-bed4-42fc0aaaaf17\") " pod="openshift-ovn-kubernetes/ovnkube-node-86758"
Oct 31 12:27:18.129073 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.129001    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6b3e77ff-4445-42fa-a7c3-f35d442c276d-system-cni-dir\") pod \"multus-ccps4\" (UID: \"6b3e77ff-4445-42fa-a7c3-f35d442c276d\") " pod="openshift-multus/multus-ccps4"
Oct 31 12:27:18.129073 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.129028    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/556dda7b-d8e6-4558-b5f3-994b01d6a866-metrics-certs\") pod \"router-default-659589798c-776bx\" (UID: \"556dda7b-d8e6-4558-b5f3-994b01d6a866\") " pod="openshift-ingress/router-default-659589798c-776bx"
Oct 31 12:27:18.129073 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.129061    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6b3e77ff-4445-42fa-a7c3-f35d442c276d-multus-daemon-config\") pod \"multus-ccps4\" (UID: \"6b3e77ff-4445-42fa-a7c3-f35d442c276d\") " pod="openshift-multus/multus-ccps4"
Oct 31 12:27:18.129186 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.129077    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5nx4\" (UniqueName: \"kubernetes.io/projected/51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee-kube-api-access-t5nx4\") pod \"tuned-qkt2l\" (UID: \"51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee\") " pod="openshift-cluster-node-tuning-operator/tuned-qkt2l"
Oct 31 12:27:18.129186 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.129116    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9557a3f0-ae0f-41f9-a9b5-15aeda410a38-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p7phb\" (UID: \"9557a3f0-ae0f-41f9-a9b5-15aeda410a38\") " pod="openshift-insights/insights-runtime-extractor-p7phb"
Oct 31 12:27:18.129259 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.129217    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/b9ea9813-55b9-454b-8f10-985533305f85-signing-key\") pod \"service-ca-5fffc97bdf-lmwkw\" (UID: \"b9ea9813-55b9-454b-8f10-985533305f85\") " pod="openshift-service-ca/service-ca-5fffc97bdf-lmwkw"
Oct 31 12:27:18.129259 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.129244    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6c661ce8-9f7b-4509-bd17-2380bcd1c425-serviceca\") pod \"node-ca-wdkzp\" (UID: \"6c661ce8-9f7b-4509-bd17-2380bcd1c425\") " pod="openshift-image-registry/node-ca-wdkzp"
Oct 31 12:27:18.129339 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.129284    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d43a4e16-3834-4d31-bed4-42fc0aaaaf17-systemd-units\") pod \"ovnkube-node-86758\" (UID: \"d43a4e16-3834-4d31-bed4-42fc0aaaaf17\") " pod="openshift-ovn-kubernetes/ovnkube-node-86758"
Oct 31 12:27:18.129339 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.129304    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d43a4e16-3834-4d31-bed4-42fc0aaaaf17-host-cni-netd\") pod \"ovnkube-node-86758\" (UID: \"d43a4e16-3834-4d31-bed4-42fc0aaaaf17\") " pod="openshift-ovn-kubernetes/ovnkube-node-86758"
Oct 31 12:27:18.129435 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.129356    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0db116b4-d971-4966-bb05-db28d077b6fe-registry-certificates\") pod \"image-registry-6dd647c7b7-9255m\" (UID: \"0db116b4-d971-4966-bb05-db28d077b6fe\") " pod="openshift-image-registry/image-registry-6dd647c7b7-9255m"
Oct 31 12:27:18.129435 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.129407    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d43a4e16-3834-4d31-bed4-42fc0aaaaf17-host-slash\") pod \"ovnkube-node-86758\" (UID: \"d43a4e16-3834-4d31-bed4-42fc0aaaaf17\") " pod="openshift-ovn-kubernetes/ovnkube-node-86758"
Oct 31 12:27:18.129536 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.129506    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/b9ea9813-55b9-454b-8f10-985533305f85-signing-key\") pod \"service-ca-5fffc97bdf-lmwkw\" (UID: \"b9ea9813-55b9-454b-8f10-985533305f85\") " pod="openshift-service-ca/service-ca-5fffc97bdf-lmwkw"
Oct 31 12:27:18.129536 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.129518    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6c661ce8-9f7b-4509-bd17-2380bcd1c425-serviceca\") pod \"node-ca-wdkzp\" (UID: \"6c661ce8-9f7b-4509-bd17-2380bcd1c425\") " pod="openshift-image-registry/node-ca-wdkzp"
Oct 31 12:27:18.129600 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.129507    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d43a4e16-3834-4d31-bed4-42fc0aaaaf17-ovnkube-script-lib\") pod \"ovnkube-node-86758\" (UID: \"d43a4e16-3834-4d31-bed4-42fc0aaaaf17\") " pod="openshift-ovn-kubernetes/ovnkube-node-86758"
Oct 31 12:27:18.129600 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.129563    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0db116b4-d971-4966-bb05-db28d077b6fe-registry-certificates\") pod \"image-registry-6dd647c7b7-9255m\" (UID: \"0db116b4-d971-4966-bb05-db28d077b6fe\") " pod="openshift-image-registry/image-registry-6dd647c7b7-9255m"
Oct 31 12:27:18.129665 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.129594    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a2b450be-ca79-4edf-a1ea-377531a49519-kubelet-config\") pod \"global-pull-secret-syncer-2bmzr\" (UID: \"a2b450be-ca79-4edf-a1ea-377531a49519\") " pod="kube-system/global-pull-secret-syncer-2bmzr"
Oct 31 12:27:18.129665 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.129624    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a2b450be-ca79-4edf-a1ea-377531a49519-dbus\") pod \"global-pull-secret-syncer-2bmzr\" (UID: \"a2b450be-ca79-4edf-a1ea-377531a49519\") " pod="kube-system/global-pull-secret-syncer-2bmzr"
Oct 31 12:27:18.129665 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.129650    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6b3e77ff-4445-42fa-a7c3-f35d442c276d-host-var-lib-kubelet\") pod \"multus-ccps4\" (UID: \"6b3e77ff-4445-42fa-a7c3-f35d442c276d\") " pod="openshift-multus/multus-ccps4"
Oct 31 12:27:18.129806 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.129690    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d43a4e16-3834-4d31-bed4-42fc0aaaaf17-ovnkube-script-lib\") pod \"ovnkube-node-86758\" (UID: \"d43a4e16-3834-4d31-bed4-42fc0aaaaf17\") " pod="openshift-ovn-kubernetes/ovnkube-node-86758"
Oct 31 12:27:18.129806 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.129690    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b5014b57-9b83-4fc9-b0ed-55793b2b3a98-konnectivity-ca\") pod \"konnectivity-agent-8jlvq\" (UID: \"b5014b57-9b83-4fc9-b0ed-55793b2b3a98\") " pod="kube-system/konnectivity-agent-8jlvq"
Oct 31 12:27:18.129806 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.129754    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a2b450be-ca79-4edf-a1ea-377531a49519-original-pull-secret\") pod \"global-pull-secret-syncer-2bmzr\" (UID: \"a2b450be-ca79-4edf-a1ea-377531a49519\") " pod="kube-system/global-pull-secret-syncer-2bmzr"
Oct 31 12:27:18.129948 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.129831    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0db116b4-d971-4966-bb05-db28d077b6fe-installation-pull-secrets\") pod \"image-registry-6dd647c7b7-9255m\" (UID: \"0db116b4-d971-4966-bb05-db28d077b6fe\") " pod="openshift-image-registry/image-registry-6dd647c7b7-9255m"
Oct 31 12:27:18.129948 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.129881    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b5014b57-9b83-4fc9-b0ed-55793b2b3a98-konnectivity-ca\") pod \"konnectivity-agent-8jlvq\" (UID: \"b5014b57-9b83-4fc9-b0ed-55793b2b3a98\") " pod="kube-system/konnectivity-agent-8jlvq"
Oct 31 12:27:18.129948 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.129912    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9083c35a-9739-4f3e-8415-bad1e6433d40-image-registry-private-configuration\") pod \"image-registry-6cfdbfcdc4-2rq84\" (UID: \"9083c35a-9739-4f3e-8415-bad1e6433d40\") " pod="openshift-image-registry/image-registry-6cfdbfcdc4-2rq84"
Oct 31 12:27:18.130103 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.129993    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d43a4e16-3834-4d31-bed4-42fc0aaaaf17-env-overrides\") pod \"ovnkube-node-86758\" (UID: \"d43a4e16-3834-4d31-bed4-42fc0aaaaf17\") " pod="openshift-ovn-kubernetes/ovnkube-node-86758"
Oct 31 12:27:18.130103 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.130034    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6b3e77ff-4445-42fa-a7c3-f35d442c276d-host-var-lib-cni-bin\") pod \"multus-ccps4\" (UID: \"6b3e77ff-4445-42fa-a7c3-f35d442c276d\") " pod="openshift-multus/multus-ccps4"
Oct 31 12:27:18.130103 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.130061    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6b3e77ff-4445-42fa-a7c3-f35d442c276d-host-run-multus-certs\") pod \"multus-ccps4\" (UID: \"6b3e77ff-4445-42fa-a7c3-f35d442c276d\") " pod="openshift-multus/multus-ccps4"
Oct 31 12:27:18.130103 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.130064    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0db116b4-d971-4966-bb05-db28d077b6fe-installation-pull-secrets\") pod \"image-registry-6dd647c7b7-9255m\" (UID: \"0db116b4-d971-4966-bb05-db28d077b6fe\") " pod="openshift-image-registry/image-registry-6dd647c7b7-9255m"
Oct 31 12:27:18.130103 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.130065    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a2b450be-ca79-4edf-a1ea-377531a49519-original-pull-secret\") pod \"global-pull-secret-syncer-2bmzr\" (UID: \"a2b450be-ca79-4edf-a1ea-377531a49519\") " pod="kube-system/global-pull-secret-syncer-2bmzr"
Oct 31 12:27:18.130103 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.130088    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9g26m\" (UniqueName: \"kubernetes.io/projected/556dda7b-d8e6-4558-b5f3-994b01d6a866-kube-api-access-9g26m\") pod \"router-default-659589798c-776bx\" (UID: \"556dda7b-d8e6-4558-b5f3-994b01d6a866\") " pod="openshift-ingress/router-default-659589798c-776bx"
Oct 31 12:27:18.130323 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.130125    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0db116b4-d971-4966-bb05-db28d077b6fe-trusted-ca\") pod \"image-registry-6dd647c7b7-9255m\" (UID: \"0db116b4-d971-4966-bb05-db28d077b6fe\") " pod="openshift-image-registry/image-registry-6dd647c7b7-9255m"
Oct 31 12:27:18.130323 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.130152    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9083c35a-9739-4f3e-8415-bad1e6433d40-image-registry-private-configuration\") pod \"image-registry-6cfdbfcdc4-2rq84\" (UID: \"9083c35a-9739-4f3e-8415-bad1e6433d40\") " pod="openshift-image-registry/image-registry-6cfdbfcdc4-2rq84"
Oct 31 12:27:18.130323 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.130178    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d43a4e16-3834-4d31-bed4-42fc0aaaaf17-env-overrides\") pod \"ovnkube-node-86758\" (UID: \"d43a4e16-3834-4d31-bed4-42fc0aaaaf17\") " pod="openshift-ovn-kubernetes/ovnkube-node-86758"
Oct 31 12:27:18.130323 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.130192    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9083c35a-9739-4f3e-8415-bad1e6433d40-registry-certificates\") pod \"image-registry-6cfdbfcdc4-2rq84\" (UID: \"9083c35a-9739-4f3e-8415-bad1e6433d40\") " pod="openshift-image-registry/image-registry-6cfdbfcdc4-2rq84"
Oct 31 12:27:18.130323 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.130263    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9083c35a-9739-4f3e-8415-bad1e6433d40-installation-pull-secrets\") pod \"image-registry-6cfdbfcdc4-2rq84\" (UID: \"9083c35a-9739-4f3e-8415-bad1e6433d40\") " pod="openshift-image-registry/image-registry-6cfdbfcdc4-2rq84"
Oct 31 12:27:18.130540 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.130329    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/99a606a8-546b-4145-8bae-a95061241069-serving-cert\") pod \"service-ca-operator-544d749c8c-c789s\" (UID: \"99a606a8-546b-4145-8bae-a95061241069\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-c789s"
Oct 31 12:27:18.130540 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.130346    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9083c35a-9739-4f3e-8415-bad1e6433d40-registry-certificates\") pod \"image-registry-6cfdbfcdc4-2rq84\" (UID: \"9083c35a-9739-4f3e-8415-bad1e6433d40\") " pod="openshift-image-registry/image-registry-6cfdbfcdc4-2rq84"
Oct 31 12:27:18.130540 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.130385    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6b3e77ff-4445-42fa-a7c3-f35d442c276d-host-run-k8s-cni-cncf-io\") pod \"multus-ccps4\" (UID: \"6b3e77ff-4445-42fa-a7c3-f35d442c276d\") " pod="openshift-multus/multus-ccps4"
Oct 31 12:27:18.130540 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.130435    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a1e578b0-2e77-4f52-8a44-379b6059c3d9-socket-dir\") pod \"aws-ebs-csi-driver-node-jmlvc\" (UID: \"a1e578b0-2e77-4f52-8a44-379b6059c3d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jmlvc"
Oct 31 12:27:18.130540 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.130467    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0db116b4-d971-4966-bb05-db28d077b6fe-trusted-ca\") pod \"image-registry-6dd647c7b7-9255m\" (UID: \"0db116b4-d971-4966-bb05-db28d077b6fe\") " pod="openshift-image-registry/image-registry-6dd647c7b7-9255m"
Oct 31 12:27:18.130540 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.130475    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/99a606a8-546b-4145-8bae-a95061241069-serving-cert\") pod \"service-ca-operator-544d749c8c-c789s\" (UID: \"99a606a8-546b-4145-8bae-a95061241069\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-c789s"
Oct 31 12:27:18.130540 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.130469    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9083c35a-9739-4f3e-8415-bad1e6433d40-registry-tls\") pod \"image-registry-6cfdbfcdc4-2rq84\" (UID: \"9083c35a-9739-4f3e-8415-bad1e6433d40\") " pod="openshift-image-registry/image-registry-6cfdbfcdc4-2rq84"
Oct 31 12:27:18.130540 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.130541    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrpck\" (UniqueName: \"kubernetes.io/projected/891b3c47-ff7c-4a7a-a09b-411361a843f7-kube-api-access-mrpck\") pod \"ingress-canary-rl55s\" (UID: \"891b3c47-ff7c-4a7a-a09b-411361a843f7\") " pod="openshift-ingress-canary/ingress-canary-rl55s"
Oct 31 12:27:18.130774 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.130589    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9083c35a-9739-4f3e-8415-bad1e6433d40-installation-pull-secrets\") pod \"image-registry-6cfdbfcdc4-2rq84\" (UID: \"9083c35a-9739-4f3e-8415-bad1e6433d40\") " pod="openshift-image-registry/image-registry-6cfdbfcdc4-2rq84"
Oct 31 12:27:18.130774 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.130587    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee-etc-sysconfig\") pod \"tuned-qkt2l\" (UID: \"51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee\") " pod="openshift-cluster-node-tuning-operator/tuned-qkt2l"
Oct 31 12:27:18.130774 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.130628    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb20e317-3ebe-4f5c-9b37-1de9ea5f063f-service-ca-bundle\") pod \"periodic-gathering-87ncb-cnk5g\" (UID: \"eb20e317-3ebe-4f5c-9b37-1de9ea5f063f\") " pod="openshift-insights/periodic-gathering-87ncb-cnk5g"
Oct 31 12:27:18.130774 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.130671    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/dbbd2d70-8911-4aa5-96cf-3a16dcdf246d-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-tcnrw\" (UID: \"dbbd2d70-8911-4aa5-96cf-3a16dcdf246d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-tcnrw"
Oct 31 12:27:18.130774 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.130695    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a1c3175b-e32e-420e-877f-71bfe64d9c6c-cni-binary-copy\") pod \"multus-additional-cni-plugins-44zj2\" (UID: \"a1c3175b-e32e-420e-877f-71bfe64d9c6c\") " pod="openshift-multus/multus-additional-cni-plugins-44zj2"
Oct 31 12:27:18.130774 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.130722    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/50ad7816-358d-461e-a9bd-fadce6246e8f-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-2bfvp\" (UID: \"50ad7816-358d-461e-a9bd-fadce6246e8f\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-2bfvp"
Oct 31 12:27:18.130774 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.130752    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0db116b4-d971-4966-bb05-db28d077b6fe-registry-tls\") pod \"image-registry-6dd647c7b7-9255m\" (UID: \"0db116b4-d971-4966-bb05-db28d077b6fe\") " pod="openshift-image-registry/image-registry-6dd647c7b7-9255m"
Oct 31 12:27:18.131063 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.130800    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9083c35a-9739-4f3e-8415-bad1e6433d40-ca-trust-extracted\") pod \"image-registry-6cfdbfcdc4-2rq84\" (UID: \"9083c35a-9739-4f3e-8415-bad1e6433d40\") " pod="openshift-image-registry/image-registry-6cfdbfcdc4-2rq84"
Oct 31 12:27:18.131063 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.130819    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee-etc-sysctl-conf\") pod \"tuned-qkt2l\" (UID: \"51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee\") " pod="openshift-cluster-node-tuning-operator/tuned-qkt2l"
Oct 31 12:27:18.131063 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.130836    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqk48\" (UniqueName: \"kubernetes.io/projected/e64bd69d-946f-42ad-94ae-b87b7de3a581-kube-api-access-gqk48\") pod \"network-metrics-daemon-94kh9\" (UID: \"e64bd69d-946f-42ad-94ae-b87b7de3a581\") " pod="openshift-multus/network-metrics-daemon-94kh9"
Oct 31 12:27:18.131063 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.130851    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/99a606a8-546b-4145-8bae-a95061241069-config\") pod \"service-ca-operator-544d749c8c-c789s\" (UID: \"99a606a8-546b-4145-8bae-a95061241069\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-c789s"
Oct 31 12:27:18.131063 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.130869    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/b9ea9813-55b9-454b-8f10-985533305f85-signing-cabundle\") pod \"service-ca-5fffc97bdf-lmwkw\" (UID: \"b9ea9813-55b9-454b-8f10-985533305f85\") " pod="openshift-service-ca/service-ca-5fffc97bdf-lmwkw"
Oct 31 12:27:18.131063 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.130899    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/dbbd2d70-8911-4aa5-96cf-3a16dcdf246d-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-tcnrw\" (UID: \"dbbd2d70-8911-4aa5-96cf-3a16dcdf246d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-tcnrw"
Oct 31 12:27:18.131063 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.130927    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhm56\" (UniqueName: \"kubernetes.io/projected/5c8865f9-fdf6-49b9-8b4f-69dbc7787084-kube-api-access-mhm56\") pod \"dns-default-r6fvt\" (UID: \"5c8865f9-fdf6-49b9-8b4f-69dbc7787084\") " pod="openshift-dns/dns-default-r6fvt"
Oct 31 12:27:18.131063 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.130941    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a1c3175b-e32e-420e-877f-71bfe64d9c6c-cni-binary-copy\") pod \"multus-additional-cni-plugins-44zj2\" (UID: \"a1c3175b-e32e-420e-877f-71bfe64d9c6c\") " pod="openshift-multus/multus-additional-cni-plugins-44zj2"
Oct 31 12:27:18.131063 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.130978    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9557a3f0-ae0f-41f9-a9b5-15aeda410a38-data-volume\") pod \"insights-runtime-extractor-p7phb\" (UID: \"9557a3f0-ae0f-41f9-a9b5-15aeda410a38\") " pod="openshift-insights/insights-runtime-extractor-p7phb"
Oct 31 12:27:18.131063 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.131042    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9083c35a-9739-4f3e-8415-bad1e6433d40-ca-trust-extracted\") pod \"image-registry-6cfdbfcdc4-2rq84\" (UID: \"9083c35a-9739-4f3e-8415-bad1e6433d40\") " pod="openshift-image-registry/image-registry-6cfdbfcdc4-2rq84"
Oct 31 12:27:18.131063 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.131057    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9557a3f0-ae0f-41f9-a9b5-15aeda410a38-data-volume\") pod \"insights-runtime-extractor-p7phb\" (UID: \"9557a3f0-ae0f-41f9-a9b5-15aeda410a38\") " pod="openshift-insights/insights-runtime-extractor-p7phb"
Oct 31 12:27:18.131063 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.131044    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d43a4e16-3834-4d31-bed4-42fc0aaaaf17-var-lib-openvswitch\") pod \"ovnkube-node-86758\" (UID: \"d43a4e16-3834-4d31-bed4-42fc0aaaaf17\") " pod="openshift-ovn-kubernetes/ovnkube-node-86758"
Oct 31 12:27:18.131462 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.131066    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/b9ea9813-55b9-454b-8f10-985533305f85-signing-cabundle\") pod \"service-ca-5fffc97bdf-lmwkw\" (UID: \"b9ea9813-55b9-454b-8f10-985533305f85\") " pod="openshift-service-ca/service-ca-5fffc97bdf-lmwkw"
Oct 31 12:27:18.131462 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.131095    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d43a4e16-3834-4d31-bed4-42fc0aaaaf17-ovnkube-config\") pod \"ovnkube-node-86758\" (UID: \"d43a4e16-3834-4d31-bed4-42fc0aaaaf17\") " pod="openshift-ovn-kubernetes/ovnkube-node-86758"
Oct 31 12:27:18.131462 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.131113    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb20e317-3ebe-4f5c-9b37-1de9ea5f063f-service-ca-bundle\") pod \"periodic-gathering-87ncb-cnk5g\" (UID: \"eb20e317-3ebe-4f5c-9b37-1de9ea5f063f\") " pod="openshift-insights/periodic-gathering-87ncb-cnk5g"
Oct 31 12:27:18.131462 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.131139    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9083c35a-9739-4f3e-8415-bad1e6433d40-bound-sa-token\") pod \"image-registry-6cfdbfcdc4-2rq84\" (UID: \"9083c35a-9739-4f3e-8415-bad1e6433d40\") " pod="openshift-image-registry/image-registry-6cfdbfcdc4-2rq84"
Oct 31 12:27:18.131462 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.131164    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b5014b57-9b83-4fc9-b0ed-55793b2b3a98-agent-certs\") pod \"konnectivity-agent-8jlvq\" (UID: \"b5014b57-9b83-4fc9-b0ed-55793b2b3a98\") " pod="kube-system/konnectivity-agent-8jlvq"
Oct 31 12:27:18.131462 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.131189    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d43a4e16-3834-4d31-bed4-42fc0aaaaf17-host-kubelet\") pod \"ovnkube-node-86758\" (UID: \"d43a4e16-3834-4d31-bed4-42fc0aaaaf17\") " pod="openshift-ovn-kubernetes/ovnkube-node-86758"
Oct 31 12:27:18.131462 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.131225    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d43a4e16-3834-4d31-bed4-42fc0aaaaf17-etc-openvswitch\") pod \"ovnkube-node-86758\" (UID: \"d43a4e16-3834-4d31-bed4-42fc0aaaaf17\") " pod="openshift-ovn-kubernetes/ovnkube-node-86758"
Oct 31 12:27:18.131462 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.131227    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/99a606a8-546b-4145-8bae-a95061241069-config\") pod \"service-ca-operator-544d749c8c-c789s\" (UID: \"99a606a8-546b-4145-8bae-a95061241069\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-c789s"
Oct 31 12:27:18.131462 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.131273    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6b3e77ff-4445-42fa-a7c3-f35d442c276d-host-var-lib-cni-multus\") pod \"multus-ccps4\" (UID: \"6b3e77ff-4445-42fa-a7c3-f35d442c276d\") " pod="openshift-multus/multus-ccps4"
Oct 31 12:27:18.131462 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.131290    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d43a4e16-3834-4d31-bed4-42fc0aaaaf17-ovnkube-config\") pod \"ovnkube-node-86758\" (UID: \"d43a4e16-3834-4d31-bed4-42fc0aaaaf17\") " pod="openshift-ovn-kubernetes/ovnkube-node-86758"
Oct 31 12:27:18.131462 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.131302    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee-tmp\") pod \"tuned-qkt2l\" (UID: \"51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee\") " pod="openshift-cluster-node-tuning-operator/tuned-qkt2l"
Oct 31 12:27:18.131462 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.131338    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bm8x9\" (UniqueName: \"kubernetes.io/projected/b9ea9813-55b9-454b-8f10-985533305f85-kube-api-access-bm8x9\") pod \"service-ca-5fffc97bdf-lmwkw\" (UID: \"b9ea9813-55b9-454b-8f10-985533305f85\") " pod="openshift-service-ca/service-ca-5fffc97bdf-lmwkw"
Oct 31 12:27:18.131462 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.131364    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a1c3175b-e32e-420e-877f-71bfe64d9c6c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-44zj2\" (UID: \"a1c3175b-e32e-420e-877f-71bfe64d9c6c\") " pod="openshift-multus/multus-additional-cni-plugins-44zj2"
Oct 31 12:27:18.131462 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.131372    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee-tmp\") pod \"tuned-qkt2l\" (UID: \"51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee\") " pod="openshift-cluster-node-tuning-operator/tuned-qkt2l"
Oct 31 12:27:18.131462 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.131357    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b5014b57-9b83-4fc9-b0ed-55793b2b3a98-agent-certs\") pod \"konnectivity-agent-8jlvq\" (UID: \"b5014b57-9b83-4fc9-b0ed-55793b2b3a98\") " pod="kube-system/konnectivity-agent-8jlvq"
Oct 31 12:27:18.131462 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.131393    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6b3e77ff-4445-42fa-a7c3-f35d442c276d-multus-socket-dir-parent\") pod \"multus-ccps4\" (UID: \"6b3e77ff-4445-42fa-a7c3-f35d442c276d\") " pod="openshift-multus/multus-ccps4"
Oct 31 12:27:18.131462 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.131420    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6b3e77ff-4445-42fa-a7c3-f35d442c276d-hostroot\") pod \"multus-ccps4\" (UID: \"6b3e77ff-4445-42fa-a7c3-f35d442c276d\") " pod="openshift-multus/multus-ccps4"
Oct 31 12:27:18.131462 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.131443    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6b3e77ff-4445-42fa-a7c3-f35d442c276d-etc-kubernetes\") pod \"multus-ccps4\" (UID: \"6b3e77ff-4445-42fa-a7c3-f35d442c276d\") " pod="openshift-multus/multus-ccps4"
Oct 31 12:27:18.131462 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.131469    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6b3e77ff-4445-42fa-a7c3-f35d442c276d-multus-cni-dir\") pod \"multus-ccps4\" (UID: \"6b3e77ff-4445-42fa-a7c3-f35d442c276d\") " pod="openshift-multus/multus-ccps4"
Oct 31 12:27:18.132108 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.131518    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0db116b4-d971-4966-bb05-db28d077b6fe-bound-sa-token\") pod \"image-registry-6dd647c7b7-9255m\" (UID: \"0db116b4-d971-4966-bb05-db28d077b6fe\") " pod="openshift-image-registry/image-registry-6dd647c7b7-9255m"
Oct 31 12:27:18.132108 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.131547    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a1c3175b-e32e-420e-877f-71bfe64d9c6c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-44zj2\" (UID: \"a1c3175b-e32e-420e-877f-71bfe64d9c6c\") " pod="openshift-multus/multus-additional-cni-plugins-44zj2"
Oct 31 12:27:18.132108 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.131558    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee-etc-modprobe-d\") pod \"tuned-qkt2l\" (UID: \"51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee\") " pod="openshift-cluster-node-tuning-operator/tuned-qkt2l"
Oct 31 12:27:18.132108 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.131595    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d43a4e16-3834-4d31-bed4-42fc0aaaaf17-host-run-ovn-kubernetes\") pod \"ovnkube-node-86758\" (UID: \"d43a4e16-3834-4d31-bed4-42fc0aaaaf17\") " pod="openshift-ovn-kubernetes/ovnkube-node-86758"
Oct 31 12:27:18.132108 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.131615    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6b3e77ff-4445-42fa-a7c3-f35d442c276d-cni-binary-copy\") pod \"multus-ccps4\" (UID: \"6b3e77ff-4445-42fa-a7c3-f35d442c276d\") " pod="openshift-multus/multus-ccps4"
Oct 31 12:27:18.132108 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.131631    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee-lib-modules\") pod \"tuned-qkt2l\" (UID: \"51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee\") " pod="openshift-cluster-node-tuning-operator/tuned-qkt2l"
Oct 31 12:27:18.132108 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.131647    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee-host\") pod \"tuned-qkt2l\" (UID: \"51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee\") " pod="openshift-cluster-node-tuning-operator/tuned-qkt2l"
Oct 31 12:27:18.132108 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.131663    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6b3e77ff-4445-42fa-a7c3-f35d442c276d-host-run-netns\") pod \"multus-ccps4\" (UID: \"6b3e77ff-4445-42fa-a7c3-f35d442c276d\") " pod="openshift-multus/multus-ccps4"
Oct 31 12:27:18.132108 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.131757    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f032909d-bc39-4c9e-99b7-f3d800842119-serving-cert\") pod \"console-operator-7cd56b8dbc-pcvvm\" (UID: \"f032909d-bc39-4c9e-99b7-f3d800842119\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-pcvvm"
Oct 31 12:27:18.132108 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.131779    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jl4xd\" (UniqueName: \"kubernetes.io/projected/9557a3f0-ae0f-41f9-a9b5-15aeda410a38-kube-api-access-jl4xd\") pod \"insights-runtime-extractor-p7phb\" (UID: \"9557a3f0-ae0f-41f9-a9b5-15aeda410a38\") " pod="openshift-insights/insights-runtime-extractor-p7phb"
Oct 31 12:27:18.132108 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.131797    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/eb20e317-3ebe-4f5c-9b37-1de9ea5f063f-archives-path\") pod \"periodic-gathering-87ncb-cnk5g\" (UID: \"eb20e317-3ebe-4f5c-9b37-1de9ea5f063f\") " pod="openshift-insights/periodic-gathering-87ncb-cnk5g"
Oct 31 12:27:18.132108 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.131798    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6b3e77ff-4445-42fa-a7c3-f35d442c276d-cni-binary-copy\") pod \"multus-ccps4\" (UID: \"6b3e77ff-4445-42fa-a7c3-f35d442c276d\") " pod="openshift-multus/multus-ccps4"
Oct 31 12:27:18.132108 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.131813    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/dbbd2d70-8911-4aa5-96cf-3a16dcdf246d-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-tcnrw\" (UID: \"dbbd2d70-8911-4aa5-96cf-3a16dcdf246d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-tcnrw"
Oct 31 12:27:18.132108 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.131832    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d43a4e16-3834-4d31-bed4-42fc0aaaaf17-ovn-node-metrics-cert\") pod \"ovnkube-node-86758\" (UID: \"d43a4e16-3834-4d31-bed4-42fc0aaaaf17\") " pod="openshift-ovn-kubernetes/ovnkube-node-86758"
Oct 31 12:27:18.132108 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.131870    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/eb20e317-3ebe-4f5c-9b37-1de9ea5f063f-archives-path\") pod \"periodic-gathering-87ncb-cnk5g\" (UID: \"eb20e317-3ebe-4f5c-9b37-1de9ea5f063f\") " pod="openshift-insights/periodic-gathering-87ncb-cnk5g"
Oct 31 12:27:18.132108 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.131939    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f032909d-bc39-4c9e-99b7-f3d800842119-serving-cert\") pod \"console-operator-7cd56b8dbc-pcvvm\" (UID: \"f032909d-bc39-4c9e-99b7-f3d800842119\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-pcvvm"
Oct 31 12:27:18.132108 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.131968    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a1e578b0-2e77-4f52-8a44-379b6059c3d9-device-dir\") pod \"aws-ebs-csi-driver-node-jmlvc\" (UID: \"a1e578b0-2e77-4f52-8a44-379b6059c3d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jmlvc"
Oct 31 12:27:18.132108 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.131977    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d43a4e16-3834-4d31-bed4-42fc0aaaaf17-ovn-node-metrics-cert\") pod \"ovnkube-node-86758\" (UID: \"d43a4e16-3834-4d31-bed4-42fc0aaaaf17\") " pod="openshift-ovn-kubernetes/ovnkube-node-86758"
Oct 31 12:27:18.132108 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.132008    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee-etc-sysctl-d\") pod \"tuned-qkt2l\" (UID: \"51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee\") " pod="openshift-cluster-node-tuning-operator/tuned-qkt2l"
Oct 31 12:27:18.132108 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.132045    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee-sys\") pod \"tuned-qkt2l\" (UID: \"51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee\") " pod="openshift-cluster-node-tuning-operator/tuned-qkt2l"
Oct 31 12:27:18.132108 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.132065    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a1c3175b-e32e-420e-877f-71bfe64d9c6c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-44zj2\" (UID: \"a1c3175b-e32e-420e-877f-71bfe64d9c6c\") " pod="openshift-multus/multus-additional-cni-plugins-44zj2"
Oct 31 12:27:18.132108 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.132094    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwj8z\" (UniqueName: \"kubernetes.io/projected/a1c3175b-e32e-420e-877f-71bfe64d9c6c-kube-api-access-nwj8z\") pod \"multus-additional-cni-plugins-44zj2\" (UID: \"a1c3175b-e32e-420e-877f-71bfe64d9c6c\") " pod="openshift-multus/multus-additional-cni-plugins-44zj2"
Oct 31 12:27:18.132799 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.132121    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a1e578b0-2e77-4f52-8a44-379b6059c3d9-registration-dir\") pod \"aws-ebs-csi-driver-node-jmlvc\" (UID: \"a1e578b0-2e77-4f52-8a44-379b6059c3d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jmlvc"
Oct 31 12:27:18.132799 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.132162    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3bb2c04e-55a2-4206-9530-7298f4469379-host-slash\") pod \"iptables-alerter-plbnz\" (UID: \"3bb2c04e-55a2-4206-9530-7298f4469379\") " pod="openshift-network-operator/iptables-alerter-plbnz"
Oct 31 12:27:18.132799 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.132249    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xv9qt\" (UniqueName: \"kubernetes.io/projected/3bb2c04e-55a2-4206-9530-7298f4469379-kube-api-access-xv9qt\") pod \"iptables-alerter-plbnz\" (UID: \"3bb2c04e-55a2-4206-9530-7298f4469379\") " pod="openshift-network-operator/iptables-alerter-plbnz"
Oct 31 12:27:18.132799 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.132267    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9557a3f0-ae0f-41f9-a9b5-15aeda410a38-crio-socket\") pod \"insights-runtime-extractor-p7phb\" (UID: \"9557a3f0-ae0f-41f9-a9b5-15aeda410a38\") " pod="openshift-insights/insights-runtime-extractor-p7phb"
Oct 31 12:27:18.132799 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.132365    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48lnl\" (UniqueName: \"kubernetes.io/projected/64649bfe-c3e6-469a-a9ff-0db1ca0fadb3-kube-api-access-48lnl\") pod \"migrator-66df5d496b-skkg4\" (UID: \"64649bfe-c3e6-469a-a9ff-0db1ca0fadb3\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-skkg4"
Oct 31 12:27:18.132799 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.132395    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb20e317-3ebe-4f5c-9b37-1de9ea5f063f-trusted-ca-bundle\") pod \"periodic-gathering-87ncb-cnk5g\" (UID: \"eb20e317-3ebe-4f5c-9b37-1de9ea5f063f\") " pod="openshift-insights/periodic-gathering-87ncb-cnk5g"
Oct 31 12:27:18.132799 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.132439    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvtwq\" (UniqueName: \"kubernetes.io/projected/eb20e317-3ebe-4f5c-9b37-1de9ea5f063f-kube-api-access-kvtwq\") pod \"periodic-gathering-87ncb-cnk5g\" (UID: \"eb20e317-3ebe-4f5c-9b37-1de9ea5f063f\") " pod="openshift-insights/periodic-gathering-87ncb-cnk5g"
Oct 31 12:27:18.132799 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.132475    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a1c3175b-e32e-420e-877f-71bfe64d9c6c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-44zj2\" (UID: \"a1c3175b-e32e-420e-877f-71bfe64d9c6c\") " pod="openshift-multus/multus-additional-cni-plugins-44zj2"
Oct 31 12:27:18.132799 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.132571    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0db116b4-d971-4966-bb05-db28d077b6fe-image-registry-private-configuration\") pod \"image-registry-6dd647c7b7-9255m\" (UID: \"0db116b4-d971-4966-bb05-db28d077b6fe\") " pod="openshift-image-registry/image-registry-6dd647c7b7-9255m"
Oct 31 12:27:18.132799 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.132645    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xwrq\" (UniqueName: \"kubernetes.io/projected/9083c35a-9739-4f3e-8415-bad1e6433d40-kube-api-access-5xwrq\") pod \"image-registry-6cfdbfcdc4-2rq84\" (UID: \"9083c35a-9739-4f3e-8415-bad1e6433d40\") " pod="openshift-image-registry/image-registry-6cfdbfcdc4-2rq84"
Oct 31 12:27:18.132799 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.132673    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee-run\") pod \"tuned-qkt2l\" (UID: \"51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee\") " pod="openshift-cluster-node-tuning-operator/tuned-qkt2l"
Oct 31 12:27:18.132799 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.132697    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d43a4e16-3834-4d31-bed4-42fc0aaaaf17-log-socket\") pod \"ovnkube-node-86758\" (UID: \"d43a4e16-3834-4d31-bed4-42fc0aaaaf17\") " pod="openshift-ovn-kubernetes/ovnkube-node-86758"
Oct 31 12:27:18.132799 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.132727    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a1c3175b-e32e-420e-877f-71bfe64d9c6c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-44zj2\" (UID: \"a1c3175b-e32e-420e-877f-71bfe64d9c6c\") " pod="openshift-multus/multus-additional-cni-plugins-44zj2"
Oct 31 12:27:18.132799 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.132722    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5c8865f9-fdf6-49b9-8b4f-69dbc7787084-metrics-tls\") pod \"dns-default-r6fvt\" (UID: \"5c8865f9-fdf6-49b9-8b4f-69dbc7787084\") " pod="openshift-dns/dns-default-r6fvt"
Oct 31 12:27:18.132799 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.132777    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/556dda7b-d8e6-4558-b5f3-994b01d6a866-default-certificate\") pod \"router-default-659589798c-776bx\" (UID: \"556dda7b-d8e6-4558-b5f3-994b01d6a866\") " pod="openshift-ingress/router-default-659589798c-776bx"
Oct 31 12:27:18.132799 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.132784    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0db116b4-d971-4966-bb05-db28d077b6fe-image-registry-private-configuration\") pod \"image-registry-6dd647c7b7-9255m\" (UID: \"0db116b4-d971-4966-bb05-db28d077b6fe\") " pod="openshift-image-registry/image-registry-6dd647c7b7-9255m"
Oct 31 12:27:18.133477 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.132827    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlgsp\" (UniqueName: \"kubernetes.io/projected/0db116b4-d971-4966-bb05-db28d077b6fe-kube-api-access-nlgsp\") pod \"image-registry-6dd647c7b7-9255m\" (UID: \"0db116b4-d971-4966-bb05-db28d077b6fe\") " pod="openshift-image-registry/image-registry-6dd647c7b7-9255m"
Oct 31 12:27:18.133477 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.132948    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee-etc-kubernetes\") pod \"tuned-qkt2l\" (UID: \"51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee\") " pod="openshift-cluster-node-tuning-operator/tuned-qkt2l"
Oct 31 12:27:18.133477 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.132970    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9557a3f0-ae0f-41f9-a9b5-15aeda410a38-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p7phb\" (UID: \"9557a3f0-ae0f-41f9-a9b5-15aeda410a38\") " pod="openshift-insights/insights-runtime-extractor-p7phb"
Oct 31 12:27:18.133477 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.132992    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpcl9\" (UniqueName: \"kubernetes.io/projected/031551df-64a5-411b-8eb8-7f0bf2d4f170-kube-api-access-wpcl9\") pod \"migrator-66df5d496b-m4fpw\" (UID: \"031551df-64a5-411b-8eb8-7f0bf2d4f170\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-m4fpw"
Oct 31 12:27:18.133477 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.133030    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7j2rp\" (UniqueName: \"kubernetes.io/projected/6bbe13aa-df4b-40b0-8595-68fe1166a082-kube-api-access-7j2rp\") pod \"node-resolver-v9ks4\" (UID: \"6bbe13aa-df4b-40b0-8595-68fe1166a082\") " pod="openshift-dns/node-resolver-v9ks4"
Oct 31 12:27:18.133477 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.133064    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3bb2c04e-55a2-4206-9530-7298f4469379-iptables-alerter-script\") pod \"iptables-alerter-plbnz\" (UID: \"3bb2c04e-55a2-4206-9530-7298f4469379\") " pod="openshift-network-operator/iptables-alerter-plbnz"
Oct 31 12:27:18.133477 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.133080    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/556dda7b-d8e6-4558-b5f3-994b01d6a866-default-certificate\") pod \"router-default-659589798c-776bx\" (UID: \"556dda7b-d8e6-4558-b5f3-994b01d6a866\") " pod="openshift-ingress/router-default-659589798c-776bx"
Oct 31 12:27:18.133477 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.133109    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vknw2\" (UniqueName: \"kubernetes.io/projected/6b3e77ff-4445-42fa-a7c3-f35d442c276d-kube-api-access-vknw2\") pod \"multus-ccps4\" (UID: \"6b3e77ff-4445-42fa-a7c3-f35d442c276d\") " pod="openshift-multus/multus-ccps4"
Oct 31 12:27:18.133477 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.133140    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c8865f9-fdf6-49b9-8b4f-69dbc7787084-tmp-dir\") pod \"dns-default-r6fvt\" (UID: \"5c8865f9-fdf6-49b9-8b4f-69dbc7787084\") " pod="openshift-dns/dns-default-r6fvt"
Oct 31 12:27:18.133477 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.133221    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c8865f9-fdf6-49b9-8b4f-69dbc7787084-tmp-dir\") pod \"dns-default-r6fvt\" (UID: \"5c8865f9-fdf6-49b9-8b4f-69dbc7787084\") " pod="openshift-dns/dns-default-r6fvt"
Oct 31 12:27:18.133477 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.133241    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9083c35a-9739-4f3e-8415-bad1e6433d40-trusted-ca\") pod \"image-registry-6cfdbfcdc4-2rq84\" (UID: \"9083c35a-9739-4f3e-8415-bad1e6433d40\") " pod="openshift-image-registry/image-registry-6cfdbfcdc4-2rq84"
Oct 31 12:27:18.133477 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.133252    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb20e317-3ebe-4f5c-9b37-1de9ea5f063f-trusted-ca-bundle\") pod \"periodic-gathering-87ncb-cnk5g\" (UID: \"eb20e317-3ebe-4f5c-9b37-1de9ea5f063f\") " pod="openshift-insights/periodic-gathering-87ncb-cnk5g"
Oct 31 12:27:18.133477 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.133279    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d43a4e16-3834-4d31-bed4-42fc0aaaaf17-run-openvswitch\") pod \"ovnkube-node-86758\" (UID: \"d43a4e16-3834-4d31-bed4-42fc0aaaaf17\") " pod="openshift-ovn-kubernetes/ovnkube-node-86758"
Oct 31 12:27:18.133477 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.133303    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3bb2c04e-55a2-4206-9530-7298f4469379-iptables-alerter-script\") pod \"iptables-alerter-plbnz\" (UID: \"3bb2c04e-55a2-4206-9530-7298f4469379\") " pod="openshift-network-operator/iptables-alerter-plbnz"
Oct 31 12:27:18.133477 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.133308    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6b3e77ff-4445-42fa-a7c3-f35d442c276d-multus-conf-dir\") pod \"multus-ccps4\" (UID: \"6b3e77ff-4445-42fa-a7c3-f35d442c276d\") " pod="openshift-multus/multus-ccps4"
Oct 31 12:27:18.133477 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.133352    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/dda0c26f-8225-4239-bc27-866b66cbb919-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-gt9rn\" (UID: \"dda0c26f-8225-4239-bc27-866b66cbb919\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gt9rn"
Oct 31 12:27:18.133477 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.133389    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a1c3175b-e32e-420e-877f-71bfe64d9c6c-system-cni-dir\") pod \"multus-additional-cni-plugins-44zj2\" (UID: \"a1c3175b-e32e-420e-877f-71bfe64d9c6c\") " pod="openshift-multus/multus-additional-cni-plugins-44zj2"
Oct 31 12:27:18.133477 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.133418    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6t4dc\" (UniqueName: \"kubernetes.io/projected/99a606a8-546b-4145-8bae-a95061241069-kube-api-access-6t4dc\") pod \"service-ca-operator-544d749c8c-c789s\" (UID: \"99a606a8-546b-4145-8bae-a95061241069\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-c789s"
Oct 31 12:27:18.133477 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.133450    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a1e578b0-2e77-4f52-8a44-379b6059c3d9-etc-selinux\") pod \"aws-ebs-csi-driver-node-jmlvc\" (UID: \"a1e578b0-2e77-4f52-8a44-379b6059c3d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jmlvc"
Oct 31 12:27:18.134084 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:27:18.133505    5060 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 31 12:27:18.134084 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.133519    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9083c35a-9739-4f3e-8415-bad1e6433d40-trusted-ca\") pod \"image-registry-6cfdbfcdc4-2rq84\" (UID: \"9083c35a-9739-4f3e-8415-bad1e6433d40\") " pod="openshift-image-registry/image-registry-6cfdbfcdc4-2rq84"
Oct 31 12:27:18.134084 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.133505    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/50ad7816-358d-461e-a9bd-fadce6246e8f-nginx-conf\") pod \"networking-console-plugin-86c478bb88-2bfvp\" (UID: \"50ad7816-358d-461e-a9bd-fadce6246e8f\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-2bfvp"
Oct 31 12:27:18.134084 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:27:18.133588    5060 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dda0c26f-8225-4239-bc27-866b66cbb919-cluster-monitoring-operator-tls podName:dda0c26f-8225-4239-bc27-866b66cbb919 nodeName:}" failed. No retries permitted until 2026-10-31 12:27:18.633566769 +0000 UTC m=+1.736634492 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/dda0c26f-8225-4239-bc27-866b66cbb919-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-gt9rn" (UID: "dda0c26f-8225-4239-bc27-866b66cbb919") : secret "cluster-monitoring-operator-tls" not found
Oct 31 12:27:18.134084 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.133610    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsg9j\" (UniqueName: \"kubernetes.io/projected/a1e578b0-2e77-4f52-8a44-379b6059c3d9-kube-api-access-dsg9j\") pod \"aws-ebs-csi-driver-node-jmlvc\" (UID: \"a1e578b0-2e77-4f52-8a44-379b6059c3d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jmlvc"
Oct 31 12:27:18.134084 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.133640    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/556dda7b-d8e6-4558-b5f3-994b01d6a866-service-ca-bundle\") pod \"router-default-659589798c-776bx\" (UID: \"556dda7b-d8e6-4558-b5f3-994b01d6a866\") " pod="openshift-ingress/router-default-659589798c-776bx"
Oct 31 12:27:18.134084 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.133690    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/50ad7816-358d-461e-a9bd-fadce6246e8f-nginx-conf\") pod \"networking-console-plugin-86c478bb88-2bfvp\" (UID: \"50ad7816-358d-461e-a9bd-fadce6246e8f\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-2bfvp"
Oct 31 12:27:18.137935 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.137914    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bs5p9\" (UniqueName: \"kubernetes.io/projected/dbbd2d70-8911-4aa5-96cf-3a16dcdf246d-kube-api-access-bs5p9\") pod \"cluster-samples-operator-6ddb6c8b94-tcnrw\" (UID: \"dbbd2d70-8911-4aa5-96cf-3a16dcdf246d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-tcnrw"
Oct 31 12:27:18.137996 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.137968    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rblt\" (UniqueName: \"kubernetes.io/projected/f032909d-bc39-4c9e-99b7-f3d800842119-kube-api-access-9rblt\") pod \"console-operator-7cd56b8dbc-pcvvm\" (UID: \"f032909d-bc39-4c9e-99b7-f3d800842119\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-pcvvm"
Oct 31 12:27:18.138133 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.138116    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nx5hk\" (UniqueName: \"kubernetes.io/projected/6c661ce8-9f7b-4509-bd17-2380bcd1c425-kube-api-access-nx5hk\") pod \"node-ca-wdkzp\" (UID: \"6c661ce8-9f7b-4509-bd17-2380bcd1c425\") " pod="openshift-image-registry/node-ca-wdkzp"
Oct 31 12:27:18.138312 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.138295    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvqpw\" (UniqueName: \"kubernetes.io/projected/25691fa3-7017-429a-b94c-c2dde4d29e55-kube-api-access-gvqpw\") pod \"network-check-source-6f78d59f99-m9xxh\" (UID: \"25691fa3-7017-429a-b94c-c2dde4d29e55\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-m9xxh"
Oct 31 12:27:18.146884 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.146835    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzcgz\" (UniqueName: \"kubernetes.io/projected/d43a4e16-3834-4d31-bed4-42fc0aaaaf17-kube-api-access-hzcgz\") pod \"ovnkube-node-86758\" (UID: \"d43a4e16-3834-4d31-bed4-42fc0aaaaf17\") " pod="openshift-ovn-kubernetes/ovnkube-node-86758"
Oct 31 12:27:18.146884 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.146854    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tf6n\" (UniqueName: \"kubernetes.io/projected/7a402472-ca33-44ae-b598-96fc8e7a5eb2-kube-api-access-2tf6n\") pod \"network-check-target-tdrmt\" (UID: \"7a402472-ca33-44ae-b598-96fc8e7a5eb2\") " pod="openshift-network-diagnostics/network-check-target-tdrmt"
Oct 31 12:27:18.147172 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.147154    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5nx4\" (UniqueName: \"kubernetes.io/projected/51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee-kube-api-access-t5nx4\") pod \"tuned-qkt2l\" (UID: \"51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee\") " pod="openshift-cluster-node-tuning-operator/tuned-qkt2l"
Oct 31 12:27:18.149343 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.149320    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsg9j\" (UniqueName: \"kubernetes.io/projected/a1e578b0-2e77-4f52-8a44-379b6059c3d9-kube-api-access-dsg9j\") pod \"aws-ebs-csi-driver-node-jmlvc\" (UID: \"a1e578b0-2e77-4f52-8a44-379b6059c3d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jmlvc"
Oct 31 12:27:18.149620 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.149596    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlgsp\" (UniqueName: \"kubernetes.io/projected/0db116b4-d971-4966-bb05-db28d077b6fe-kube-api-access-nlgsp\") pod \"image-registry-6dd647c7b7-9255m\" (UID: \"0db116b4-d971-4966-bb05-db28d077b6fe\") " pod="openshift-image-registry/image-registry-6dd647c7b7-9255m"
Oct 31 12:27:18.149829 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.149811    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqk48\" (UniqueName: \"kubernetes.io/projected/e64bd69d-946f-42ad-94ae-b87b7de3a581-kube-api-access-gqk48\") pod \"network-metrics-daemon-94kh9\" (UID: \"e64bd69d-946f-42ad-94ae-b87b7de3a581\") " pod="openshift-multus/network-metrics-daemon-94kh9"
Oct 31 12:27:18.149920 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.149904    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvtwq\" (UniqueName: \"kubernetes.io/projected/eb20e317-3ebe-4f5c-9b37-1de9ea5f063f-kube-api-access-kvtwq\") pod \"periodic-gathering-87ncb-cnk5g\" (UID: \"eb20e317-3ebe-4f5c-9b37-1de9ea5f063f\") " pod="openshift-insights/periodic-gathering-87ncb-cnk5g"
Oct 31 12:27:18.149955 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.149909    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0db116b4-d971-4966-bb05-db28d077b6fe-bound-sa-token\") pod \"image-registry-6dd647c7b7-9255m\" (UID: \"0db116b4-d971-4966-bb05-db28d077b6fe\") " pod="openshift-image-registry/image-registry-6dd647c7b7-9255m"
Oct 31 12:27:18.149986 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.149966    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7j2rp\" (UniqueName: \"kubernetes.io/projected/6bbe13aa-df4b-40b0-8595-68fe1166a082-kube-api-access-7j2rp\") pod \"node-resolver-v9ks4\" (UID: \"6bbe13aa-df4b-40b0-8595-68fe1166a082\") " pod="openshift-dns/node-resolver-v9ks4"
Oct 31 12:27:18.150036 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.150021    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9g26m\" (UniqueName: \"kubernetes.io/projected/556dda7b-d8e6-4558-b5f3-994b01d6a866-kube-api-access-9g26m\") pod \"router-default-659589798c-776bx\" (UID: \"556dda7b-d8e6-4558-b5f3-994b01d6a866\") " pod="openshift-ingress/router-default-659589798c-776bx"
Oct 31 12:27:18.150510 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.150480    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48lnl\" (UniqueName: \"kubernetes.io/projected/64649bfe-c3e6-469a-a9ff-0db1ca0fadb3-kube-api-access-48lnl\") pod \"migrator-66df5d496b-skkg4\" (UID: \"64649bfe-c3e6-469a-a9ff-0db1ca0fadb3\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-skkg4"
Oct 31 12:27:18.151955 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.151937    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xv9qt\" (UniqueName: \"kubernetes.io/projected/3bb2c04e-55a2-4206-9530-7298f4469379-kube-api-access-xv9qt\") pod \"iptables-alerter-plbnz\" (UID: \"3bb2c04e-55a2-4206-9530-7298f4469379\") " pod="openshift-network-operator/iptables-alerter-plbnz"
Oct 31 12:27:18.152277 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.152239    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9083c35a-9739-4f3e-8415-bad1e6433d40-bound-sa-token\") pod \"image-registry-6cfdbfcdc4-2rq84\" (UID: \"9083c35a-9739-4f3e-8415-bad1e6433d40\") " pod="openshift-image-registry/image-registry-6cfdbfcdc4-2rq84"
Oct 31 12:27:18.152277 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.152262    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6t4dc\" (UniqueName: \"kubernetes.io/projected/99a606a8-546b-4145-8bae-a95061241069-kube-api-access-6t4dc\") pod \"service-ca-operator-544d749c8c-c789s\" (UID: \"99a606a8-546b-4145-8bae-a95061241069\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-c789s"
Oct 31 12:27:18.152525 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.152507    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vknw2\" (UniqueName: \"kubernetes.io/projected/6b3e77ff-4445-42fa-a7c3-f35d442c276d-kube-api-access-vknw2\") pod \"multus-ccps4\" (UID: \"6b3e77ff-4445-42fa-a7c3-f35d442c276d\") " pod="openshift-multus/multus-ccps4"
Oct 31 12:27:18.155565 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.154691    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwj8z\" (UniqueName: \"kubernetes.io/projected/a1c3175b-e32e-420e-877f-71bfe64d9c6c-kube-api-access-nwj8z\") pod \"multus-additional-cni-plugins-44zj2\" (UID: \"a1c3175b-e32e-420e-877f-71bfe64d9c6c\") " pod="openshift-multus/multus-additional-cni-plugins-44zj2"
Oct 31 12:27:18.155565 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.154806    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jl4xd\" (UniqueName: \"kubernetes.io/projected/9557a3f0-ae0f-41f9-a9b5-15aeda410a38-kube-api-access-jl4xd\") pod \"insights-runtime-extractor-p7phb\" (UID: \"9557a3f0-ae0f-41f9-a9b5-15aeda410a38\") " pod="openshift-insights/insights-runtime-extractor-p7phb"
Oct 31 12:27:18.155565 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.155102    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xwrq\" (UniqueName: \"kubernetes.io/projected/9083c35a-9739-4f3e-8415-bad1e6433d40-kube-api-access-5xwrq\") pod \"image-registry-6cfdbfcdc4-2rq84\" (UID: \"9083c35a-9739-4f3e-8415-bad1e6433d40\") " pod="openshift-image-registry/image-registry-6cfdbfcdc4-2rq84"
Oct 31 12:27:18.155565 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.155116    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhm56\" (UniqueName: \"kubernetes.io/projected/5c8865f9-fdf6-49b9-8b4f-69dbc7787084-kube-api-access-mhm56\") pod \"dns-default-r6fvt\" (UID: \"5c8865f9-fdf6-49b9-8b4f-69dbc7787084\") " pod="openshift-dns/dns-default-r6fvt"
Oct 31 12:27:18.155565 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.155184    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrpck\" (UniqueName: \"kubernetes.io/projected/891b3c47-ff7c-4a7a-a09b-411361a843f7-kube-api-access-mrpck\") pod \"ingress-canary-rl55s\" (UID: \"891b3c47-ff7c-4a7a-a09b-411361a843f7\") " pod="openshift-ingress-canary/ingress-canary-rl55s"
Oct 31 12:27:18.155565 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.155511    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bm8x9\" (UniqueName: \"kubernetes.io/projected/b9ea9813-55b9-454b-8f10-985533305f85-kube-api-access-bm8x9\") pod \"service-ca-5fffc97bdf-lmwkw\" (UID: \"b9ea9813-55b9-454b-8f10-985533305f85\") " pod="openshift-service-ca/service-ca-5fffc97bdf-lmwkw"
Oct 31 12:27:18.155793 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.155565    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpcl9\" (UniqueName: \"kubernetes.io/projected/031551df-64a5-411b-8eb8-7f0bf2d4f170-kube-api-access-wpcl9\") pod \"migrator-66df5d496b-m4fpw\" (UID: \"031551df-64a5-411b-8eb8-7f0bf2d4f170\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-m4fpw"
Oct 31 12:27:18.234576 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.234540    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/556dda7b-d8e6-4558-b5f3-994b01d6a866-metrics-certs\") pod \"router-default-659589798c-776bx\" (UID: \"556dda7b-d8e6-4558-b5f3-994b01d6a866\") " pod="openshift-ingress/router-default-659589798c-776bx"
Oct 31 12:27:18.234576 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.234580    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9557a3f0-ae0f-41f9-a9b5-15aeda410a38-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p7phb\" (UID: \"9557a3f0-ae0f-41f9-a9b5-15aeda410a38\") " pod="openshift-insights/insights-runtime-extractor-p7phb"
Oct 31 12:27:18.234841 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.234599    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d43a4e16-3834-4d31-bed4-42fc0aaaaf17-systemd-units\") pod \"ovnkube-node-86758\" (UID: \"d43a4e16-3834-4d31-bed4-42fc0aaaaf17\") " pod="openshift-ovn-kubernetes/ovnkube-node-86758"
Oct 31 12:27:18.234841 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.234616    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d43a4e16-3834-4d31-bed4-42fc0aaaaf17-host-cni-netd\") pod \"ovnkube-node-86758\" (UID: \"d43a4e16-3834-4d31-bed4-42fc0aaaaf17\") " pod="openshift-ovn-kubernetes/ovnkube-node-86758"
Oct 31 12:27:18.234841 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.234633    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d43a4e16-3834-4d31-bed4-42fc0aaaaf17-host-slash\") pod \"ovnkube-node-86758\" (UID: \"d43a4e16-3834-4d31-bed4-42fc0aaaaf17\") " pod="openshift-ovn-kubernetes/ovnkube-node-86758"
Oct 31 12:27:18.234841 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.234656    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a2b450be-ca79-4edf-a1ea-377531a49519-kubelet-config\") pod \"global-pull-secret-syncer-2bmzr\" (UID: \"a2b450be-ca79-4edf-a1ea-377531a49519\") " pod="kube-system/global-pull-secret-syncer-2bmzr"
Oct 31 12:27:18.234841 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.234678    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a2b450be-ca79-4edf-a1ea-377531a49519-dbus\") pod \"global-pull-secret-syncer-2bmzr\" (UID: \"a2b450be-ca79-4edf-a1ea-377531a49519\") " pod="kube-system/global-pull-secret-syncer-2bmzr"
Oct 31 12:27:18.234841 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.234692    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d43a4e16-3834-4d31-bed4-42fc0aaaaf17-systemd-units\") pod \"ovnkube-node-86758\" (UID: \"d43a4e16-3834-4d31-bed4-42fc0aaaaf17\") " pod="openshift-ovn-kubernetes/ovnkube-node-86758"
Oct 31 12:27:18.234841 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.234709    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d43a4e16-3834-4d31-bed4-42fc0aaaaf17-host-slash\") pod \"ovnkube-node-86758\" (UID: \"d43a4e16-3834-4d31-bed4-42fc0aaaaf17\") " pod="openshift-ovn-kubernetes/ovnkube-node-86758"
Oct 31 12:27:18.234841 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.234698    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d43a4e16-3834-4d31-bed4-42fc0aaaaf17-host-cni-netd\") pod \"ovnkube-node-86758\" (UID: \"d43a4e16-3834-4d31-bed4-42fc0aaaaf17\") " pod="openshift-ovn-kubernetes/ovnkube-node-86758"
Oct 31 12:27:18.234841 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.234702    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6b3e77ff-4445-42fa-a7c3-f35d442c276d-host-var-lib-kubelet\") pod \"multus-ccps4\" (UID: \"6b3e77ff-4445-42fa-a7c3-f35d442c276d\") " pod="openshift-multus/multus-ccps4"
Oct 31 12:27:18.234841 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:27:18.234734    5060 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 31 12:27:18.234841 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.234751    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a2b450be-ca79-4edf-a1ea-377531a49519-dbus\") pod \"global-pull-secret-syncer-2bmzr\" (UID: \"a2b450be-ca79-4edf-a1ea-377531a49519\") " pod="kube-system/global-pull-secret-syncer-2bmzr"
Oct 31 12:27:18.234841 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.234754    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a2b450be-ca79-4edf-a1ea-377531a49519-kubelet-config\") pod \"global-pull-secret-syncer-2bmzr\" (UID: \"a2b450be-ca79-4edf-a1ea-377531a49519\") " pod="kube-system/global-pull-secret-syncer-2bmzr"
Oct 31 12:27:18.234841 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.234772    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6b3e77ff-4445-42fa-a7c3-f35d442c276d-host-var-lib-cni-bin\") pod \"multus-ccps4\" (UID: \"6b3e77ff-4445-42fa-a7c3-f35d442c276d\") " pod="openshift-multus/multus-ccps4"
Oct 31 12:27:18.234841 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.234799    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6b3e77ff-4445-42fa-a7c3-f35d442c276d-host-var-lib-kubelet\") pod \"multus-ccps4\" (UID: \"6b3e77ff-4445-42fa-a7c3-f35d442c276d\") " pod="openshift-multus/multus-ccps4"
Oct 31 12:27:18.234841 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:27:18.234815    5060 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/556dda7b-d8e6-4558-b5f3-994b01d6a866-metrics-certs podName:556dda7b-d8e6-4558-b5f3-994b01d6a866 nodeName:}" failed. No retries permitted until 2026-10-31 12:27:18.734794897 +0000 UTC m=+1.837862601 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/556dda7b-d8e6-4558-b5f3-994b01d6a866-metrics-certs") pod "router-default-659589798c-776bx" (UID: "556dda7b-d8e6-4558-b5f3-994b01d6a866") : secret "router-metrics-certs-default" not found
Oct 31 12:27:18.234841 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.234805    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6b3e77ff-4445-42fa-a7c3-f35d442c276d-host-var-lib-cni-bin\") pod \"multus-ccps4\" (UID: \"6b3e77ff-4445-42fa-a7c3-f35d442c276d\") " pod="openshift-multus/multus-ccps4"
Oct 31 12:27:18.235601 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.234887    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6b3e77ff-4445-42fa-a7c3-f35d442c276d-host-run-multus-certs\") pod \"multus-ccps4\" (UID: \"6b3e77ff-4445-42fa-a7c3-f35d442c276d\") " pod="openshift-multus/multus-ccps4"
Oct 31 12:27:18.235601 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.234929    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6b3e77ff-4445-42fa-a7c3-f35d442c276d-host-run-k8s-cni-cncf-io\") pod \"multus-ccps4\" (UID: \"6b3e77ff-4445-42fa-a7c3-f35d442c276d\") " pod="openshift-multus/multus-ccps4"
Oct 31 12:27:18.235601 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.234955    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a1e578b0-2e77-4f52-8a44-379b6059c3d9-socket-dir\") pod \"aws-ebs-csi-driver-node-jmlvc\" (UID: \"a1e578b0-2e77-4f52-8a44-379b6059c3d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jmlvc"
Oct 31 12:27:18.235601 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.234980    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9083c35a-9739-4f3e-8415-bad1e6433d40-registry-tls\") pod \"image-registry-6cfdbfcdc4-2rq84\" (UID: \"9083c35a-9739-4f3e-8415-bad1e6433d40\") " pod="openshift-image-registry/image-registry-6cfdbfcdc4-2rq84"
Oct 31 12:27:18.235601 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.234992    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6b3e77ff-4445-42fa-a7c3-f35d442c276d-host-run-k8s-cni-cncf-io\") pod \"multus-ccps4\" (UID: \"6b3e77ff-4445-42fa-a7c3-f35d442c276d\") " pod="openshift-multus/multus-ccps4"
Oct 31 12:27:18.235601 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.234957    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6b3e77ff-4445-42fa-a7c3-f35d442c276d-host-run-multus-certs\") pod \"multus-ccps4\" (UID: \"6b3e77ff-4445-42fa-a7c3-f35d442c276d\") " pod="openshift-multus/multus-ccps4"
Oct 31 12:27:18.235601 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.235005    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee-etc-sysconfig\") pod \"tuned-qkt2l\" (UID: \"51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee\") " pod="openshift-cluster-node-tuning-operator/tuned-qkt2l"
Oct 31 12:27:18.235601 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.234997    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a1e578b0-2e77-4f52-8a44-379b6059c3d9-socket-dir\") pod \"aws-ebs-csi-driver-node-jmlvc\" (UID: \"a1e578b0-2e77-4f52-8a44-379b6059c3d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jmlvc"
Oct 31 12:27:18.235601 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.235037    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/50ad7816-358d-461e-a9bd-fadce6246e8f-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-2bfvp\" (UID: \"50ad7816-358d-461e-a9bd-fadce6246e8f\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-2bfvp"
Oct 31 12:27:18.235601 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.235065    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0db116b4-d971-4966-bb05-db28d077b6fe-registry-tls\") pod \"image-registry-6dd647c7b7-9255m\" (UID: \"0db116b4-d971-4966-bb05-db28d077b6fe\") " pod="openshift-image-registry/image-registry-6dd647c7b7-9255m"
Oct 31 12:27:18.235601 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:27:18.235082    5060 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 31 12:27:18.235601 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.235089    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee-etc-sysctl-conf\") pod \"tuned-qkt2l\" (UID: \"51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee\") " pod="openshift-cluster-node-tuning-operator/tuned-qkt2l"
Oct 31 12:27:18.235601 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:27:18.235097    5060 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6cfdbfcdc4-2rq84: secret "image-registry-tls" not found
Oct 31 12:27:18.235601 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.235098    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee-etc-sysconfig\") pod \"tuned-qkt2l\" (UID: \"51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee\") " pod="openshift-cluster-node-tuning-operator/tuned-qkt2l"
Oct 31 12:27:18.235601 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.235118    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d43a4e16-3834-4d31-bed4-42fc0aaaaf17-var-lib-openvswitch\") pod \"ovnkube-node-86758\" (UID: \"d43a4e16-3834-4d31-bed4-42fc0aaaaf17\") " pod="openshift-ovn-kubernetes/ovnkube-node-86758"
Oct 31 12:27:18.235601 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.235146    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d43a4e16-3834-4d31-bed4-42fc0aaaaf17-host-kubelet\") pod \"ovnkube-node-86758\" (UID: \"d43a4e16-3834-4d31-bed4-42fc0aaaaf17\") " pod="openshift-ovn-kubernetes/ovnkube-node-86758"
Oct 31 12:27:18.235601 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:27:18.235160    5060 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9083c35a-9739-4f3e-8415-bad1e6433d40-registry-tls podName:9083c35a-9739-4f3e-8415-bad1e6433d40 nodeName:}" failed. No retries permitted until 2026-10-31 12:27:18.735143775 +0000 UTC m=+1.838211480 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9083c35a-9739-4f3e-8415-bad1e6433d40-registry-tls") pod "image-registry-6cfdbfcdc4-2rq84" (UID: "9083c35a-9739-4f3e-8415-bad1e6433d40") : secret "image-registry-tls" not found
Oct 31 12:27:18.235601 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:27:18.235164    5060 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 31 12:27:18.235601 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:27:18.235179    5060 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6dd647c7b7-9255m: secret "image-registry-tls" not found
Oct 31 12:27:18.235601 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:27:18.235183    5060 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 31 12:27:18.235601 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.235186    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d43a4e16-3834-4d31-bed4-42fc0aaaaf17-etc-openvswitch\") pod \"ovnkube-node-86758\" (UID: \"d43a4e16-3834-4d31-bed4-42fc0aaaaf17\") " pod="openshift-ovn-kubernetes/ovnkube-node-86758"
Oct 31 12:27:18.235601 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.235185    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9557a3f0-ae0f-41f9-a9b5-15aeda410a38-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p7phb\" (UID: \"9557a3f0-ae0f-41f9-a9b5-15aeda410a38\") " pod="openshift-insights/insights-runtime-extractor-p7phb"
Oct 31 12:27:18.235601 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.235208    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6b3e77ff-4445-42fa-a7c3-f35d442c276d-host-var-lib-cni-multus\") pod \"multus-ccps4\" (UID: \"6b3e77ff-4445-42fa-a7c3-f35d442c276d\") " pod="openshift-multus/multus-ccps4"
Oct 31 12:27:18.235601 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.235186    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d43a4e16-3834-4d31-bed4-42fc0aaaaf17-var-lib-openvswitch\") pod \"ovnkube-node-86758\" (UID: \"d43a4e16-3834-4d31-bed4-42fc0aaaaf17\") " pod="openshift-ovn-kubernetes/ovnkube-node-86758"
Oct 31 12:27:18.235601 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:27:18.235227    5060 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50ad7816-358d-461e-a9bd-fadce6246e8f-networking-console-plugin-cert podName:50ad7816-358d-461e-a9bd-fadce6246e8f nodeName:}" failed. No retries permitted until 2026-10-31 12:27:18.735213051 +0000 UTC m=+1.838280761 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/50ad7816-358d-461e-a9bd-fadce6246e8f-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-2bfvp" (UID: "50ad7816-358d-461e-a9bd-fadce6246e8f") : secret "networking-console-plugin-cert" not found
Oct 31 12:27:18.235601 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.235251    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d43a4e16-3834-4d31-bed4-42fc0aaaaf17-etc-openvswitch\") pod \"ovnkube-node-86758\" (UID: \"d43a4e16-3834-4d31-bed4-42fc0aaaaf17\") " pod="openshift-ovn-kubernetes/ovnkube-node-86758"
Oct 31 12:27:18.235601 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:27:18.235265    5060 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0db116b4-d971-4966-bb05-db28d077b6fe-registry-tls podName:0db116b4-d971-4966-bb05-db28d077b6fe nodeName:}" failed. No retries permitted until 2026-10-31 12:27:18.735252468 +0000 UTC m=+1.838320164 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0db116b4-d971-4966-bb05-db28d077b6fe-registry-tls") pod "image-registry-6dd647c7b7-9255m" (UID: "0db116b4-d971-4966-bb05-db28d077b6fe") : secret "image-registry-tls" not found
Oct 31 12:27:18.235601 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.235259    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d43a4e16-3834-4d31-bed4-42fc0aaaaf17-host-kubelet\") pod \"ovnkube-node-86758\" (UID: \"d43a4e16-3834-4d31-bed4-42fc0aaaaf17\") " pod="openshift-ovn-kubernetes/ovnkube-node-86758"
Oct 31 12:27:18.235601 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.235240    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6b3e77ff-4445-42fa-a7c3-f35d442c276d-host-var-lib-cni-multus\") pod \"multus-ccps4\" (UID: \"6b3e77ff-4445-42fa-a7c3-f35d442c276d\") " pod="openshift-multus/multus-ccps4"
Oct 31 12:27:18.235601 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.235288    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee-etc-sysctl-conf\") pod \"tuned-qkt2l\" (UID: \"51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee\") " pod="openshift-cluster-node-tuning-operator/tuned-qkt2l"
Oct 31 12:27:18.235601 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.235295    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6b3e77ff-4445-42fa-a7c3-f35d442c276d-multus-socket-dir-parent\") pod \"multus-ccps4\" (UID: \"6b3e77ff-4445-42fa-a7c3-f35d442c276d\") " pod="openshift-multus/multus-ccps4"
Oct 31 12:27:18.235601 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.235322    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6b3e77ff-4445-42fa-a7c3-f35d442c276d-hostroot\") pod \"multus-ccps4\" (UID: \"6b3e77ff-4445-42fa-a7c3-f35d442c276d\") " pod="openshift-multus/multus-ccps4"
Oct 31 12:27:18.235601 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.235331    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6b3e77ff-4445-42fa-a7c3-f35d442c276d-multus-socket-dir-parent\") pod \"multus-ccps4\" (UID: \"6b3e77ff-4445-42fa-a7c3-f35d442c276d\") " pod="openshift-multus/multus-ccps4"
Oct 31 12:27:18.235601 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.235343    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6b3e77ff-4445-42fa-a7c3-f35d442c276d-etc-kubernetes\") pod \"multus-ccps4\" (UID: \"6b3e77ff-4445-42fa-a7c3-f35d442c276d\") " pod="openshift-multus/multus-ccps4"
Oct 31 12:27:18.235601 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.235362    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6b3e77ff-4445-42fa-a7c3-f35d442c276d-multus-cni-dir\") pod \"multus-ccps4\" (UID: \"6b3e77ff-4445-42fa-a7c3-f35d442c276d\") " pod="openshift-multus/multus-ccps4"
Oct 31 12:27:18.235601 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.235379    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee-etc-modprobe-d\") pod \"tuned-qkt2l\" (UID: \"51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee\") " pod="openshift-cluster-node-tuning-operator/tuned-qkt2l"
Oct 31 12:27:18.235601 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.235383    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6b3e77ff-4445-42fa-a7c3-f35d442c276d-etc-kubernetes\") pod \"multus-ccps4\" (UID: \"6b3e77ff-4445-42fa-a7c3-f35d442c276d\") " pod="openshift-multus/multus-ccps4"
Oct 31 12:27:18.235601 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.235397    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d43a4e16-3834-4d31-bed4-42fc0aaaaf17-host-run-ovn-kubernetes\") pod \"ovnkube-node-86758\" (UID: \"d43a4e16-3834-4d31-bed4-42fc0aaaaf17\") " pod="openshift-ovn-kubernetes/ovnkube-node-86758"
Oct 31 12:27:18.235601 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.235376    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6b3e77ff-4445-42fa-a7c3-f35d442c276d-hostroot\") pod \"multus-ccps4\" (UID: \"6b3e77ff-4445-42fa-a7c3-f35d442c276d\") " pod="openshift-multus/multus-ccps4"
Oct 31 12:27:18.235601 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.235419    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee-lib-modules\") pod \"tuned-qkt2l\" (UID: \"51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee\") " pod="openshift-cluster-node-tuning-operator/tuned-qkt2l"
Oct 31 12:27:18.235601 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.235433    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee-host\") pod \"tuned-qkt2l\" (UID: \"51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee\") " pod="openshift-cluster-node-tuning-operator/tuned-qkt2l"
Oct 31 12:27:18.235601 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.235436    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d43a4e16-3834-4d31-bed4-42fc0aaaaf17-host-run-ovn-kubernetes\") pod \"ovnkube-node-86758\" (UID: \"d43a4e16-3834-4d31-bed4-42fc0aaaaf17\") " pod="openshift-ovn-kubernetes/ovnkube-node-86758"
Oct 31 12:27:18.235601 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.235432    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6b3e77ff-4445-42fa-a7c3-f35d442c276d-multus-cni-dir\") pod \"multus-ccps4\" (UID: \"6b3e77ff-4445-42fa-a7c3-f35d442c276d\") " pod="openshift-multus/multus-ccps4"
Oct 31 12:27:18.235601 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.235455    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6b3e77ff-4445-42fa-a7c3-f35d442c276d-host-run-netns\") pod \"multus-ccps4\" (UID: \"6b3e77ff-4445-42fa-a7c3-f35d442c276d\") " pod="openshift-multus/multus-ccps4"
Oct 31 12:27:18.235601 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.235480    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/dbbd2d70-8911-4aa5-96cf-3a16dcdf246d-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-tcnrw\" (UID: \"dbbd2d70-8911-4aa5-96cf-3a16dcdf246d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-tcnrw"
Oct 31 12:27:18.235601 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.235508    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee-etc-modprobe-d\") pod \"tuned-qkt2l\" (UID: \"51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee\") " pod="openshift-cluster-node-tuning-operator/tuned-qkt2l"
Oct 31 12:27:18.235601 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.235528    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6b3e77ff-4445-42fa-a7c3-f35d442c276d-host-run-netns\") pod \"multus-ccps4\" (UID: \"6b3e77ff-4445-42fa-a7c3-f35d442c276d\") " pod="openshift-multus/multus-ccps4"
Oct 31 12:27:18.235601 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.235482    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee-host\") pod \"tuned-qkt2l\" (UID: \"51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee\") " pod="openshift-cluster-node-tuning-operator/tuned-qkt2l"
Oct 31 12:27:18.235601 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.235531    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a1e578b0-2e77-4f52-8a44-379b6059c3d9-device-dir\") pod \"aws-ebs-csi-driver-node-jmlvc\" (UID: \"a1e578b0-2e77-4f52-8a44-379b6059c3d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jmlvc"
Oct 31 12:27:18.235601 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.235545    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee-lib-modules\") pod \"tuned-qkt2l\" (UID: \"51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee\") " pod="openshift-cluster-node-tuning-operator/tuned-qkt2l"
Oct 31 12:27:18.235601 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.235567    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee-etc-sysctl-d\") pod \"tuned-qkt2l\" (UID: \"51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee\") " pod="openshift-cluster-node-tuning-operator/tuned-qkt2l"
Oct 31 12:27:18.235601 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.235584    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee-sys\") pod \"tuned-qkt2l\" (UID: \"51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee\") " pod="openshift-cluster-node-tuning-operator/tuned-qkt2l"
Oct 31 12:27:18.235601 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:27:18.235592    5060 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 31 12:27:18.235601 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.235600    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a1c3175b-e32e-420e-877f-71bfe64d9c6c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-44zj2\" (UID: \"a1c3175b-e32e-420e-877f-71bfe64d9c6c\") " pod="openshift-multus/multus-additional-cni-plugins-44zj2"
Oct 31 12:27:18.235601 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:27:18.235643    5060 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dbbd2d70-8911-4aa5-96cf-3a16dcdf246d-samples-operator-tls podName:dbbd2d70-8911-4aa5-96cf-3a16dcdf246d nodeName:}" failed. No retries permitted until 2026-10-31 12:27:18.735630294 +0000 UTC m=+1.838697997 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/dbbd2d70-8911-4aa5-96cf-3a16dcdf246d-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-tcnrw" (UID: "dbbd2d70-8911-4aa5-96cf-3a16dcdf246d") : secret "samples-operator-tls" not found
Oct 31 12:27:18.235601 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.235647    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a1e578b0-2e77-4f52-8a44-379b6059c3d9-device-dir\") pod \"aws-ebs-csi-driver-node-jmlvc\" (UID: \"a1e578b0-2e77-4f52-8a44-379b6059c3d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jmlvc"
Oct 31 12:27:18.237483 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.235656    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee-sys\") pod \"tuned-qkt2l\" (UID: \"51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee\") " pod="openshift-cluster-node-tuning-operator/tuned-qkt2l"
Oct 31 12:27:18.237483 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.235670    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a1e578b0-2e77-4f52-8a44-379b6059c3d9-registration-dir\") pod \"aws-ebs-csi-driver-node-jmlvc\" (UID: \"a1e578b0-2e77-4f52-8a44-379b6059c3d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jmlvc"
Oct 31 12:27:18.237483 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.235678    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a1c3175b-e32e-420e-877f-71bfe64d9c6c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-44zj2\" (UID: \"a1c3175b-e32e-420e-877f-71bfe64d9c6c\") " pod="openshift-multus/multus-additional-cni-plugins-44zj2"
Oct 31 12:27:18.237483 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.235695    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3bb2c04e-55a2-4206-9530-7298f4469379-host-slash\") pod \"iptables-alerter-plbnz\" (UID: \"3bb2c04e-55a2-4206-9530-7298f4469379\") " pod="openshift-network-operator/iptables-alerter-plbnz"
Oct 31 12:27:18.237483 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.235701    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee-etc-sysctl-d\") pod \"tuned-qkt2l\" (UID: \"51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee\") " pod="openshift-cluster-node-tuning-operator/tuned-qkt2l"
Oct 31 12:27:18.237483 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.235733    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a1e578b0-2e77-4f52-8a44-379b6059c3d9-registration-dir\") pod \"aws-ebs-csi-driver-node-jmlvc\" (UID: \"a1e578b0-2e77-4f52-8a44-379b6059c3d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jmlvc"
Oct 31 12:27:18.237483 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.235739    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3bb2c04e-55a2-4206-9530-7298f4469379-host-slash\") pod \"iptables-alerter-plbnz\" (UID: \"3bb2c04e-55a2-4206-9530-7298f4469379\") " pod="openshift-network-operator/iptables-alerter-plbnz"
Oct 31 12:27:18.237483 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.235732    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9557a3f0-ae0f-41f9-a9b5-15aeda410a38-crio-socket\") pod \"insights-runtime-extractor-p7phb\" (UID: \"9557a3f0-ae0f-41f9-a9b5-15aeda410a38\") " pod="openshift-insights/insights-runtime-extractor-p7phb"
Oct 31 12:27:18.237483 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.235766    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9557a3f0-ae0f-41f9-a9b5-15aeda410a38-crio-socket\") pod \"insights-runtime-extractor-p7phb\" (UID: \"9557a3f0-ae0f-41f9-a9b5-15aeda410a38\") " pod="openshift-insights/insights-runtime-extractor-p7phb"
Oct 31 12:27:18.237483 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.235768    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee-run\") pod \"tuned-qkt2l\" (UID: \"51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee\") " pod="openshift-cluster-node-tuning-operator/tuned-qkt2l"
Oct 31 12:27:18.237483 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.235788    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d43a4e16-3834-4d31-bed4-42fc0aaaaf17-log-socket\") pod \"ovnkube-node-86758\" (UID: \"d43a4e16-3834-4d31-bed4-42fc0aaaaf17\") " pod="openshift-ovn-kubernetes/ovnkube-node-86758"
Oct 31 12:27:18.237483 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.235790    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee-run\") pod \"tuned-qkt2l\" (UID: \"51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee\") " pod="openshift-cluster-node-tuning-operator/tuned-qkt2l"
Oct 31 12:27:18.237483 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.235813    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5c8865f9-fdf6-49b9-8b4f-69dbc7787084-metrics-tls\") pod \"dns-default-r6fvt\" (UID: \"5c8865f9-fdf6-49b9-8b4f-69dbc7787084\") " pod="openshift-dns/dns-default-r6fvt"
Oct 31 12:27:18.237483 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.235829    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee-etc-kubernetes\") pod \"tuned-qkt2l\" (UID: \"51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee\") " pod="openshift-cluster-node-tuning-operator/tuned-qkt2l"
Oct 31 12:27:18.237483 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.235844    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9557a3f0-ae0f-41f9-a9b5-15aeda410a38-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p7phb\" (UID: \"9557a3f0-ae0f-41f9-a9b5-15aeda410a38\") " pod="openshift-insights/insights-runtime-extractor-p7phb"
Oct 31 12:27:18.237483 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.235866    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d43a4e16-3834-4d31-bed4-42fc0aaaaf17-run-openvswitch\") pod \"ovnkube-node-86758\" (UID: \"d43a4e16-3834-4d31-bed4-42fc0aaaaf17\") " pod="openshift-ovn-kubernetes/ovnkube-node-86758"
Oct 31 12:27:18.237483 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.235885    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6b3e77ff-4445-42fa-a7c3-f35d442c276d-multus-conf-dir\") pod \"multus-ccps4\" (UID: \"6b3e77ff-4445-42fa-a7c3-f35d442c276d\") " pod="openshift-multus/multus-ccps4"
Oct 31 12:27:18.237483 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.235891    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d43a4e16-3834-4d31-bed4-42fc0aaaaf17-log-socket\") pod \"ovnkube-node-86758\" (UID: \"d43a4e16-3834-4d31-bed4-42fc0aaaaf17\") " pod="openshift-ovn-kubernetes/ovnkube-node-86758"
Oct 31 12:27:18.237483 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.235906    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee-etc-kubernetes\") pod \"tuned-qkt2l\" (UID: \"51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee\") " pod="openshift-cluster-node-tuning-operator/tuned-qkt2l"
Oct 31 12:27:18.237483 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:27:18.235913    5060 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 31 12:27:18.237483 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.235927    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6b3e77ff-4445-42fa-a7c3-f35d442c276d-multus-conf-dir\") pod \"multus-ccps4\" (UID: \"6b3e77ff-4445-42fa-a7c3-f35d442c276d\") " pod="openshift-multus/multus-ccps4"
Oct 31 12:27:18.237483 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.235938    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d43a4e16-3834-4d31-bed4-42fc0aaaaf17-run-openvswitch\") pod \"ovnkube-node-86758\" (UID: \"d43a4e16-3834-4d31-bed4-42fc0aaaaf17\") " pod="openshift-ovn-kubernetes/ovnkube-node-86758"
Oct 31 12:27:18.237483 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:27:18.235948    5060 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 31 12:27:18.237483 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:27:18.235958    5060 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5c8865f9-fdf6-49b9-8b4f-69dbc7787084-metrics-tls podName:5c8865f9-fdf6-49b9-8b4f-69dbc7787084 nodeName:}" failed. No retries permitted until 2026-10-31 12:27:18.735945108 +0000 UTC m=+1.839012797 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5c8865f9-fdf6-49b9-8b4f-69dbc7787084-metrics-tls") pod "dns-default-r6fvt" (UID: "5c8865f9-fdf6-49b9-8b4f-69dbc7787084") : secret "dns-default-metrics-tls" not found
Oct 31 12:27:18.237483 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:27:18.235985    5060 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9557a3f0-ae0f-41f9-a9b5-15aeda410a38-insights-runtime-extractor-tls podName:9557a3f0-ae0f-41f9-a9b5-15aeda410a38 nodeName:}" failed. No retries permitted until 2026-10-31 12:27:18.735971731 +0000 UTC m=+1.839039440 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9557a3f0-ae0f-41f9-a9b5-15aeda410a38-insights-runtime-extractor-tls") pod "insights-runtime-extractor-p7phb" (UID: "9557a3f0-ae0f-41f9-a9b5-15aeda410a38") : secret "insights-runtime-extractor-tls" not found
Oct 31 12:27:18.237483 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.235982    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a1c3175b-e32e-420e-877f-71bfe64d9c6c-system-cni-dir\") pod \"multus-additional-cni-plugins-44zj2\" (UID: \"a1c3175b-e32e-420e-877f-71bfe64d9c6c\") " pod="openshift-multus/multus-additional-cni-plugins-44zj2"
Oct 31 12:27:18.237483 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.236008    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a1e578b0-2e77-4f52-8a44-379b6059c3d9-etc-selinux\") pod \"aws-ebs-csi-driver-node-jmlvc\" (UID: \"a1e578b0-2e77-4f52-8a44-379b6059c3d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jmlvc"
Oct 31 12:27:18.237483 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.236016    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a1c3175b-e32e-420e-877f-71bfe64d9c6c-system-cni-dir\") pod \"multus-additional-cni-plugins-44zj2\" (UID: \"a1c3175b-e32e-420e-877f-71bfe64d9c6c\") " pod="openshift-multus/multus-additional-cni-plugins-44zj2"
Oct 31 12:27:18.237483 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.236024    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/556dda7b-d8e6-4558-b5f3-994b01d6a866-service-ca-bundle\") pod \"router-default-659589798c-776bx\" (UID: \"556dda7b-d8e6-4558-b5f3-994b01d6a866\") " pod="openshift-ingress/router-default-659589798c-776bx"
Oct 31 12:27:18.237483 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.236060    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d43a4e16-3834-4d31-bed4-42fc0aaaaf17-run-systemd\") pod \"ovnkube-node-86758\" (UID: \"d43a4e16-3834-4d31-bed4-42fc0aaaaf17\") " pod="openshift-ovn-kubernetes/ovnkube-node-86758"
Oct 31 12:27:18.237483 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.236081    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a1e578b0-2e77-4f52-8a44-379b6059c3d9-etc-selinux\") pod \"aws-ebs-csi-driver-node-jmlvc\" (UID: \"a1e578b0-2e77-4f52-8a44-379b6059c3d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jmlvc"
Oct 31 12:27:18.237483 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.236087    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee-var-lib-kubelet\") pod \"tuned-qkt2l\" (UID: \"51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee\") " pod="openshift-cluster-node-tuning-operator/tuned-qkt2l"
Oct 31 12:27:18.237483 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:27:18.236093    5060 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/556dda7b-d8e6-4558-b5f3-994b01d6a866-service-ca-bundle podName:556dda7b-d8e6-4558-b5f3-994b01d6a866 nodeName:}" failed. No retries permitted until 2026-10-31 12:27:18.736081245 +0000 UTC m=+1.839148961 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/556dda7b-d8e6-4558-b5f3-994b01d6a866-service-ca-bundle") pod "router-default-659589798c-776bx" (UID: "556dda7b-d8e6-4558-b5f3-994b01d6a866") : configmap references non-existent config key: service-ca.crt
Oct 31 12:27:18.237483 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.236120    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d43a4e16-3834-4d31-bed4-42fc0aaaaf17-run-ovn\") pod \"ovnkube-node-86758\" (UID: \"d43a4e16-3834-4d31-bed4-42fc0aaaaf17\") " pod="openshift-ovn-kubernetes/ovnkube-node-86758"
Oct 31 12:27:18.237483 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.236124    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d43a4e16-3834-4d31-bed4-42fc0aaaaf17-run-systemd\") pod \"ovnkube-node-86758\" (UID: \"d43a4e16-3834-4d31-bed4-42fc0aaaaf17\") " pod="openshift-ovn-kubernetes/ovnkube-node-86758"
Oct 31 12:27:18.237483 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.236137    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee-var-lib-kubelet\") pod \"tuned-qkt2l\" (UID: \"51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee\") " pod="openshift-cluster-node-tuning-operator/tuned-qkt2l"
Oct 31 12:27:18.237483 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.236143    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6bbe13aa-df4b-40b0-8595-68fe1166a082-hosts-file\") pod \"node-resolver-v9ks4\" (UID: \"6bbe13aa-df4b-40b0-8595-68fe1166a082\") " pod="openshift-dns/node-resolver-v9ks4"
Oct 31 12:27:18.237483 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.236162    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d43a4e16-3834-4d31-bed4-42fc0aaaaf17-run-ovn\") pod \"ovnkube-node-86758\" (UID: \"d43a4e16-3834-4d31-bed4-42fc0aaaaf17\") " pod="openshift-ovn-kubernetes/ovnkube-node-86758"
Oct 31 12:27:18.237483 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.236170    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e64bd69d-946f-42ad-94ae-b87b7de3a581-metrics-certs\") pod \"network-metrics-daemon-94kh9\" (UID: \"e64bd69d-946f-42ad-94ae-b87b7de3a581\") " pod="openshift-multus/network-metrics-daemon-94kh9"
Oct 31 12:27:18.237483 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.236194    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d43a4e16-3834-4d31-bed4-42fc0aaaaf17-node-log\") pod \"ovnkube-node-86758\" (UID: \"d43a4e16-3834-4d31-bed4-42fc0aaaaf17\") " pod="openshift-ovn-kubernetes/ovnkube-node-86758"
Oct 31 12:27:18.237483 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.236217    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6bbe13aa-df4b-40b0-8595-68fe1166a082-hosts-file\") pod \"node-resolver-v9ks4\" (UID: \"6bbe13aa-df4b-40b0-8595-68fe1166a082\") " pod="openshift-dns/node-resolver-v9ks4"
Oct 31 12:27:18.237483 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.236217    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a1e578b0-2e77-4f52-8a44-379b6059c3d9-sys-fs\") pod \"aws-ebs-csi-driver-node-jmlvc\" (UID: \"a1e578b0-2e77-4f52-8a44-379b6059c3d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jmlvc"
Oct 31 12:27:18.237483 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.236253    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d43a4e16-3834-4d31-bed4-42fc0aaaaf17-host-run-netns\") pod \"ovnkube-node-86758\" (UID: \"d43a4e16-3834-4d31-bed4-42fc0aaaaf17\") " pod="openshift-ovn-kubernetes/ovnkube-node-86758"
Oct 31 12:27:18.237483 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.236261    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a1e578b0-2e77-4f52-8a44-379b6059c3d9-sys-fs\") pod \"aws-ebs-csi-driver-node-jmlvc\" (UID: \"a1e578b0-2e77-4f52-8a44-379b6059c3d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jmlvc"
Oct 31 12:27:18.237483 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.236269    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a1c3175b-e32e-420e-877f-71bfe64d9c6c-cnibin\") pod \"multus-additional-cni-plugins-44zj2\" (UID: \"a1c3175b-e32e-420e-877f-71bfe64d9c6c\") " pod="openshift-multus/multus-additional-cni-plugins-44zj2"
Oct 31 12:27:18.237483 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.236286    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6b3e77ff-4445-42fa-a7c3-f35d442c276d-cnibin\") pod \"multus-ccps4\" (UID: \"6b3e77ff-4445-42fa-a7c3-f35d442c276d\") " pod="openshift-multus/multus-ccps4"
Oct 31 12:27:18.237483 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.236300    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d43a4e16-3834-4d31-bed4-42fc0aaaaf17-host-run-netns\") pod \"ovnkube-node-86758\" (UID: \"d43a4e16-3834-4d31-bed4-42fc0aaaaf17\") " pod="openshift-ovn-kubernetes/ovnkube-node-86758"
Oct 31 12:27:18.237483 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.236300    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d43a4e16-3834-4d31-bed4-42fc0aaaaf17-host-cni-bin\") pod \"ovnkube-node-86758\" (UID: \"d43a4e16-3834-4d31-bed4-42fc0aaaaf17\") " pod="openshift-ovn-kubernetes/ovnkube-node-86758"
Oct 31 12:27:18.237483 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.236252    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d43a4e16-3834-4d31-bed4-42fc0aaaaf17-node-log\") pod \"ovnkube-node-86758\" (UID: \"d43a4e16-3834-4d31-bed4-42fc0aaaaf17\") " pod="openshift-ovn-kubernetes/ovnkube-node-86758"
Oct 31 12:27:18.237483 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.236327    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d43a4e16-3834-4d31-bed4-42fc0aaaaf17-host-cni-bin\") pod \"ovnkube-node-86758\" (UID: \"d43a4e16-3834-4d31-bed4-42fc0aaaaf17\") " pod="openshift-ovn-kubernetes/ovnkube-node-86758"
Oct 31 12:27:18.237483 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.236331    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a1e578b0-2e77-4f52-8a44-379b6059c3d9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jmlvc\" (UID: \"a1e578b0-2e77-4f52-8a44-379b6059c3d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jmlvc"
Oct 31 12:27:18.237483 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.236352    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a1c3175b-e32e-420e-877f-71bfe64d9c6c-cnibin\") pod \"multus-additional-cni-plugins-44zj2\" (UID: \"a1c3175b-e32e-420e-877f-71bfe64d9c6c\") " pod="openshift-multus/multus-additional-cni-plugins-44zj2"
Oct 31 12:27:18.237483 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.236359    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee-etc-systemd\") pod \"tuned-qkt2l\" (UID: \"51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee\") " pod="openshift-cluster-node-tuning-operator/tuned-qkt2l"
Oct 31 12:27:18.237483 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.236368    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6b3e77ff-4445-42fa-a7c3-f35d442c276d-cnibin\") pod \"multus-ccps4\" (UID: \"6b3e77ff-4445-42fa-a7c3-f35d442c276d\") " pod="openshift-multus/multus-ccps4"
Oct 31 12:27:18.237483 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.236393    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6c661ce8-9f7b-4509-bd17-2380bcd1c425-host\") pod \"node-ca-wdkzp\" (UID: \"6c661ce8-9f7b-4509-bd17-2380bcd1c425\") " pod="openshift-image-registry/node-ca-wdkzp"
Oct 31 12:27:18.237483 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.236397    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee-etc-systemd\") pod \"tuned-qkt2l\" (UID: \"51a5b8c5-e56a-4c0c-b270-b6a0ec0db9ee\") " pod="openshift-cluster-node-tuning-operator/tuned-qkt2l"
Oct 31 12:27:18.237483 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.236413    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a1e578b0-2e77-4f52-8a44-379b6059c3d9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jmlvc\" (UID: \"a1e578b0-2e77-4f52-8a44-379b6059c3d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jmlvc"
Oct 31 12:27:18.237483 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.236428    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d43a4e16-3834-4d31-bed4-42fc0aaaaf17-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-86758\" (UID: \"d43a4e16-3834-4d31-bed4-42fc0aaaaf17\") " pod="openshift-ovn-kubernetes/ovnkube-node-86758"
Oct 31 12:27:18.237483 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.236457    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6b3e77ff-4445-42fa-a7c3-f35d442c276d-system-cni-dir\") pod \"multus-ccps4\" (UID: \"6b3e77ff-4445-42fa-a7c3-f35d442c276d\") " pod="openshift-multus/multus-ccps4"
Oct 31 12:27:18.237483 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.236432    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6c661ce8-9f7b-4509-bd17-2380bcd1c425-host\") pod \"node-ca-wdkzp\" (UID: \"6c661ce8-9f7b-4509-bd17-2380bcd1c425\") " pod="openshift-image-registry/node-ca-wdkzp"
Oct 31 12:27:18.237483 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.236515    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d43a4e16-3834-4d31-bed4-42fc0aaaaf17-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-86758\" (UID: \"d43a4e16-3834-4d31-bed4-42fc0aaaaf17\") " pod="openshift-ovn-kubernetes/ovnkube-node-86758"
Oct 31 12:27:18.237483 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.236552    5060 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 31 12:27:18.237483 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.236581    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6b3e77ff-4445-42fa-a7c3-f35d442c276d-system-cni-dir\") pod \"multus-ccps4\" (UID: \"6b3e77ff-4445-42fa-a7c3-f35d442c276d\") " pod="openshift-multus/multus-ccps4"
Oct 31 12:27:18.239212 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.238514    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e64bd69d-946f-42ad-94ae-b87b7de3a581-metrics-certs\") pod \"network-metrics-daemon-94kh9\" (UID: \"e64bd69d-946f-42ad-94ae-b87b7de3a581\") " pod="openshift-multus/network-metrics-daemon-94kh9"
Oct 31 12:27:18.310113 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.310083    5060 scope.go:122] "RemoveContainer" containerID="c98998f609d234b42991ed336e4bc8d348b0169b157e725e00a2b4bc0c1a44a4"
Oct 31 12:27:18.315146 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.315123    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-94kh9"
Oct 31 12:27:18.617260 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.617181    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-94kh9"]
Oct 31 12:27:18.622157 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:18.622127    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode64bd69d_946f_42ad_94ae_b87b7de3a581.slice/crio-e09133b38c10e1f32fe4a97d2a509c2f5fa146a28e874fde531fbdbcc39164bb WatchSource:0}: Error finding container e09133b38c10e1f32fe4a97d2a509c2f5fa146a28e874fde531fbdbcc39164bb: Status 404 returned error can't find the container with id e09133b38c10e1f32fe4a97d2a509c2f5fa146a28e874fde531fbdbcc39164bb
Oct 31 12:27:18.624351 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.624330    5060 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 12:27:18.639000 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.638973    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/dda0c26f-8225-4239-bc27-866b66cbb919-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-gt9rn\" (UID: \"dda0c26f-8225-4239-bc27-866b66cbb919\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gt9rn"
Oct 31 12:27:18.639132 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:27:18.639113    5060 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 31 12:27:18.639185 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:27:18.639173    5060 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dda0c26f-8225-4239-bc27-866b66cbb919-cluster-monitoring-operator-tls podName:dda0c26f-8225-4239-bc27-866b66cbb919 nodeName:}" failed. No retries permitted until 2026-10-31 12:27:19.639159514 +0000 UTC m=+2.742227221 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/dda0c26f-8225-4239-bc27-866b66cbb919-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-gt9rn" (UID: "dda0c26f-8225-4239-bc27-866b66cbb919") : secret "cluster-monitoring-operator-tls" not found
Oct 31 12:27:18.740197 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.740157    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9083c35a-9739-4f3e-8415-bad1e6433d40-registry-tls\") pod \"image-registry-6cfdbfcdc4-2rq84\" (UID: \"9083c35a-9739-4f3e-8415-bad1e6433d40\") " pod="openshift-image-registry/image-registry-6cfdbfcdc4-2rq84"
Oct 31 12:27:18.740370 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.740205    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/50ad7816-358d-461e-a9bd-fadce6246e8f-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-2bfvp\" (UID: \"50ad7816-358d-461e-a9bd-fadce6246e8f\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-2bfvp"
Oct 31 12:27:18.740370 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.740230    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0db116b4-d971-4966-bb05-db28d077b6fe-registry-tls\") pod \"image-registry-6dd647c7b7-9255m\" (UID: \"0db116b4-d971-4966-bb05-db28d077b6fe\") " pod="openshift-image-registry/image-registry-6dd647c7b7-9255m"
Oct 31 12:27:18.740370 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.740280    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/dbbd2d70-8911-4aa5-96cf-3a16dcdf246d-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-tcnrw\" (UID: \"dbbd2d70-8911-4aa5-96cf-3a16dcdf246d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-tcnrw"
Oct 31 12:27:18.740370 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:27:18.740311    5060 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 31 12:27:18.740370 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.740322    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5c8865f9-fdf6-49b9-8b4f-69dbc7787084-metrics-tls\") pod \"dns-default-r6fvt\" (UID: \"5c8865f9-fdf6-49b9-8b4f-69dbc7787084\") " pod="openshift-dns/dns-default-r6fvt"
Oct 31 12:27:18.740676 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:27:18.740339    5060 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6cfdbfcdc4-2rq84: secret "image-registry-tls" not found
Oct 31 12:27:18.740676 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.740380    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9557a3f0-ae0f-41f9-a9b5-15aeda410a38-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p7phb\" (UID: \"9557a3f0-ae0f-41f9-a9b5-15aeda410a38\") " pod="openshift-insights/insights-runtime-extractor-p7phb"
Oct 31 12:27:18.740676 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:27:18.740399    5060 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 31 12:27:18.740676 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:27:18.740409    5060 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 31 12:27:18.740676 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.740439    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/556dda7b-d8e6-4558-b5f3-994b01d6a866-service-ca-bundle\") pod \"router-default-659589798c-776bx\" (UID: \"556dda7b-d8e6-4558-b5f3-994b01d6a866\") " pod="openshift-ingress/router-default-659589798c-776bx"
Oct 31 12:27:18.740676 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:27:18.740322    5060 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 31 12:27:18.740676 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:27:18.740371    5060 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 31 12:27:18.740676 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:27:18.740523    5060 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6dd647c7b7-9255m: secret "image-registry-tls" not found
Oct 31 12:27:18.740676 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:27:18.740451    5060 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5c8865f9-fdf6-49b9-8b4f-69dbc7787084-metrics-tls podName:5c8865f9-fdf6-49b9-8b4f-69dbc7787084 nodeName:}" failed. No retries permitted until 2026-10-31 12:27:19.740433658 +0000 UTC m=+2.843501365 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5c8865f9-fdf6-49b9-8b4f-69dbc7787084-metrics-tls") pod "dns-default-r6fvt" (UID: "5c8865f9-fdf6-49b9-8b4f-69dbc7787084") : secret "dns-default-metrics-tls" not found
Oct 31 12:27:18.740676 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:27:18.740510    5060 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 31 12:27:18.740676 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:27:18.740568    5060 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dbbd2d70-8911-4aa5-96cf-3a16dcdf246d-samples-operator-tls podName:dbbd2d70-8911-4aa5-96cf-3a16dcdf246d nodeName:}" failed. No retries permitted until 2026-10-31 12:27:19.740555647 +0000 UTC m=+2.843623343 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/dbbd2d70-8911-4aa5-96cf-3a16dcdf246d-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-tcnrw" (UID: "dbbd2d70-8911-4aa5-96cf-3a16dcdf246d") : secret "samples-operator-tls" not found
Oct 31 12:27:18.740676 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:27:18.740588    5060 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9083c35a-9739-4f3e-8415-bad1e6433d40-registry-tls podName:9083c35a-9739-4f3e-8415-bad1e6433d40 nodeName:}" failed. No retries permitted until 2026-10-31 12:27:19.740579523 +0000 UTC m=+2.843647216 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9083c35a-9739-4f3e-8415-bad1e6433d40-registry-tls") pod "image-registry-6cfdbfcdc4-2rq84" (UID: "9083c35a-9739-4f3e-8415-bad1e6433d40") : secret "image-registry-tls" not found
Oct 31 12:27:18.740676 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:27:18.740618    5060 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50ad7816-358d-461e-a9bd-fadce6246e8f-networking-console-plugin-cert podName:50ad7816-358d-461e-a9bd-fadce6246e8f nodeName:}" failed. No retries permitted until 2026-10-31 12:27:19.740606902 +0000 UTC m=+2.843674596 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/50ad7816-358d-461e-a9bd-fadce6246e8f-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-2bfvp" (UID: "50ad7816-358d-461e-a9bd-fadce6246e8f") : secret "networking-console-plugin-cert" not found
Oct 31 12:27:18.740676 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:27:18.740634    5060 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/556dda7b-d8e6-4558-b5f3-994b01d6a866-service-ca-bundle podName:556dda7b-d8e6-4558-b5f3-994b01d6a866 nodeName:}" failed. No retries permitted until 2026-10-31 12:27:19.740626756 +0000 UTC m=+2.843694452 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/556dda7b-d8e6-4558-b5f3-994b01d6a866-service-ca-bundle") pod "router-default-659589798c-776bx" (UID: "556dda7b-d8e6-4558-b5f3-994b01d6a866") : configmap references non-existent config key: service-ca.crt
Oct 31 12:27:18.740676 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:27:18.740651    5060 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0db116b4-d971-4966-bb05-db28d077b6fe-registry-tls podName:0db116b4-d971-4966-bb05-db28d077b6fe nodeName:}" failed. No retries permitted until 2026-10-31 12:27:19.740642701 +0000 UTC m=+2.843710397 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0db116b4-d971-4966-bb05-db28d077b6fe-registry-tls") pod "image-registry-6dd647c7b7-9255m" (UID: "0db116b4-d971-4966-bb05-db28d077b6fe") : secret "image-registry-tls" not found
Oct 31 12:27:18.740676 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:27:18.740670    5060 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9557a3f0-ae0f-41f9-a9b5-15aeda410a38-insights-runtime-extractor-tls podName:9557a3f0-ae0f-41f9-a9b5-15aeda410a38 nodeName:}" failed. No retries permitted until 2026-10-31 12:27:19.740662253 +0000 UTC m=+2.843729988 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9557a3f0-ae0f-41f9-a9b5-15aeda410a38-insights-runtime-extractor-tls") pod "insights-runtime-extractor-p7phb" (UID: "9557a3f0-ae0f-41f9-a9b5-15aeda410a38") : secret "insights-runtime-extractor-tls" not found
Oct 31 12:27:18.741222 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:18.740735    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/556dda7b-d8e6-4558-b5f3-994b01d6a866-metrics-certs\") pod \"router-default-659589798c-776bx\" (UID: \"556dda7b-d8e6-4558-b5f3-994b01d6a866\") " pod="openshift-ingress/router-default-659589798c-776bx"
Oct 31 12:27:18.741222 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:27:18.740842    5060 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 31 12:27:18.741222 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:27:18.740884    5060 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/556dda7b-d8e6-4558-b5f3-994b01d6a866-metrics-certs podName:556dda7b-d8e6-4558-b5f3-994b01d6a866 nodeName:}" failed. No retries permitted until 2026-10-31 12:27:19.740874196 +0000 UTC m=+2.843941900 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/556dda7b-d8e6-4558-b5f3-994b01d6a866-metrics-certs") pod "router-default-659589798c-776bx" (UID: "556dda7b-d8e6-4558-b5f3-994b01d6a866") : secret "router-metrics-certs-default" not found
Oct 31 12:27:19.104622 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:19.104583    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pcvvm_f032909d-bc39-4c9e-99b7-f3d800842119/console-operator/2.log"
Oct 31 12:27:19.105133 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:19.105063    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pcvvm_f032909d-bc39-4c9e-99b7-f3d800842119/console-operator/1.log"
Oct 31 12:27:19.105133 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:19.105103    5060 generic.go:356] "Generic (PLEG): container finished" podID="f032909d-bc39-4c9e-99b7-f3d800842119" containerID="2ddca170fab98ab89426aa1598ea575169dba5f01fc84f419efc709bb7a13601" exitCode=255
Oct 31 12:27:19.105232 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:19.105193    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-pcvvm" event={"ID":"f032909d-bc39-4c9e-99b7-f3d800842119","Type":"ContainerDied","Data":"2ddca170fab98ab89426aa1598ea575169dba5f01fc84f419efc709bb7a13601"}
Oct 31 12:27:19.105285 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:19.105266    5060 scope.go:122] "RemoveContainer" containerID="c98998f609d234b42991ed336e4bc8d348b0169b157e725e00a2b4bc0c1a44a4"
Oct 31 12:27:19.105640 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:19.105619    5060 scope.go:122] "RemoveContainer" containerID="2ddca170fab98ab89426aa1598ea575169dba5f01fc84f419efc709bb7a13601"
Oct 31 12:27:19.105873 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:27:19.105854    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-pcvvm_openshift-console-operator(f032909d-bc39-4c9e-99b7-f3d800842119)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-pcvvm" podUID="f032909d-bc39-4c9e-99b7-f3d800842119"
Oct 31 12:27:19.108274 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:19.108246    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-94kh9" event={"ID":"e64bd69d-946f-42ad-94ae-b87b7de3a581","Type":"ContainerStarted","Data":"e09133b38c10e1f32fe4a97d2a509c2f5fa146a28e874fde531fbdbcc39164bb"}
Oct 31 12:27:19.650776 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:19.650729    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/dda0c26f-8225-4239-bc27-866b66cbb919-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-gt9rn\" (UID: \"dda0c26f-8225-4239-bc27-866b66cbb919\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gt9rn"
Oct 31 12:27:19.650925 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:27:19.650906    5060 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 31 12:27:19.650997 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:27:19.650987    5060 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dda0c26f-8225-4239-bc27-866b66cbb919-cluster-monitoring-operator-tls podName:dda0c26f-8225-4239-bc27-866b66cbb919 nodeName:}" failed. No retries permitted until 2026-10-31 12:27:21.650966458 +0000 UTC m=+4.754034163 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/dda0c26f-8225-4239-bc27-866b66cbb919-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-gt9rn" (UID: "dda0c26f-8225-4239-bc27-866b66cbb919") : secret "cluster-monitoring-operator-tls" not found
Oct 31 12:27:19.751196 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:19.751165    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/50ad7816-358d-461e-a9bd-fadce6246e8f-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-2bfvp\" (UID: \"50ad7816-358d-461e-a9bd-fadce6246e8f\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-2bfvp"
Oct 31 12:27:19.751346 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:19.751205    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0db116b4-d971-4966-bb05-db28d077b6fe-registry-tls\") pod \"image-registry-6dd647c7b7-9255m\" (UID: \"0db116b4-d971-4966-bb05-db28d077b6fe\") " pod="openshift-image-registry/image-registry-6dd647c7b7-9255m"
Oct 31 12:27:19.751346 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:19.751243    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/dbbd2d70-8911-4aa5-96cf-3a16dcdf246d-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-tcnrw\" (UID: \"dbbd2d70-8911-4aa5-96cf-3a16dcdf246d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-tcnrw"
Oct 31 12:27:19.751346 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:19.751269    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5c8865f9-fdf6-49b9-8b4f-69dbc7787084-metrics-tls\") pod \"dns-default-r6fvt\" (UID: \"5c8865f9-fdf6-49b9-8b4f-69dbc7787084\") " pod="openshift-dns/dns-default-r6fvt"
Oct 31 12:27:19.751346 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:19.751288    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9557a3f0-ae0f-41f9-a9b5-15aeda410a38-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p7phb\" (UID: \"9557a3f0-ae0f-41f9-a9b5-15aeda410a38\") " pod="openshift-insights/insights-runtime-extractor-p7phb"
Oct 31 12:27:19.751590 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:27:19.751376    5060 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 31 12:27:19.751590 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:27:19.751396    5060 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 31 12:27:19.751590 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:27:19.751441    5060 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dbbd2d70-8911-4aa5-96cf-3a16dcdf246d-samples-operator-tls podName:dbbd2d70-8911-4aa5-96cf-3a16dcdf246d nodeName:}" failed. No retries permitted until 2026-10-31 12:27:21.751425424 +0000 UTC m=+4.854493114 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/dbbd2d70-8911-4aa5-96cf-3a16dcdf246d-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-tcnrw" (UID: "dbbd2d70-8911-4aa5-96cf-3a16dcdf246d") : secret "samples-operator-tls" not found
Oct 31 12:27:19.751590 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:27:19.751461    5060 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9557a3f0-ae0f-41f9-a9b5-15aeda410a38-insights-runtime-extractor-tls podName:9557a3f0-ae0f-41f9-a9b5-15aeda410a38 nodeName:}" failed. No retries permitted until 2026-10-31 12:27:21.751451536 +0000 UTC m=+4.854519243 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9557a3f0-ae0f-41f9-a9b5-15aeda410a38-insights-runtime-extractor-tls") pod "insights-runtime-extractor-p7phb" (UID: "9557a3f0-ae0f-41f9-a9b5-15aeda410a38") : secret "insights-runtime-extractor-tls" not found
Oct 31 12:27:19.751590 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:19.751484    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/556dda7b-d8e6-4558-b5f3-994b01d6a866-service-ca-bundle\") pod \"router-default-659589798c-776bx\" (UID: \"556dda7b-d8e6-4558-b5f3-994b01d6a866\") " pod="openshift-ingress/router-default-659589798c-776bx"
Oct 31 12:27:19.751590 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:27:19.751508    5060 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 31 12:27:19.751590 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:27:19.751547    5060 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5c8865f9-fdf6-49b9-8b4f-69dbc7787084-metrics-tls podName:5c8865f9-fdf6-49b9-8b4f-69dbc7787084 nodeName:}" failed. No retries permitted until 2026-10-31 12:27:21.75153616 +0000 UTC m=+4.854603872 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5c8865f9-fdf6-49b9-8b4f-69dbc7787084-metrics-tls") pod "dns-default-r6fvt" (UID: "5c8865f9-fdf6-49b9-8b4f-69dbc7787084") : secret "dns-default-metrics-tls" not found
Oct 31 12:27:19.751590 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:19.751584    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/556dda7b-d8e6-4558-b5f3-994b01d6a866-metrics-certs\") pod \"router-default-659589798c-776bx\" (UID: \"556dda7b-d8e6-4558-b5f3-994b01d6a866\") " pod="openshift-ingress/router-default-659589798c-776bx"
Oct 31 12:27:19.751590 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:27:19.751595    5060 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/556dda7b-d8e6-4558-b5f3-994b01d6a866-service-ca-bundle podName:556dda7b-d8e6-4558-b5f3-994b01d6a866 nodeName:}" failed. No retries permitted until 2026-10-31 12:27:21.751585296 +0000 UTC m=+4.854652987 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/556dda7b-d8e6-4558-b5f3-994b01d6a866-service-ca-bundle") pod "router-default-659589798c-776bx" (UID: "556dda7b-d8e6-4558-b5f3-994b01d6a866") : configmap references non-existent config key: service-ca.crt
Oct 31 12:27:19.751935 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:27:19.751639    5060 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 31 12:27:19.751935 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:19.751672    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9083c35a-9739-4f3e-8415-bad1e6433d40-registry-tls\") pod \"image-registry-6cfdbfcdc4-2rq84\" (UID: \"9083c35a-9739-4f3e-8415-bad1e6433d40\") " pod="openshift-image-registry/image-registry-6cfdbfcdc4-2rq84"
Oct 31 12:27:19.751935 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:27:19.751744    5060 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/556dda7b-d8e6-4558-b5f3-994b01d6a866-metrics-certs podName:556dda7b-d8e6-4558-b5f3-994b01d6a866 nodeName:}" failed. No retries permitted until 2026-10-31 12:27:21.751731878 +0000 UTC m=+4.854799567 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/556dda7b-d8e6-4558-b5f3-994b01d6a866-metrics-certs") pod "router-default-659589798c-776bx" (UID: "556dda7b-d8e6-4558-b5f3-994b01d6a866") : secret "router-metrics-certs-default" not found
Oct 31 12:27:19.753832 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:19.753804    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9083c35a-9739-4f3e-8415-bad1e6433d40-registry-tls\") pod \"image-registry-6cfdbfcdc4-2rq84\" (UID: \"9083c35a-9739-4f3e-8415-bad1e6433d40\") " pod="openshift-image-registry/image-registry-6cfdbfcdc4-2rq84"
Oct 31 12:27:19.754159 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:19.754140    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/50ad7816-358d-461e-a9bd-fadce6246e8f-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-2bfvp\" (UID: \"50ad7816-358d-461e-a9bd-fadce6246e8f\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-2bfvp"
Oct 31 12:27:19.754281 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:19.754255    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0db116b4-d971-4966-bb05-db28d077b6fe-registry-tls\") pod \"image-registry-6dd647c7b7-9255m\" (UID: \"0db116b4-d971-4966-bb05-db28d077b6fe\") " pod="openshift-image-registry/image-registry-6dd647c7b7-9255m"
Oct 31 12:27:19.812015 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:19.811972    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-2bfvp"
Oct 31 12:27:19.812202 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:19.811997    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6cfdbfcdc4-2rq84"
Oct 31 12:27:19.824266 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:19.824240    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6dd647c7b7-9255m"
Oct 31 12:27:20.098407 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:20.098358    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-2bfvp"]
Oct 31 12:27:20.101664 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:20.101570    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6cfdbfcdc4-2rq84"]
Oct 31 12:27:20.102893 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:20.102872    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50ad7816_358d_461e_a9bd_fadce6246e8f.slice/crio-1a4ac1679411b731fec30cc1731c86c1cd0f1fe49eaff6e2f424334a91575e9b WatchSource:0}: Error finding container 1a4ac1679411b731fec30cc1731c86c1cd0f1fe49eaff6e2f424334a91575e9b: Status 404 returned error can't find the container with id 1a4ac1679411b731fec30cc1731c86c1cd0f1fe49eaff6e2f424334a91575e9b
Oct 31 12:27:20.104861 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:20.104812    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9083c35a_9739_4f3e_8415_bad1e6433d40.slice/crio-a39d94bd466572faf6c06458dcf2f18a2ef1ccd3b298e363f6a0b042eeac36a3 WatchSource:0}: Error finding container a39d94bd466572faf6c06458dcf2f18a2ef1ccd3b298e363f6a0b042eeac36a3: Status 404 returned error can't find the container with id a39d94bd466572faf6c06458dcf2f18a2ef1ccd3b298e363f6a0b042eeac36a3
Oct 31 12:27:20.112455 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:20.112428    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-2bfvp" event={"ID":"50ad7816-358d-461e-a9bd-fadce6246e8f","Type":"ContainerStarted","Data":"1a4ac1679411b731fec30cc1731c86c1cd0f1fe49eaff6e2f424334a91575e9b"}
Oct 31 12:27:20.113677 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:20.113653    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6cfdbfcdc4-2rq84" event={"ID":"9083c35a-9739-4f3e-8415-bad1e6433d40","Type":"ContainerStarted","Data":"a39d94bd466572faf6c06458dcf2f18a2ef1ccd3b298e363f6a0b042eeac36a3"}
Oct 31 12:27:20.115026 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:20.115006    5060 generic.go:356] "Generic (PLEG): container finished" podID="ddb04b2e-2724-44e9-948a-877a90d8a071" containerID="5c2ba5d038743ef59b4681530bba2d1485c47084c72bebb1b6d6c53c482fdebf" exitCode=0
Oct 31 12:27:20.115118 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:20.115076    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-z879s" event={"ID":"ddb04b2e-2724-44e9-948a-877a90d8a071","Type":"ContainerDied","Data":"5c2ba5d038743ef59b4681530bba2d1485c47084c72bebb1b6d6c53c482fdebf"}
Oct 31 12:27:20.115393 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:20.115374    5060 scope.go:122] "RemoveContainer" containerID="5c2ba5d038743ef59b4681530bba2d1485c47084c72bebb1b6d6c53c482fdebf"
Oct 31 12:27:20.116965 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:20.116946    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-94kh9" event={"ID":"e64bd69d-946f-42ad-94ae-b87b7de3a581","Type":"ContainerStarted","Data":"cee4826b7785252379113722a1192cfaf2e9f2e972b6e84bf57b5d4c3d772a51"}
Oct 31 12:27:20.117043 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:20.116998    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-94kh9" event={"ID":"e64bd69d-946f-42ad-94ae-b87b7de3a581","Type":"ContainerStarted","Data":"cbaceb599348277b4a712495bd3247e8b2f955aa17e79589f0d8280da242bc45"}
Oct 31 12:27:20.118682 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:20.118665    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pcvvm_f032909d-bc39-4c9e-99b7-f3d800842119/console-operator/2.log"
Oct 31 12:27:20.121997 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:20.121976    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6dd647c7b7-9255m"]
Oct 31 12:27:20.125933 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:20.125906    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0db116b4_d971_4966_bb05_db28d077b6fe.slice/crio-fd96831c9d48bccee78c39707230e6b7ce7bccd57174c000192520e9608f46c1 WatchSource:0}: Error finding container fd96831c9d48bccee78c39707230e6b7ce7bccd57174c000192520e9608f46c1: Status 404 returned error can't find the container with id fd96831c9d48bccee78c39707230e6b7ce7bccd57174c000192520e9608f46c1
Oct 31 12:27:21.124553 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:21.124526    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-z879s" event={"ID":"ddb04b2e-2724-44e9-948a-877a90d8a071","Type":"ContainerStarted","Data":"aaa923775c12b51b805acd8a77d1b77a5c9dd38cb071e52f7ce28dd3cfd8d8e5"}
Oct 31 12:27:21.126252 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:21.126225    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6dd647c7b7-9255m" event={"ID":"0db116b4-d971-4966-bb05-db28d077b6fe","Type":"ContainerStarted","Data":"c66777009b60b7ce3008312915aab833aa19ed4537505808b5d66d593e39ba3a"}
Oct 31 12:27:21.126372 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:21.126273    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6dd647c7b7-9255m" event={"ID":"0db116b4-d971-4966-bb05-db28d077b6fe","Type":"ContainerStarted","Data":"fd96831c9d48bccee78c39707230e6b7ce7bccd57174c000192520e9608f46c1"}
Oct 31 12:27:21.127741 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:21.127717    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6cfdbfcdc4-2rq84" event={"ID":"9083c35a-9739-4f3e-8415-bad1e6433d40","Type":"ContainerStarted","Data":"8e04f4d0cbdb741e50f23a4a307636a4dbfca396ef8fbdec5f701edf3edd342c"}
Oct 31 12:27:21.127867 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:21.127833    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6cfdbfcdc4-2rq84"
Oct 31 12:27:21.230170 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:21.230139    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-86758"
Oct 31 12:27:21.672199 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:21.672162    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/dda0c26f-8225-4239-bc27-866b66cbb919-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-gt9rn\" (UID: \"dda0c26f-8225-4239-bc27-866b66cbb919\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gt9rn"
Oct 31 12:27:21.672379 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:27:21.672309    5060 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 31 12:27:21.672379 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:27:21.672376    5060 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dda0c26f-8225-4239-bc27-866b66cbb919-cluster-monitoring-operator-tls podName:dda0c26f-8225-4239-bc27-866b66cbb919 nodeName:}" failed. No retries permitted until 2026-10-31 12:27:25.672361824 +0000 UTC m=+8.775429530 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/dda0c26f-8225-4239-bc27-866b66cbb919-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-gt9rn" (UID: "dda0c26f-8225-4239-bc27-866b66cbb919") : secret "cluster-monitoring-operator-tls" not found
Oct 31 12:27:21.772744 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:21.772708    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/556dda7b-d8e6-4558-b5f3-994b01d6a866-metrics-certs\") pod \"router-default-659589798c-776bx\" (UID: \"556dda7b-d8e6-4558-b5f3-994b01d6a866\") " pod="openshift-ingress/router-default-659589798c-776bx"
Oct 31 12:27:21.772923 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:21.772774    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/dbbd2d70-8911-4aa5-96cf-3a16dcdf246d-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-tcnrw\" (UID: \"dbbd2d70-8911-4aa5-96cf-3a16dcdf246d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-tcnrw"
Oct 31 12:27:21.772923 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:21.772800    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5c8865f9-fdf6-49b9-8b4f-69dbc7787084-metrics-tls\") pod \"dns-default-r6fvt\" (UID: \"5c8865f9-fdf6-49b9-8b4f-69dbc7787084\") " pod="openshift-dns/dns-default-r6fvt"
Oct 31 12:27:21.772923 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:21.772819    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9557a3f0-ae0f-41f9-a9b5-15aeda410a38-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p7phb\" (UID: \"9557a3f0-ae0f-41f9-a9b5-15aeda410a38\") " pod="openshift-insights/insights-runtime-extractor-p7phb"
Oct 31 12:27:21.772923 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:21.772850    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/556dda7b-d8e6-4558-b5f3-994b01d6a866-service-ca-bundle\") pod \"router-default-659589798c-776bx\" (UID: \"556dda7b-d8e6-4558-b5f3-994b01d6a866\") " pod="openshift-ingress/router-default-659589798c-776bx"
Oct 31 12:27:21.772923 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:27:21.772852    5060 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 31 12:27:21.772923 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:27:21.772918    5060 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 31 12:27:21.773237 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:27:21.772922    5060 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/556dda7b-d8e6-4558-b5f3-994b01d6a866-metrics-certs podName:556dda7b-d8e6-4558-b5f3-994b01d6a866 nodeName:}" failed. No retries permitted until 2026-10-31 12:27:25.772902862 +0000 UTC m=+8.875970568 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/556dda7b-d8e6-4558-b5f3-994b01d6a866-metrics-certs") pod "router-default-659589798c-776bx" (UID: "556dda7b-d8e6-4558-b5f3-994b01d6a866") : secret "router-metrics-certs-default" not found
Oct 31 12:27:21.773237 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:27:21.772948    5060 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 31 12:27:21.773237 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:27:21.772971    5060 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/556dda7b-d8e6-4558-b5f3-994b01d6a866-service-ca-bundle podName:556dda7b-d8e6-4558-b5f3-994b01d6a866 nodeName:}" failed. No retries permitted until 2026-10-31 12:27:25.772957659 +0000 UTC m=+8.876025348 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/556dda7b-d8e6-4558-b5f3-994b01d6a866-service-ca-bundle") pod "router-default-659589798c-776bx" (UID: "556dda7b-d8e6-4558-b5f3-994b01d6a866") : configmap references non-existent config key: service-ca.crt
Oct 31 12:27:21.773237 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:27:21.772989    5060 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5c8865f9-fdf6-49b9-8b4f-69dbc7787084-metrics-tls podName:5c8865f9-fdf6-49b9-8b4f-69dbc7787084 nodeName:}" failed. No retries permitted until 2026-10-31 12:27:25.772980101 +0000 UTC m=+8.876047801 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5c8865f9-fdf6-49b9-8b4f-69dbc7787084-metrics-tls") pod "dns-default-r6fvt" (UID: "5c8865f9-fdf6-49b9-8b4f-69dbc7787084") : secret "dns-default-metrics-tls" not found
Oct 31 12:27:21.773237 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:27:21.773006    5060 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9557a3f0-ae0f-41f9-a9b5-15aeda410a38-insights-runtime-extractor-tls podName:9557a3f0-ae0f-41f9-a9b5-15aeda410a38 nodeName:}" failed. No retries permitted until 2026-10-31 12:27:25.773000315 +0000 UTC m=+8.876068006 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9557a3f0-ae0f-41f9-a9b5-15aeda410a38-insights-runtime-extractor-tls") pod "insights-runtime-extractor-p7phb" (UID: "9557a3f0-ae0f-41f9-a9b5-15aeda410a38") : secret "insights-runtime-extractor-tls" not found
Oct 31 12:27:21.775246 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:21.775221    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/dbbd2d70-8911-4aa5-96cf-3a16dcdf246d-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-tcnrw\" (UID: \"dbbd2d70-8911-4aa5-96cf-3a16dcdf246d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-tcnrw"
Oct 31 12:27:21.911327 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:21.911286    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-tcnrw"
Oct 31 12:27:22.132629 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:22.132589    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-2bfvp" event={"ID":"50ad7816-358d-461e-a9bd-fadce6246e8f","Type":"ContainerStarted","Data":"ec3dff382d3ed174c280faf5c2d2837d60c07aecf0d49b646d7e3a6429f658e7"}
Oct 31 12:27:22.133063 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:22.132609    5060 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 31 12:27:22.184394 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:22.184328    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-tcnrw"]
Oct 31 12:27:23.138241 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:23.138207    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-tcnrw" event={"ID":"dbbd2d70-8911-4aa5-96cf-3a16dcdf246d","Type":"ContainerStarted","Data":"7ad0ef0407cf33b571145b922b1a4782af195b88a2dcd160e5906353d57907b6"}
Oct 31 12:27:24.143478 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:24.143439    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-tcnrw" event={"ID":"dbbd2d70-8911-4aa5-96cf-3a16dcdf246d","Type":"ContainerStarted","Data":"d1caac661ed880f8c7384960dd35324b4d8943abd516651407e302f7bb6c9f0c"}
Oct 31 12:27:24.143905 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:24.143514    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-tcnrw" event={"ID":"dbbd2d70-8911-4aa5-96cf-3a16dcdf246d","Type":"ContainerStarted","Data":"c6e35822ebe3c81f1c67a59e2758b755dc76f542a29bb67953cdf555f18554cc"}
Oct 31 12:27:24.397626 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:24.397585    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-pcvvm"
Oct 31 12:27:24.398093 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:24.398075    5060 scope.go:122] "RemoveContainer" containerID="2ddca170fab98ab89426aa1598ea575169dba5f01fc84f419efc709bb7a13601"
Oct 31 12:27:24.398256 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:27:24.398236    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-pcvvm_openshift-console-operator(f032909d-bc39-4c9e-99b7-f3d800842119)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-pcvvm" podUID="f032909d-bc39-4c9e-99b7-f3d800842119"
Oct 31 12:27:25.109419 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:25.109389    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-86758"
Oct 31 12:27:25.109684 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:25.109667    5060 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 31 12:27:25.711860 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:25.711809    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/dda0c26f-8225-4239-bc27-866b66cbb919-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-gt9rn\" (UID: \"dda0c26f-8225-4239-bc27-866b66cbb919\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gt9rn"
Oct 31 12:27:25.714337 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:25.714311    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/dda0c26f-8225-4239-bc27-866b66cbb919-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-gt9rn\" (UID: \"dda0c26f-8225-4239-bc27-866b66cbb919\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gt9rn"
Oct 31 12:27:25.809704 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:25.809665    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gt9rn"
Oct 31 12:27:25.813093 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:25.813069    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5c8865f9-fdf6-49b9-8b4f-69dbc7787084-metrics-tls\") pod \"dns-default-r6fvt\" (UID: \"5c8865f9-fdf6-49b9-8b4f-69dbc7787084\") " pod="openshift-dns/dns-default-r6fvt"
Oct 31 12:27:25.813156 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:25.813104    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9557a3f0-ae0f-41f9-a9b5-15aeda410a38-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p7phb\" (UID: \"9557a3f0-ae0f-41f9-a9b5-15aeda410a38\") " pod="openshift-insights/insights-runtime-extractor-p7phb"
Oct 31 12:27:25.813156 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:25.813130    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/556dda7b-d8e6-4558-b5f3-994b01d6a866-service-ca-bundle\") pod \"router-default-659589798c-776bx\" (UID: \"556dda7b-d8e6-4558-b5f3-994b01d6a866\") " pod="openshift-ingress/router-default-659589798c-776bx"
Oct 31 12:27:25.813292 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:25.813267    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/556dda7b-d8e6-4558-b5f3-994b01d6a866-metrics-certs\") pod \"router-default-659589798c-776bx\" (UID: \"556dda7b-d8e6-4558-b5f3-994b01d6a866\") " pod="openshift-ingress/router-default-659589798c-776bx"
Oct 31 12:27:25.813782 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:25.813758    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/556dda7b-d8e6-4558-b5f3-994b01d6a866-service-ca-bundle\") pod \"router-default-659589798c-776bx\" (UID: \"556dda7b-d8e6-4558-b5f3-994b01d6a866\") " pod="openshift-ingress/router-default-659589798c-776bx"
Oct 31 12:27:25.815555 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:25.815468    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5c8865f9-fdf6-49b9-8b4f-69dbc7787084-metrics-tls\") pod \"dns-default-r6fvt\" (UID: \"5c8865f9-fdf6-49b9-8b4f-69dbc7787084\") " pod="openshift-dns/dns-default-r6fvt"
Oct 31 12:27:25.815751 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:25.815731    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/556dda7b-d8e6-4558-b5f3-994b01d6a866-metrics-certs\") pod \"router-default-659589798c-776bx\" (UID: \"556dda7b-d8e6-4558-b5f3-994b01d6a866\") " pod="openshift-ingress/router-default-659589798c-776bx"
Oct 31 12:27:25.815826 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:25.815803    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9557a3f0-ae0f-41f9-a9b5-15aeda410a38-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p7phb\" (UID: \"9557a3f0-ae0f-41f9-a9b5-15aeda410a38\") " pod="openshift-insights/insights-runtime-extractor-p7phb"
Oct 31 12:27:25.824718 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:25.824695    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-r6fvt"
Oct 31 12:27:25.825789 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:25.825763    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-p7phb"
Oct 31 12:27:26.087239 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:26.087210    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-gt9rn"]
Oct 31 12:27:26.090451 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:26.090426    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddda0c26f_8225_4239_bc27_866b66cbb919.slice/crio-b916cc090ea869df431257bb407be1ab114c99832349160d1d97086b419008cd WatchSource:0}: Error finding container b916cc090ea869df431257bb407be1ab114c99832349160d1d97086b419008cd: Status 404 returned error can't find the container with id b916cc090ea869df431257bb407be1ab114c99832349160d1d97086b419008cd
Oct 31 12:27:26.111220 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:26.111162    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-659589798c-776bx"
Oct 31 12:27:26.119176 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:26.119152    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-r6fvt"]
Oct 31 12:27:26.122517 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:26.122464    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-p7phb"]
Oct 31 12:27:26.123069 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:26.122994    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c8865f9_fdf6_49b9_8b4f_69dbc7787084.slice/crio-5153c04e45597da564035dc1bca9957f09945977681c7bacff452dd073d15d44 WatchSource:0}: Error finding container 5153c04e45597da564035dc1bca9957f09945977681c7bacff452dd073d15d44: Status 404 returned error can't find the container with id 5153c04e45597da564035dc1bca9957f09945977681c7bacff452dd073d15d44
Oct 31 12:27:26.126008 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:26.125977    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9557a3f0_ae0f_41f9_a9b5_15aeda410a38.slice/crio-7854d2aa93e7da4262ca0eff60a59bb0b5ec43c579b7a7c5d0716a695bc20868 WatchSource:0}: Error finding container 7854d2aa93e7da4262ca0eff60a59bb0b5ec43c579b7a7c5d0716a695bc20868: Status 404 returned error can't find the container with id 7854d2aa93e7da4262ca0eff60a59bb0b5ec43c579b7a7c5d0716a695bc20868
Oct 31 12:27:26.152151 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:26.152110    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-r6fvt" event={"ID":"5c8865f9-fdf6-49b9-8b4f-69dbc7787084","Type":"ContainerStarted","Data":"5153c04e45597da564035dc1bca9957f09945977681c7bacff452dd073d15d44"}
Oct 31 12:27:26.153383 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:26.153355    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gt9rn" event={"ID":"dda0c26f-8225-4239-bc27-866b66cbb919","Type":"ContainerStarted","Data":"b916cc090ea869df431257bb407be1ab114c99832349160d1d97086b419008cd"}
Oct 31 12:27:26.154985 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:26.154959    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-p7phb" event={"ID":"9557a3f0-ae0f-41f9-a9b5-15aeda410a38","Type":"ContainerStarted","Data":"7854d2aa93e7da4262ca0eff60a59bb0b5ec43c579b7a7c5d0716a695bc20868"}
Oct 31 12:27:26.399463 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:26.399379    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-659589798c-776bx"]
Oct 31 12:27:26.402482 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:26.402448    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod556dda7b_d8e6_4558_b5f3_994b01d6a866.slice/crio-d19c8791a864ac47841ddf73536b1d962952585d8eb8b8a6d45bc7aedf542fad WatchSource:0}: Error finding container d19c8791a864ac47841ddf73536b1d962952585d8eb8b8a6d45bc7aedf542fad: Status 404 returned error can't find the container with id d19c8791a864ac47841ddf73536b1d962952585d8eb8b8a6d45bc7aedf542fad
Oct 31 12:27:27.162656 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:27.162619    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-p7phb" event={"ID":"9557a3f0-ae0f-41f9-a9b5-15aeda410a38","Type":"ContainerStarted","Data":"de1b20b1b8c2d5b0c7f1df7128c28e63d3eb997440742eb1982cd7a61a62c102"}
Oct 31 12:27:27.164909 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:27.164867    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-659589798c-776bx" event={"ID":"556dda7b-d8e6-4558-b5f3-994b01d6a866","Type":"ContainerStarted","Data":"52df44e82de96c89c0e71e429b918266509329ccb7e554963250e134d8fa7311"}
Oct 31 12:27:27.166206 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:27.165571    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-659589798c-776bx" event={"ID":"556dda7b-d8e6-4558-b5f3-994b01d6a866","Type":"ContainerStarted","Data":"d19c8791a864ac47841ddf73536b1d962952585d8eb8b8a6d45bc7aedf542fad"}
Oct 31 12:27:27.541372 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:27.541333    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6dd647c7b7-9255m"
Oct 31 12:27:27.712187 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:27.712131    5060 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-pcvvm"
Oct 31 12:27:27.712951 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:27.712928    5060 scope.go:122] "RemoveContainer" containerID="2ddca170fab98ab89426aa1598ea575169dba5f01fc84f419efc709bb7a13601"
Oct 31 12:27:27.713254 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:27:27.713227    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-pcvvm_openshift-console-operator(f032909d-bc39-4c9e-99b7-f3d800842119)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-pcvvm" podUID="f032909d-bc39-4c9e-99b7-f3d800842119"
Oct 31 12:27:27.842644 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:27.842564    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-tdrmt"
Oct 31 12:27:28.112258 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:28.112180    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-659589798c-776bx"
Oct 31 12:27:28.115122 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:28.115095    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-659589798c-776bx"
Oct 31 12:27:28.172413 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:28.172378    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-p7phb" event={"ID":"9557a3f0-ae0f-41f9-a9b5-15aeda410a38","Type":"ContainerStarted","Data":"ea315a4019bd58fa1180fb78eb00f881ccad73032f8527367efd67f7764a2114"}
Oct 31 12:27:29.043308 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:29.043271    5060 scope.go:122] "RemoveContainer" containerID="2ddca170fab98ab89426aa1598ea575169dba5f01fc84f419efc709bb7a13601"
Oct 31 12:27:29.182065 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:29.182035    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pcvvm_f032909d-bc39-4c9e-99b7-f3d800842119/console-operator/2.log"
Oct 31 12:27:29.182563 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:29.182185    5060 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 31 12:27:29.182563 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:29.182362    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-pcvvm" event={"ID":"f032909d-bc39-4c9e-99b7-f3d800842119","Type":"ContainerStarted","Data":"fd8f04f6d335e02ba549770427a876288bb97fdabb3102ae8b5ac5265fea200a"}
Oct 31 12:27:29.182961 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:29.182933    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-pcvvm"
Oct 31 12:27:29.184095 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:29.184054    5060 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-pcvvm container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.14:8443/readyz\": dial tcp 10.132.0.14:8443: connect: connection refused" start-of-body=
Oct 31 12:27:29.184206 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:29.184115    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-7cd56b8dbc-pcvvm" podUID="f032909d-bc39-4c9e-99b7-f3d800842119" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.14:8443/readyz\": dial tcp 10.132.0.14:8443: connect: connection refused"
Oct 31 12:27:29.408960 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:29.408934    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-86758"
Oct 31 12:27:30.189341 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:30.189303    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-r6fvt" event={"ID":"5c8865f9-fdf6-49b9-8b4f-69dbc7787084","Type":"ContainerStarted","Data":"3a5ea7e81730805d591903052b7932a2ede6fa6b52b73e7df2991c13ba7d4989"}
Oct 31 12:27:30.189718 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:30.189401    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-r6fvt" event={"ID":"5c8865f9-fdf6-49b9-8b4f-69dbc7787084","Type":"ContainerStarted","Data":"12abdf6a3f75aaf2cd3e239e8f22da79478f4a36d197fce98feb170875376b81"}
Oct 31 12:27:30.189718 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:30.189433    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-r6fvt"
Oct 31 12:27:30.190976 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:30.190951    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gt9rn" event={"ID":"dda0c26f-8225-4239-bc27-866b66cbb919","Type":"ContainerStarted","Data":"a816992501badf09f0b9a531f944fb419cd002cc720af9ba8488df372ca63748"}
Oct 31 12:27:30.195450 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:30.195428    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-pcvvm"
Oct 31 12:27:30.849560 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:30.849505    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-rrx9r-zvwfn"]
Oct 31 12:27:30.853325 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:30.853295    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-rrx9r-zvwfn"
Oct 31 12:27:30.887912 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:30.887878    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-rrx9r-zvwfn"]
Oct 31 12:27:30.917760 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:30.917708    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6cfdbfcdc4-2rq84"]
Oct 31 12:27:30.960550 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:30.960478    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f7f000c-c637-4b51-b4ad-805ea3205f29-trusted-ca-bundle\") pod \"periodic-gathering-rrx9r-zvwfn\" (UID: \"9f7f000c-c637-4b51-b4ad-805ea3205f29\") " pod="openshift-insights/periodic-gathering-rrx9r-zvwfn"
Oct 31 12:27:30.960703 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:30.960613    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7c87\" (UniqueName: \"kubernetes.io/projected/9f7f000c-c637-4b51-b4ad-805ea3205f29-kube-api-access-x7c87\") pod \"periodic-gathering-rrx9r-zvwfn\" (UID: \"9f7f000c-c637-4b51-b4ad-805ea3205f29\") " pod="openshift-insights/periodic-gathering-rrx9r-zvwfn"
Oct 31 12:27:30.960703 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:30.960648    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/9f7f000c-c637-4b51-b4ad-805ea3205f29-archives-path\") pod \"periodic-gathering-rrx9r-zvwfn\" (UID: \"9f7f000c-c637-4b51-b4ad-805ea3205f29\") " pod="openshift-insights/periodic-gathering-rrx9r-zvwfn"
Oct 31 12:27:30.960800 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:30.960777    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f7f000c-c637-4b51-b4ad-805ea3205f29-service-ca-bundle\") pod \"periodic-gathering-rrx9r-zvwfn\" (UID: \"9f7f000c-c637-4b51-b4ad-805ea3205f29\") " pod="openshift-insights/periodic-gathering-rrx9r-zvwfn"
Oct 31 12:27:31.062182 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:31.062145    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7c87\" (UniqueName: \"kubernetes.io/projected/9f7f000c-c637-4b51-b4ad-805ea3205f29-kube-api-access-x7c87\") pod \"periodic-gathering-rrx9r-zvwfn\" (UID: \"9f7f000c-c637-4b51-b4ad-805ea3205f29\") " pod="openshift-insights/periodic-gathering-rrx9r-zvwfn"
Oct 31 12:27:31.062182 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:31.062190    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/9f7f000c-c637-4b51-b4ad-805ea3205f29-archives-path\") pod \"periodic-gathering-rrx9r-zvwfn\" (UID: \"9f7f000c-c637-4b51-b4ad-805ea3205f29\") " pod="openshift-insights/periodic-gathering-rrx9r-zvwfn"
Oct 31 12:27:31.062455 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:31.062257    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f7f000c-c637-4b51-b4ad-805ea3205f29-service-ca-bundle\") pod \"periodic-gathering-rrx9r-zvwfn\" (UID: \"9f7f000c-c637-4b51-b4ad-805ea3205f29\") " pod="openshift-insights/periodic-gathering-rrx9r-zvwfn"
Oct 31 12:27:31.062455 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:31.062292    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f7f000c-c637-4b51-b4ad-805ea3205f29-trusted-ca-bundle\") pod \"periodic-gathering-rrx9r-zvwfn\" (UID: \"9f7f000c-c637-4b51-b4ad-805ea3205f29\") " pod="openshift-insights/periodic-gathering-rrx9r-zvwfn"
Oct 31 12:27:31.062721 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:31.062693    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/9f7f000c-c637-4b51-b4ad-805ea3205f29-archives-path\") pod \"periodic-gathering-rrx9r-zvwfn\" (UID: \"9f7f000c-c637-4b51-b4ad-805ea3205f29\") " pod="openshift-insights/periodic-gathering-rrx9r-zvwfn"
Oct 31 12:27:31.062958 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:31.062935    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f7f000c-c637-4b51-b4ad-805ea3205f29-service-ca-bundle\") pod \"periodic-gathering-rrx9r-zvwfn\" (UID: \"9f7f000c-c637-4b51-b4ad-805ea3205f29\") " pod="openshift-insights/periodic-gathering-rrx9r-zvwfn"
Oct 31 12:27:31.063105 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:31.063086    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f7f000c-c637-4b51-b4ad-805ea3205f29-trusted-ca-bundle\") pod \"periodic-gathering-rrx9r-zvwfn\" (UID: \"9f7f000c-c637-4b51-b4ad-805ea3205f29\") " pod="openshift-insights/periodic-gathering-rrx9r-zvwfn"
Oct 31 12:27:31.074636 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:31.074610    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7c87\" (UniqueName: \"kubernetes.io/projected/9f7f000c-c637-4b51-b4ad-805ea3205f29-kube-api-access-x7c87\") pod \"periodic-gathering-rrx9r-zvwfn\" (UID: \"9f7f000c-c637-4b51-b4ad-805ea3205f29\") " pod="openshift-insights/periodic-gathering-rrx9r-zvwfn"
Oct 31 12:27:31.167873 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:31.167798    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-rrx9r-zvwfn"
Oct 31 12:27:31.197447 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:31.197408    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-p7phb" event={"ID":"9557a3f0-ae0f-41f9-a9b5-15aeda410a38","Type":"ContainerStarted","Data":"2f74ab86e237baa46647cf8039a985b8d1ceae25298277d4b05be2913e418c2f"}
Oct 31 12:27:31.442308 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:31.442238    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-rrx9r-zvwfn"]
Oct 31 12:27:31.445656 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:31.445628    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f7f000c_c637_4b51_b4ad_805ea3205f29.slice/crio-4f14c40ac2a6f12a28385a283a0f92f83512e34ca9413db9813bec0c36c6d237 WatchSource:0}: Error finding container 4f14c40ac2a6f12a28385a283a0f92f83512e34ca9413db9813bec0c36c6d237: Status 404 returned error can't find the container with id 4f14c40ac2a6f12a28385a283a0f92f83512e34ca9413db9813bec0c36c6d237
Oct 31 12:27:32.203109 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:32.203015    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-rrx9r-zvwfn" event={"ID":"9f7f000c-c637-4b51-b4ad-805ea3205f29","Type":"ContainerStarted","Data":"43aaaeaf224e7d4f82f3f279bcb6fc07a9f000d3ef5cf008c7658e5e63713f75"}
Oct 31 12:27:32.203109 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:32.203080    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-rrx9r-zvwfn" event={"ID":"9f7f000c-c637-4b51-b4ad-805ea3205f29","Type":"ContainerStarted","Data":"4f14c40ac2a6f12a28385a283a0f92f83512e34ca9413db9813bec0c36c6d237"}
Oct 31 12:27:32.223010 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:32.222957    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-rrx9r-zvwfn" podStartSLOduration=2.222943099 podStartE2EDuration="2.222943099s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:27:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:27:32.221922979 +0000 UTC m=+15.324990691" watchObservedRunningTime="2026-10-31 12:27:32.222943099 +0000 UTC m=+15.326010855"
Oct 31 12:27:32.686902 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:32.686871    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-r6fvt_5c8865f9-fdf6-49b9-8b4f-69dbc7787084/dns/0.log"
Oct 31 12:27:32.857162 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:32.857128    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-r6fvt_5c8865f9-fdf6-49b9-8b4f-69dbc7787084/kube-rbac-proxy/0.log"
Oct 31 12:27:33.256327 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:33.256294    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-v9ks4_6bbe13aa-df4b-40b0-8595-68fe1166a082/dns-node-resolver/0.log"
Oct 31 12:27:33.626996 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:33.626956    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-659589798c-776bx"
Oct 31 12:27:33.627185 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:33.627097    5060 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 31 12:27:33.628384 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:33.628354    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-659589798c-776bx"
Oct 31 12:27:33.657423 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:33.657400    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-6cfdbfcdc4-2rq84_9083c35a-9739-4f3e-8415-bad1e6433d40/registry/0.log"
Oct 31 12:27:33.857012 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:33.856975    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-6dd647c7b7-9255m_0db116b4-d971-4966-bb05-db28d077b6fe/registry/0.log"
Oct 31 12:27:34.656964 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:34.656935    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-wdkzp_6c661ce8-9f7b-4509-bd17-2380bcd1c425/node-ca/0.log"
Oct 31 12:27:34.856621 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:34.856590    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-659589798c-776bx_556dda7b-d8e6-4558-b5f3-994b01d6a866/router/0.log"
Oct 31 12:27:35.256587 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:35.256558    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-rl55s_891b3c47-ff7c-4a7a-a09b-411361a843f7/serve-healthcheck-canary/0.log"
Oct 31 12:27:35.974096 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:35.974060    5060 patch_prober.go:29] interesting pod/image-registry-6dd647c7b7-9255m container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Oct 31 12:27:35.974486 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:35.974112    5060 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-6dd647c7b7-9255m" podUID="0db116b4-d971-4966-bb05-db28d077b6fe" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 12:27:37.545952 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:37.545914    5060 patch_prober.go:29] interesting pod/image-registry-6dd647c7b7-9255m container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Oct 31 12:27:37.546334 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:37.545976    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-6dd647c7b7-9255m" podUID="0db116b4-d971-4966-bb05-db28d077b6fe" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 12:27:38.521162 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:38.521124    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-8jlvq"
Oct 31 12:27:39.376528 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:39.376473    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-gv4qj"]
Oct 31 12:27:39.381617 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:39.381592    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-gv4qj"
Oct 31 12:27:39.450982 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:39.450949    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-9c5q9\""
Oct 31 12:27:39.451173 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:39.451021    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 31 12:27:39.451173 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:39.451040    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 31 12:27:39.469142 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:39.469111    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 31 12:27:39.469292 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:39.469225    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 31 12:27:39.530918 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:39.530886    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/77b0599f-2680-4374-be49-1c2dd34e2618-root\") pod \"node-exporter-gv4qj\" (UID: \"77b0599f-2680-4374-be49-1c2dd34e2618\") " pod="openshift-monitoring/node-exporter-gv4qj"
Oct 31 12:27:39.531065 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:39.530924    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/77b0599f-2680-4374-be49-1c2dd34e2618-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gv4qj\" (UID: \"77b0599f-2680-4374-be49-1c2dd34e2618\") " pod="openshift-monitoring/node-exporter-gv4qj"
Oct 31 12:27:39.531065 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:39.531025    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/77b0599f-2680-4374-be49-1c2dd34e2618-metrics-client-ca\") pod \"node-exporter-gv4qj\" (UID: \"77b0599f-2680-4374-be49-1c2dd34e2618\") " pod="openshift-monitoring/node-exporter-gv4qj"
Oct 31 12:27:39.531065 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:39.531053    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/77b0599f-2680-4374-be49-1c2dd34e2618-node-exporter-accelerators-collector-config\") pod \"node-exporter-gv4qj\" (UID: \"77b0599f-2680-4374-be49-1c2dd34e2618\") " pod="openshift-monitoring/node-exporter-gv4qj"
Oct 31 12:27:39.531178 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:39.531071    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7j8kc\" (UniqueName: \"kubernetes.io/projected/77b0599f-2680-4374-be49-1c2dd34e2618-kube-api-access-7j8kc\") pod \"node-exporter-gv4qj\" (UID: \"77b0599f-2680-4374-be49-1c2dd34e2618\") " pod="openshift-monitoring/node-exporter-gv4qj"
Oct 31 12:27:39.531178 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:39.531095    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/77b0599f-2680-4374-be49-1c2dd34e2618-node-exporter-wtmp\") pod \"node-exporter-gv4qj\" (UID: \"77b0599f-2680-4374-be49-1c2dd34e2618\") " pod="openshift-monitoring/node-exporter-gv4qj"
Oct 31 12:27:39.531238 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:39.531185    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/77b0599f-2680-4374-be49-1c2dd34e2618-node-exporter-textfile\") pod \"node-exporter-gv4qj\" (UID: \"77b0599f-2680-4374-be49-1c2dd34e2618\") " pod="openshift-monitoring/node-exporter-gv4qj"
Oct 31 12:27:39.531238 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:39.531222    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/77b0599f-2680-4374-be49-1c2dd34e2618-node-exporter-tls\") pod \"node-exporter-gv4qj\" (UID: \"77b0599f-2680-4374-be49-1c2dd34e2618\") " pod="openshift-monitoring/node-exporter-gv4qj"
Oct 31 12:27:39.531308 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:39.531244    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/77b0599f-2680-4374-be49-1c2dd34e2618-sys\") pod \"node-exporter-gv4qj\" (UID: \"77b0599f-2680-4374-be49-1c2dd34e2618\") " pod="openshift-monitoring/node-exporter-gv4qj"
Oct 31 12:27:39.632078 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:39.631995    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/77b0599f-2680-4374-be49-1c2dd34e2618-sys\") pod \"node-exporter-gv4qj\" (UID: \"77b0599f-2680-4374-be49-1c2dd34e2618\") " pod="openshift-monitoring/node-exporter-gv4qj"
Oct 31 12:27:39.632078 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:39.632041    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/77b0599f-2680-4374-be49-1c2dd34e2618-root\") pod \"node-exporter-gv4qj\" (UID: \"77b0599f-2680-4374-be49-1c2dd34e2618\") " pod="openshift-monitoring/node-exporter-gv4qj"
Oct 31 12:27:39.632078 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:39.632060    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/77b0599f-2680-4374-be49-1c2dd34e2618-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gv4qj\" (UID: \"77b0599f-2680-4374-be49-1c2dd34e2618\") " pod="openshift-monitoring/node-exporter-gv4qj"
Oct 31 12:27:39.632337 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:39.632099    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/77b0599f-2680-4374-be49-1c2dd34e2618-metrics-client-ca\") pod \"node-exporter-gv4qj\" (UID: \"77b0599f-2680-4374-be49-1c2dd34e2618\") " pod="openshift-monitoring/node-exporter-gv4qj"
Oct 31 12:27:39.632337 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:39.632106    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/77b0599f-2680-4374-be49-1c2dd34e2618-sys\") pod \"node-exporter-gv4qj\" (UID: \"77b0599f-2680-4374-be49-1c2dd34e2618\") " pod="openshift-monitoring/node-exporter-gv4qj"
Oct 31 12:27:39.632337 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:39.632118    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/77b0599f-2680-4374-be49-1c2dd34e2618-node-exporter-accelerators-collector-config\") pod \"node-exporter-gv4qj\" (UID: \"77b0599f-2680-4374-be49-1c2dd34e2618\") " pod="openshift-monitoring/node-exporter-gv4qj"
Oct 31 12:27:39.632337 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:39.632135    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7j8kc\" (UniqueName: \"kubernetes.io/projected/77b0599f-2680-4374-be49-1c2dd34e2618-kube-api-access-7j8kc\") pod \"node-exporter-gv4qj\" (UID: \"77b0599f-2680-4374-be49-1c2dd34e2618\") " pod="openshift-monitoring/node-exporter-gv4qj"
Oct 31 12:27:39.632337 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:39.632108    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/77b0599f-2680-4374-be49-1c2dd34e2618-root\") pod \"node-exporter-gv4qj\" (UID: \"77b0599f-2680-4374-be49-1c2dd34e2618\") " pod="openshift-monitoring/node-exporter-gv4qj"
Oct 31 12:27:39.632337 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:39.632155    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/77b0599f-2680-4374-be49-1c2dd34e2618-node-exporter-wtmp\") pod \"node-exporter-gv4qj\" (UID: \"77b0599f-2680-4374-be49-1c2dd34e2618\") " pod="openshift-monitoring/node-exporter-gv4qj"
Oct 31 12:27:39.632337 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:39.632200    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/77b0599f-2680-4374-be49-1c2dd34e2618-node-exporter-textfile\") pod \"node-exporter-gv4qj\" (UID: \"77b0599f-2680-4374-be49-1c2dd34e2618\") " pod="openshift-monitoring/node-exporter-gv4qj"
Oct 31 12:27:39.632337 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:39.632321    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/77b0599f-2680-4374-be49-1c2dd34e2618-node-exporter-tls\") pod \"node-exporter-gv4qj\" (UID: \"77b0599f-2680-4374-be49-1c2dd34e2618\") " pod="openshift-monitoring/node-exporter-gv4qj"
Oct 31 12:27:39.632770 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:39.632371    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/77b0599f-2680-4374-be49-1c2dd34e2618-node-exporter-wtmp\") pod \"node-exporter-gv4qj\" (UID: \"77b0599f-2680-4374-be49-1c2dd34e2618\") " pod="openshift-monitoring/node-exporter-gv4qj"
Oct 31 12:27:39.632770 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:39.632614    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/77b0599f-2680-4374-be49-1c2dd34e2618-node-exporter-textfile\") pod \"node-exporter-gv4qj\" (UID: \"77b0599f-2680-4374-be49-1c2dd34e2618\") " pod="openshift-monitoring/node-exporter-gv4qj"
Oct 31 12:27:39.632770 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:39.632751    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/77b0599f-2680-4374-be49-1c2dd34e2618-node-exporter-accelerators-collector-config\") pod \"node-exporter-gv4qj\" (UID: \"77b0599f-2680-4374-be49-1c2dd34e2618\") " pod="openshift-monitoring/node-exporter-gv4qj"
Oct 31 12:27:39.632909 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:39.632849    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/77b0599f-2680-4374-be49-1c2dd34e2618-metrics-client-ca\") pod \"node-exporter-gv4qj\" (UID: \"77b0599f-2680-4374-be49-1c2dd34e2618\") " pod="openshift-monitoring/node-exporter-gv4qj"
Oct 31 12:27:39.634528 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:39.634509    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/77b0599f-2680-4374-be49-1c2dd34e2618-node-exporter-tls\") pod \"node-exporter-gv4qj\" (UID: \"77b0599f-2680-4374-be49-1c2dd34e2618\") " pod="openshift-monitoring/node-exporter-gv4qj"
Oct 31 12:27:39.634578 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:39.634560    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/77b0599f-2680-4374-be49-1c2dd34e2618-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gv4qj\" (UID: \"77b0599f-2680-4374-be49-1c2dd34e2618\") " pod="openshift-monitoring/node-exporter-gv4qj"
Oct 31 12:27:39.644385 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:39.644359    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7j8kc\" (UniqueName: \"kubernetes.io/projected/77b0599f-2680-4374-be49-1c2dd34e2618-kube-api-access-7j8kc\") pod \"node-exporter-gv4qj\" (UID: \"77b0599f-2680-4374-be49-1c2dd34e2618\") " pod="openshift-monitoring/node-exporter-gv4qj"
Oct 31 12:27:39.704560 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:39.704511    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-gv4qj"
Oct 31 12:27:39.717787 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:27:39.717756    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77b0599f_2680_4374_be49_1c2dd34e2618.slice/crio-4b509030337ef7aa216de5e9b645b6ba5df4b3372d44b2ff8b5fe05b82745430 WatchSource:0}: Error finding container 4b509030337ef7aa216de5e9b645b6ba5df4b3372d44b2ff8b5fe05b82745430: Status 404 returned error can't find the container with id 4b509030337ef7aa216de5e9b645b6ba5df4b3372d44b2ff8b5fe05b82745430
Oct 31 12:27:40.199946 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:40.199915    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-r6fvt"
Oct 31 12:27:40.235581 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:40.235545    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gv4qj" event={"ID":"77b0599f-2680-4374-be49-1c2dd34e2618","Type":"ContainerStarted","Data":"4b509030337ef7aa216de5e9b645b6ba5df4b3372d44b2ff8b5fe05b82745430"}
Oct 31 12:27:40.924398 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:40.924370    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6cfdbfcdc4-2rq84"
Oct 31 12:27:42.245595 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:42.245561    5060 generic.go:356] "Generic (PLEG): container finished" podID="77b0599f-2680-4374-be49-1c2dd34e2618" containerID="d6d7197271ee68b539dbda71d2e4f9296117d4bfb107daf1fcbbc6ef2e9ea6ca" exitCode=0
Oct 31 12:27:42.246010 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:42.245617    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gv4qj" event={"ID":"77b0599f-2680-4374-be49-1c2dd34e2618","Type":"ContainerDied","Data":"d6d7197271ee68b539dbda71d2e4f9296117d4bfb107daf1fcbbc6ef2e9ea6ca"}
Oct 31 12:27:43.252232 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:43.252195    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gv4qj" event={"ID":"77b0599f-2680-4374-be49-1c2dd34e2618","Type":"ContainerStarted","Data":"5faaae1b29382c04bd3908a2cc6b7cc81db68c5b34d613fe00dcc7f01d740dc9"}
Oct 31 12:27:43.252657 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:43.252251    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gv4qj" event={"ID":"77b0599f-2680-4374-be49-1c2dd34e2618","Type":"ContainerStarted","Data":"ef6733ca4562221dd77a4a32693d7249b787a2400abcee4cc1616a73c8db740f"}
Oct 31 12:27:43.276476 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:43.276427    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-gv4qj" podStartSLOduration=2.348228386 podStartE2EDuration="4.276412764s" totalImagesPullingTime="1.850009214s" totalInitContainerRuntime="78.175164ms" isStatefulPod=true podCreationTimestamp="2026-10-31 12:27:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:27:43.275086214 +0000 UTC m=+26.378153937" watchObservedRunningTime="2026-10-31 12:27:43.276412764 +0000 UTC m=+26.379480473"
Oct 31 12:27:47.545743 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:47.545711    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6dd647c7b7-9255m"
Oct 31 12:27:47.846153 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:47.846080    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-tdrmt"
Oct 31 12:27:53.616501 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:53.616459    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6dd647c7b7-9255m"]
Oct 31 12:27:55.938909 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:55.938837    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-6cfdbfcdc4-2rq84" podUID="9083c35a-9739-4f3e-8415-bad1e6433d40" containerName="registry" containerID="cri-o://8e04f4d0cbdb741e50f23a4a307636a4dbfca396ef8fbdec5f701edf3edd342c" gracePeriod=30
Oct 31 12:27:56.199871 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:56.199808    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6cfdbfcdc4-2rq84"
Oct 31 12:27:56.276224 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:56.276192    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9083c35a-9739-4f3e-8415-bad1e6433d40-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9083c35a-9739-4f3e-8415-bad1e6433d40-image-registry-private-configuration\") pod \"9083c35a-9739-4f3e-8415-bad1e6433d40\" (UID: \"9083c35a-9739-4f3e-8415-bad1e6433d40\") "
Oct 31 12:27:56.276394 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:56.276249    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9083c35a-9739-4f3e-8415-bad1e6433d40-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9083c35a-9739-4f3e-8415-bad1e6433d40-ca-trust-extracted\") pod \"9083c35a-9739-4f3e-8415-bad1e6433d40\" (UID: \"9083c35a-9739-4f3e-8415-bad1e6433d40\") "
Oct 31 12:27:56.276394 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:56.276269    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9083c35a-9739-4f3e-8415-bad1e6433d40-registry-tls\" (UniqueName: \"kubernetes.io/projected/9083c35a-9739-4f3e-8415-bad1e6433d40-registry-tls\") pod \"9083c35a-9739-4f3e-8415-bad1e6433d40\" (UID: \"9083c35a-9739-4f3e-8415-bad1e6433d40\") "
Oct 31 12:27:56.276394 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:56.276303    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9083c35a-9739-4f3e-8415-bad1e6433d40-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9083c35a-9739-4f3e-8415-bad1e6433d40-installation-pull-secrets\") pod \"9083c35a-9739-4f3e-8415-bad1e6433d40\" (UID: \"9083c35a-9739-4f3e-8415-bad1e6433d40\") "
Oct 31 12:27:56.276394 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:56.276324    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9083c35a-9739-4f3e-8415-bad1e6433d40-kube-api-access-5xwrq\" (UniqueName: \"kubernetes.io/projected/9083c35a-9739-4f3e-8415-bad1e6433d40-kube-api-access-5xwrq\") pod \"9083c35a-9739-4f3e-8415-bad1e6433d40\" (UID: \"9083c35a-9739-4f3e-8415-bad1e6433d40\") "
Oct 31 12:27:56.276394 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:56.276346    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9083c35a-9739-4f3e-8415-bad1e6433d40-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9083c35a-9739-4f3e-8415-bad1e6433d40-registry-certificates\") pod \"9083c35a-9739-4f3e-8415-bad1e6433d40\" (UID: \"9083c35a-9739-4f3e-8415-bad1e6433d40\") "
Oct 31 12:27:56.276394 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:56.276391    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9083c35a-9739-4f3e-8415-bad1e6433d40-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9083c35a-9739-4f3e-8415-bad1e6433d40-trusted-ca\") pod \"9083c35a-9739-4f3e-8415-bad1e6433d40\" (UID: \"9083c35a-9739-4f3e-8415-bad1e6433d40\") "
Oct 31 12:27:56.276736 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:56.276431    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9083c35a-9739-4f3e-8415-bad1e6433d40-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9083c35a-9739-4f3e-8415-bad1e6433d40-bound-sa-token\") pod \"9083c35a-9739-4f3e-8415-bad1e6433d40\" (UID: \"9083c35a-9739-4f3e-8415-bad1e6433d40\") "
Oct 31 12:27:56.276909 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:56.276818    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9083c35a-9739-4f3e-8415-bad1e6433d40-registry-certificates" pod "9083c35a-9739-4f3e-8415-bad1e6433d40" (UID: "9083c35a-9739-4f3e-8415-bad1e6433d40"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:27:56.277059 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:56.276895    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9083c35a-9739-4f3e-8415-bad1e6433d40-trusted-ca" pod "9083c35a-9739-4f3e-8415-bad1e6433d40" (UID: "9083c35a-9739-4f3e-8415-bad1e6433d40"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:27:56.278972 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:56.278943    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9083c35a-9739-4f3e-8415-bad1e6433d40-bound-sa-token" pod "9083c35a-9739-4f3e-8415-bad1e6433d40" (UID: "9083c35a-9739-4f3e-8415-bad1e6433d40"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:27:56.279105 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:56.278974    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9083c35a-9739-4f3e-8415-bad1e6433d40-registry-tls" pod "9083c35a-9739-4f3e-8415-bad1e6433d40" (UID: "9083c35a-9739-4f3e-8415-bad1e6433d40"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:27:56.279105 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:56.279036    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9083c35a-9739-4f3e-8415-bad1e6433d40-image-registry-private-configuration" pod "9083c35a-9739-4f3e-8415-bad1e6433d40" (UID: "9083c35a-9739-4f3e-8415-bad1e6433d40"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:27:56.279295 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:56.279271    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9083c35a-9739-4f3e-8415-bad1e6433d40-installation-pull-secrets" pod "9083c35a-9739-4f3e-8415-bad1e6433d40" (UID: "9083c35a-9739-4f3e-8415-bad1e6433d40"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:27:56.279355 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:56.279304    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9083c35a-9739-4f3e-8415-bad1e6433d40-kube-api-access-5xwrq" pod "9083c35a-9739-4f3e-8415-bad1e6433d40" (UID: "9083c35a-9739-4f3e-8415-bad1e6433d40"). InnerVolumeSpecName "kube-api-access-5xwrq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:27:56.283242 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:56.283217    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9083c35a-9739-4f3e-8415-bad1e6433d40-ca-trust-extracted" pod "9083c35a-9739-4f3e-8415-bad1e6433d40" (UID: "9083c35a-9739-4f3e-8415-bad1e6433d40"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:27:56.309054 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:56.309024    5060 generic.go:356] "Generic (PLEG): container finished" podID="9083c35a-9739-4f3e-8415-bad1e6433d40" containerID="8e04f4d0cbdb741e50f23a4a307636a4dbfca396ef8fbdec5f701edf3edd342c" exitCode=0
Oct 31 12:27:56.309220 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:56.309069    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6cfdbfcdc4-2rq84"
Oct 31 12:27:56.309220 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:56.309100    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6cfdbfcdc4-2rq84" event={"ID":"9083c35a-9739-4f3e-8415-bad1e6433d40","Type":"ContainerDied","Data":"8e04f4d0cbdb741e50f23a4a307636a4dbfca396ef8fbdec5f701edf3edd342c"}
Oct 31 12:27:56.309220 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:56.309162    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6cfdbfcdc4-2rq84" event={"ID":"9083c35a-9739-4f3e-8415-bad1e6433d40","Type":"ContainerDied","Data":"a39d94bd466572faf6c06458dcf2f18a2ef1ccd3b298e363f6a0b042eeac36a3"}
Oct 31 12:27:56.309220 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:56.309189    5060 scope.go:122] "RemoveContainer" containerID="8e04f4d0cbdb741e50f23a4a307636a4dbfca396ef8fbdec5f701edf3edd342c"
Oct 31 12:27:56.318674 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:56.318635    5060 scope.go:122] "RemoveContainer" containerID="8e04f4d0cbdb741e50f23a4a307636a4dbfca396ef8fbdec5f701edf3edd342c"
Oct 31 12:27:56.318924 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:27:56.318906    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e04f4d0cbdb741e50f23a4a307636a4dbfca396ef8fbdec5f701edf3edd342c\": container with ID starting with 8e04f4d0cbdb741e50f23a4a307636a4dbfca396ef8fbdec5f701edf3edd342c not found: ID does not exist" containerID="8e04f4d0cbdb741e50f23a4a307636a4dbfca396ef8fbdec5f701edf3edd342c"
Oct 31 12:27:56.318971 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:56.318932    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e04f4d0cbdb741e50f23a4a307636a4dbfca396ef8fbdec5f701edf3edd342c"} err="failed to get container status \"8e04f4d0cbdb741e50f23a4a307636a4dbfca396ef8fbdec5f701edf3edd342c\": rpc error: code = NotFound desc = could not find container \"8e04f4d0cbdb741e50f23a4a307636a4dbfca396ef8fbdec5f701edf3edd342c\": container with ID starting with 8e04f4d0cbdb741e50f23a4a307636a4dbfca396ef8fbdec5f701edf3edd342c not found: ID does not exist"
Oct 31 12:27:56.338787 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:56.338759    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6cfdbfcdc4-2rq84"]
Oct 31 12:27:56.344673 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:56.344646    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6cfdbfcdc4-2rq84"]
Oct 31 12:27:56.377022 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:56.376991    5060 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9083c35a-9739-4f3e-8415-bad1e6433d40-image-registry-private-configuration\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:27:56.377022 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:56.377020    5060 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9083c35a-9739-4f3e-8415-bad1e6433d40-ca-trust-extracted\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:27:56.377208 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:56.377033    5060 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9083c35a-9739-4f3e-8415-bad1e6433d40-registry-tls\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:27:56.377208 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:56.377042    5060 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9083c35a-9739-4f3e-8415-bad1e6433d40-installation-pull-secrets\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:27:56.377208 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:56.377052    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xwrq\" (UniqueName: \"kubernetes.io/projected/9083c35a-9739-4f3e-8415-bad1e6433d40-kube-api-access-5xwrq\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:27:56.377208 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:56.377060    5060 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9083c35a-9739-4f3e-8415-bad1e6433d40-registry-certificates\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:27:56.377208 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:56.377069    5060 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9083c35a-9739-4f3e-8415-bad1e6433d40-trusted-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:27:56.377208 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:56.377078    5060 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9083c35a-9739-4f3e-8415-bad1e6433d40-bound-sa-token\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:27:57.045694 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:27:57.045660    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9083c35a-9739-4f3e-8415-bad1e6433d40" path="/var/lib/kubelet/pods/9083c35a-9739-4f3e-8415-bad1e6433d40/volumes"
Oct 31 12:27:59.920166 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:27:59.920076    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7phb" containerName="extractor"
Oct 31 12:28:03.462258 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:28:03.462221    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-87ncb-cnk5g"]
Oct 31 12:28:03.462806 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:28:03.462450    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-87ncb-cnk5g" podUID="eb20e317-3ebe-4f5c-9b37-1de9ea5f063f" containerName="insights-gathering" containerID="cri-o://32de846f50254cf1d3b4006aa0d3d1ce3771e004167002ba1b955de5afb8eebe" gracePeriod=30
Oct 31 12:28:03.724385 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:28:03.724320    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-87ncb-cnk5g"
Oct 31 12:28:03.843208 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:28:03.843166    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb20e317-3ebe-4f5c-9b37-1de9ea5f063f-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb20e317-3ebe-4f5c-9b37-1de9ea5f063f-trusted-ca-bundle\") pod \"eb20e317-3ebe-4f5c-9b37-1de9ea5f063f\" (UID: \"eb20e317-3ebe-4f5c-9b37-1de9ea5f063f\") "
Oct 31 12:28:03.843410 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:28:03.843226    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb20e317-3ebe-4f5c-9b37-1de9ea5f063f-kube-api-access-kvtwq\" (UniqueName: \"kubernetes.io/projected/eb20e317-3ebe-4f5c-9b37-1de9ea5f063f-kube-api-access-kvtwq\") pod \"eb20e317-3ebe-4f5c-9b37-1de9ea5f063f\" (UID: \"eb20e317-3ebe-4f5c-9b37-1de9ea5f063f\") "
Oct 31 12:28:03.843410 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:28:03.843250    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb20e317-3ebe-4f5c-9b37-1de9ea5f063f-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb20e317-3ebe-4f5c-9b37-1de9ea5f063f-service-ca-bundle\") pod \"eb20e317-3ebe-4f5c-9b37-1de9ea5f063f\" (UID: \"eb20e317-3ebe-4f5c-9b37-1de9ea5f063f\") "
Oct 31 12:28:03.843410 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:28:03.843360    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb20e317-3ebe-4f5c-9b37-1de9ea5f063f-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/eb20e317-3ebe-4f5c-9b37-1de9ea5f063f-archives-path\") pod \"eb20e317-3ebe-4f5c-9b37-1de9ea5f063f\" (UID: \"eb20e317-3ebe-4f5c-9b37-1de9ea5f063f\") "
Oct 31 12:28:03.843715 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:28:03.843689    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb20e317-3ebe-4f5c-9b37-1de9ea5f063f-service-ca-bundle" pod "eb20e317-3ebe-4f5c-9b37-1de9ea5f063f" (UID: "eb20e317-3ebe-4f5c-9b37-1de9ea5f063f"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:28:03.843773 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:28:03.843712    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb20e317-3ebe-4f5c-9b37-1de9ea5f063f-archives-path" pod "eb20e317-3ebe-4f5c-9b37-1de9ea5f063f" (UID: "eb20e317-3ebe-4f5c-9b37-1de9ea5f063f"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:28:03.843773 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:28:03.843692    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb20e317-3ebe-4f5c-9b37-1de9ea5f063f-trusted-ca-bundle" pod "eb20e317-3ebe-4f5c-9b37-1de9ea5f063f" (UID: "eb20e317-3ebe-4f5c-9b37-1de9ea5f063f"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:28:03.845811 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:28:03.845789    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb20e317-3ebe-4f5c-9b37-1de9ea5f063f-kube-api-access-kvtwq" pod "eb20e317-3ebe-4f5c-9b37-1de9ea5f063f" (UID: "eb20e317-3ebe-4f5c-9b37-1de9ea5f063f"). InnerVolumeSpecName "kube-api-access-kvtwq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:28:03.944185 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:28:03.944147    5060 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb20e317-3ebe-4f5c-9b37-1de9ea5f063f-trusted-ca-bundle\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:28:03.944185 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:28:03.944179    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kvtwq\" (UniqueName: \"kubernetes.io/projected/eb20e317-3ebe-4f5c-9b37-1de9ea5f063f-kube-api-access-kvtwq\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:28:03.944185 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:28:03.944191    5060 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb20e317-3ebe-4f5c-9b37-1de9ea5f063f-service-ca-bundle\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:28:03.944413 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:28:03.944200    5060 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/eb20e317-3ebe-4f5c-9b37-1de9ea5f063f-archives-path\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:28:04.342440 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:28:04.342401    5060 generic.go:356] "Generic (PLEG): container finished" podID="eb20e317-3ebe-4f5c-9b37-1de9ea5f063f" containerID="32de846f50254cf1d3b4006aa0d3d1ce3771e004167002ba1b955de5afb8eebe" exitCode=2
Oct 31 12:28:04.342642 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:28:04.342445    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-87ncb-cnk5g"
Oct 31 12:28:04.342642 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:28:04.342484    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-87ncb-cnk5g" event={"ID":"eb20e317-3ebe-4f5c-9b37-1de9ea5f063f","Type":"ContainerDied","Data":"32de846f50254cf1d3b4006aa0d3d1ce3771e004167002ba1b955de5afb8eebe"}
Oct 31 12:28:04.342642 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:28:04.342556    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-87ncb-cnk5g" event={"ID":"eb20e317-3ebe-4f5c-9b37-1de9ea5f063f","Type":"ContainerDied","Data":"8738d804fa86e15f0597349bd890e4bfa1c5c1415beaacbab01aad9c000fbd0a"}
Oct 31 12:28:04.342642 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:28:04.342578    5060 scope.go:122] "RemoveContainer" containerID="32de846f50254cf1d3b4006aa0d3d1ce3771e004167002ba1b955de5afb8eebe"
Oct 31 12:28:04.351194 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:28:04.351178    5060 scope.go:122] "RemoveContainer" containerID="32de846f50254cf1d3b4006aa0d3d1ce3771e004167002ba1b955de5afb8eebe"
Oct 31 12:28:04.351414 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:28:04.351399    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32de846f50254cf1d3b4006aa0d3d1ce3771e004167002ba1b955de5afb8eebe\": container with ID starting with 32de846f50254cf1d3b4006aa0d3d1ce3771e004167002ba1b955de5afb8eebe not found: ID does not exist" containerID="32de846f50254cf1d3b4006aa0d3d1ce3771e004167002ba1b955de5afb8eebe"
Oct 31 12:28:04.351467 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:28:04.351422    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32de846f50254cf1d3b4006aa0d3d1ce3771e004167002ba1b955de5afb8eebe"} err="failed to get container status \"32de846f50254cf1d3b4006aa0d3d1ce3771e004167002ba1b955de5afb8eebe\": rpc error: code = NotFound desc = could not find container \"32de846f50254cf1d3b4006aa0d3d1ce3771e004167002ba1b955de5afb8eebe\": container with ID starting with 32de846f50254cf1d3b4006aa0d3d1ce3771e004167002ba1b955de5afb8eebe not found: ID does not exist"
Oct 31 12:28:04.362289 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:28:04.362260    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-87ncb-cnk5g"]
Oct 31 12:28:04.366711 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:28:04.366689    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-87ncb-cnk5g"]
Oct 31 12:28:05.045868 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:28:05.045836    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb20e317-3ebe-4f5c-9b37-1de9ea5f063f" path="/var/lib/kubelet/pods/eb20e317-3ebe-4f5c-9b37-1de9ea5f063f/volumes"
Oct 31 12:28:18.645971 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:28:18.645907    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-6dd647c7b7-9255m" podUID="0db116b4-d971-4966-bb05-db28d077b6fe" containerName="registry" containerID="cri-o://c66777009b60b7ce3008312915aab833aa19ed4537505808b5d66d593e39ba3a" gracePeriod=30
Oct 31 12:28:18.898095 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:28:18.898039    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6dd647c7b7-9255m"
Oct 31 12:28:18.969460 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:28:18.969428    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0db116b4-d971-4966-bb05-db28d077b6fe-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0db116b4-d971-4966-bb05-db28d077b6fe-registry-certificates\") pod \"0db116b4-d971-4966-bb05-db28d077b6fe\" (UID: \"0db116b4-d971-4966-bb05-db28d077b6fe\") "
Oct 31 12:28:18.969655 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:28:18.969486    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0db116b4-d971-4966-bb05-db28d077b6fe-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0db116b4-d971-4966-bb05-db28d077b6fe-trusted-ca\") pod \"0db116b4-d971-4966-bb05-db28d077b6fe\" (UID: \"0db116b4-d971-4966-bb05-db28d077b6fe\") "
Oct 31 12:28:18.969655 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:28:18.969526    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0db116b4-d971-4966-bb05-db28d077b6fe-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0db116b4-d971-4966-bb05-db28d077b6fe-installation-pull-secrets\") pod \"0db116b4-d971-4966-bb05-db28d077b6fe\" (UID: \"0db116b4-d971-4966-bb05-db28d077b6fe\") "
Oct 31 12:28:18.969655 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:28:18.969563    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0db116b4-d971-4966-bb05-db28d077b6fe-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0db116b4-d971-4966-bb05-db28d077b6fe-ca-trust-extracted\") pod \"0db116b4-d971-4966-bb05-db28d077b6fe\" (UID: \"0db116b4-d971-4966-bb05-db28d077b6fe\") "
Oct 31 12:28:18.969655 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:28:18.969602    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0db116b4-d971-4966-bb05-db28d077b6fe-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0db116b4-d971-4966-bb05-db28d077b6fe-image-registry-private-configuration\") pod \"0db116b4-d971-4966-bb05-db28d077b6fe\" (UID: \"0db116b4-d971-4966-bb05-db28d077b6fe\") "
Oct 31 12:28:18.969655 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:28:18.969633    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0db116b4-d971-4966-bb05-db28d077b6fe-registry-tls\" (UniqueName: \"kubernetes.io/projected/0db116b4-d971-4966-bb05-db28d077b6fe-registry-tls\") pod \"0db116b4-d971-4966-bb05-db28d077b6fe\" (UID: \"0db116b4-d971-4966-bb05-db28d077b6fe\") "
Oct 31 12:28:18.969923 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:28:18.969669    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0db116b4-d971-4966-bb05-db28d077b6fe-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0db116b4-d971-4966-bb05-db28d077b6fe-bound-sa-token\") pod \"0db116b4-d971-4966-bb05-db28d077b6fe\" (UID: \"0db116b4-d971-4966-bb05-db28d077b6fe\") "
Oct 31 12:28:18.969923 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:28:18.969698    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0db116b4-d971-4966-bb05-db28d077b6fe-kube-api-access-nlgsp\" (UniqueName: \"kubernetes.io/projected/0db116b4-d971-4966-bb05-db28d077b6fe-kube-api-access-nlgsp\") pod \"0db116b4-d971-4966-bb05-db28d077b6fe\" (UID: \"0db116b4-d971-4966-bb05-db28d077b6fe\") "
Oct 31 12:28:18.969923 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:28:18.969864    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0db116b4-d971-4966-bb05-db28d077b6fe-registry-certificates" pod "0db116b4-d971-4966-bb05-db28d077b6fe" (UID: "0db116b4-d971-4966-bb05-db28d077b6fe"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:28:18.969923 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:28:18.969907    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0db116b4-d971-4966-bb05-db28d077b6fe-trusted-ca" pod "0db116b4-d971-4966-bb05-db28d077b6fe" (UID: "0db116b4-d971-4966-bb05-db28d077b6fe"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:28:18.972142 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:28:18.972096    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0db116b4-d971-4966-bb05-db28d077b6fe-kube-api-access-nlgsp" pod "0db116b4-d971-4966-bb05-db28d077b6fe" (UID: "0db116b4-d971-4966-bb05-db28d077b6fe"). InnerVolumeSpecName "kube-api-access-nlgsp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:28:18.972249 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:28:18.972213    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0db116b4-d971-4966-bb05-db28d077b6fe-image-registry-private-configuration" pod "0db116b4-d971-4966-bb05-db28d077b6fe" (UID: "0db116b4-d971-4966-bb05-db28d077b6fe"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:28:18.972249 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:28:18.972231    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0db116b4-d971-4966-bb05-db28d077b6fe-registry-tls" pod "0db116b4-d971-4966-bb05-db28d077b6fe" (UID: "0db116b4-d971-4966-bb05-db28d077b6fe"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:28:18.972356 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:28:18.972281    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0db116b4-d971-4966-bb05-db28d077b6fe-installation-pull-secrets" pod "0db116b4-d971-4966-bb05-db28d077b6fe" (UID: "0db116b4-d971-4966-bb05-db28d077b6fe"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:28:18.972474 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:28:18.972454    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0db116b4-d971-4966-bb05-db28d077b6fe-bound-sa-token" pod "0db116b4-d971-4966-bb05-db28d077b6fe" (UID: "0db116b4-d971-4966-bb05-db28d077b6fe"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:28:18.976917 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:28:18.976894    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0db116b4-d971-4966-bb05-db28d077b6fe-ca-trust-extracted" pod "0db116b4-d971-4966-bb05-db28d077b6fe" (UID: "0db116b4-d971-4966-bb05-db28d077b6fe"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:28:19.070778 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:28:19.070746    5060 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0db116b4-d971-4966-bb05-db28d077b6fe-installation-pull-secrets\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:28:19.070778 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:28:19.070773    5060 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0db116b4-d971-4966-bb05-db28d077b6fe-ca-trust-extracted\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:28:19.070952 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:28:19.070785    5060 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0db116b4-d971-4966-bb05-db28d077b6fe-image-registry-private-configuration\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:28:19.070952 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:28:19.070795    5060 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0db116b4-d971-4966-bb05-db28d077b6fe-registry-tls\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:28:19.070952 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:28:19.070804    5060 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0db116b4-d971-4966-bb05-db28d077b6fe-bound-sa-token\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:28:19.070952 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:28:19.070812    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nlgsp\" (UniqueName: \"kubernetes.io/projected/0db116b4-d971-4966-bb05-db28d077b6fe-kube-api-access-nlgsp\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:28:19.070952 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:28:19.070821    5060 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0db116b4-d971-4966-bb05-db28d077b6fe-registry-certificates\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:28:19.070952 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:28:19.070830    5060 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0db116b4-d971-4966-bb05-db28d077b6fe-trusted-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:28:19.402502 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:28:19.402457    5060 generic.go:356] "Generic (PLEG): container finished" podID="0db116b4-d971-4966-bb05-db28d077b6fe" containerID="c66777009b60b7ce3008312915aab833aa19ed4537505808b5d66d593e39ba3a" exitCode=0
Oct 31 12:28:19.402665 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:28:19.402520    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6dd647c7b7-9255m"
Oct 31 12:28:19.402665 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:28:19.402548    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6dd647c7b7-9255m" event={"ID":"0db116b4-d971-4966-bb05-db28d077b6fe","Type":"ContainerDied","Data":"c66777009b60b7ce3008312915aab833aa19ed4537505808b5d66d593e39ba3a"}
Oct 31 12:28:19.402665 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:28:19.402606    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6dd647c7b7-9255m" event={"ID":"0db116b4-d971-4966-bb05-db28d077b6fe","Type":"ContainerDied","Data":"fd96831c9d48bccee78c39707230e6b7ce7bccd57174c000192520e9608f46c1"}
Oct 31 12:28:19.402665 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:28:19.402629    5060 scope.go:122] "RemoveContainer" containerID="c66777009b60b7ce3008312915aab833aa19ed4537505808b5d66d593e39ba3a"
Oct 31 12:28:19.411423 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:28:19.411401    5060 scope.go:122] "RemoveContainer" containerID="c66777009b60b7ce3008312915aab833aa19ed4537505808b5d66d593e39ba3a"
Oct 31 12:28:19.411726 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:28:19.411699    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c66777009b60b7ce3008312915aab833aa19ed4537505808b5d66d593e39ba3a\": container with ID starting with c66777009b60b7ce3008312915aab833aa19ed4537505808b5d66d593e39ba3a not found: ID does not exist" containerID="c66777009b60b7ce3008312915aab833aa19ed4537505808b5d66d593e39ba3a"
Oct 31 12:28:19.411810 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:28:19.411736    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c66777009b60b7ce3008312915aab833aa19ed4537505808b5d66d593e39ba3a"} err="failed to get container status \"c66777009b60b7ce3008312915aab833aa19ed4537505808b5d66d593e39ba3a\": rpc error: code = NotFound desc = could not find container \"c66777009b60b7ce3008312915aab833aa19ed4537505808b5d66d593e39ba3a\": container with ID starting with c66777009b60b7ce3008312915aab833aa19ed4537505808b5d66d593e39ba3a not found: ID does not exist"
Oct 31 12:28:19.419823 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:28:19.419797    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6dd647c7b7-9255m"]
Oct 31 12:28:19.425100 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:28:19.425079    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6dd647c7b7-9255m"]
Oct 31 12:28:21.045770 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:28:21.045736    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0db116b4-d971-4966-bb05-db28d077b6fe" path="/var/lib/kubelet/pods/0db116b4-d971-4966-bb05-db28d077b6fe/volumes"
Oct 31 12:28:29.920009 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:28:29.919962    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7phb" containerName="extractor"
Oct 31 12:28:47.509135 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:28:47.509053    5060 generic.go:356] "Generic (PLEG): container finished" podID="99a606a8-546b-4145-8bae-a95061241069" containerID="7ee4f366c61fbdc3c2de10a9399dd08e4a45086da64aadf86ea70e41c05ffafb" exitCode=0
Oct 31 12:28:47.509135 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:28:47.509110    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-c789s" event={"ID":"99a606a8-546b-4145-8bae-a95061241069","Type":"ContainerDied","Data":"7ee4f366c61fbdc3c2de10a9399dd08e4a45086da64aadf86ea70e41c05ffafb"}
Oct 31 12:28:47.509593 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:28:47.509392    5060 scope.go:122] "RemoveContainer" containerID="7ee4f366c61fbdc3c2de10a9399dd08e4a45086da64aadf86ea70e41c05ffafb"
Oct 31 12:28:48.514500 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:28:48.514464    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-c789s" event={"ID":"99a606a8-546b-4145-8bae-a95061241069","Type":"ContainerStarted","Data":"f9491318bbd9f445db89adc5de6467a6c876d696b0e2263bdd7cb94ab7b4f132"}
Oct 31 12:28:51.527914 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:28:51.527877    5060 generic.go:356] "Generic (PLEG): container finished" podID="febbc28e-bbcc-4961-bde8-09944d9ca680" containerID="3db3a0ce1da4d3c071e219120fef1b0e1e1c7f5f370abd27b3f588ba9bc84073" exitCode=0
Oct 31 12:28:51.528286 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:28:51.527943    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-vqsv9" event={"ID":"febbc28e-bbcc-4961-bde8-09944d9ca680","Type":"ContainerDied","Data":"3db3a0ce1da4d3c071e219120fef1b0e1e1c7f5f370abd27b3f588ba9bc84073"}
Oct 31 12:28:51.528286 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:28:51.528256    5060 scope.go:122] "RemoveContainer" containerID="3db3a0ce1da4d3c071e219120fef1b0e1e1c7f5f370abd27b3f588ba9bc84073"
Oct 31 12:28:52.533237 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:28:52.533194    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-vqsv9" event={"ID":"febbc28e-bbcc-4961-bde8-09944d9ca680","Type":"ContainerStarted","Data":"b90728d5db5c5726036de1b9cd905420c2134857f5a6dd5d15e59f934e03e5ba"}
Oct 31 12:28:57.553148 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:28:57.553111    5060 generic.go:356] "Generic (PLEG): container finished" podID="9f7f000c-c637-4b51-b4ad-805ea3205f29" containerID="43aaaeaf224e7d4f82f3f279bcb6fc07a9f000d3ef5cf008c7658e5e63713f75" exitCode=0
Oct 31 12:28:57.553557 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:28:57.553152    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-rrx9r-zvwfn" event={"ID":"9f7f000c-c637-4b51-b4ad-805ea3205f29","Type":"ContainerDied","Data":"43aaaeaf224e7d4f82f3f279bcb6fc07a9f000d3ef5cf008c7658e5e63713f75"}
Oct 31 12:28:58.691919 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:28:58.691890    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-rrx9r-zvwfn"
Oct 31 12:28:58.796189 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:28:58.796150    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f7f000c-c637-4b51-b4ad-805ea3205f29-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f7f000c-c637-4b51-b4ad-805ea3205f29-trusted-ca-bundle\") pod \"9f7f000c-c637-4b51-b4ad-805ea3205f29\" (UID: \"9f7f000c-c637-4b51-b4ad-805ea3205f29\") "
Oct 31 12:28:58.796189 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:28:58.796199    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f7f000c-c637-4b51-b4ad-805ea3205f29-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f7f000c-c637-4b51-b4ad-805ea3205f29-service-ca-bundle\") pod \"9f7f000c-c637-4b51-b4ad-805ea3205f29\" (UID: \"9f7f000c-c637-4b51-b4ad-805ea3205f29\") "
Oct 31 12:28:58.796446 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:28:58.796231    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f7f000c-c637-4b51-b4ad-805ea3205f29-kube-api-access-x7c87\" (UniqueName: \"kubernetes.io/projected/9f7f000c-c637-4b51-b4ad-805ea3205f29-kube-api-access-x7c87\") pod \"9f7f000c-c637-4b51-b4ad-805ea3205f29\" (UID: \"9f7f000c-c637-4b51-b4ad-805ea3205f29\") "
Oct 31 12:28:58.796446 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:28:58.796260    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f7f000c-c637-4b51-b4ad-805ea3205f29-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/9f7f000c-c637-4b51-b4ad-805ea3205f29-archives-path\") pod \"9f7f000c-c637-4b51-b4ad-805ea3205f29\" (UID: \"9f7f000c-c637-4b51-b4ad-805ea3205f29\") "
Oct 31 12:28:58.796720 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:28:58.796671    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f7f000c-c637-4b51-b4ad-805ea3205f29-service-ca-bundle" pod "9f7f000c-c637-4b51-b4ad-805ea3205f29" (UID: "9f7f000c-c637-4b51-b4ad-805ea3205f29"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:28:58.796837 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:28:58.796678    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f7f000c-c637-4b51-b4ad-805ea3205f29-archives-path" pod "9f7f000c-c637-4b51-b4ad-805ea3205f29" (UID: "9f7f000c-c637-4b51-b4ad-805ea3205f29"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:28:58.796837 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:28:58.796681    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f7f000c-c637-4b51-b4ad-805ea3205f29-trusted-ca-bundle" pod "9f7f000c-c637-4b51-b4ad-805ea3205f29" (UID: "9f7f000c-c637-4b51-b4ad-805ea3205f29"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:28:58.798364 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:28:58.798341    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f7f000c-c637-4b51-b4ad-805ea3205f29-kube-api-access-x7c87" pod "9f7f000c-c637-4b51-b4ad-805ea3205f29" (UID: "9f7f000c-c637-4b51-b4ad-805ea3205f29"). InnerVolumeSpecName "kube-api-access-x7c87". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:28:58.896807 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:28:58.896731    5060 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f7f000c-c637-4b51-b4ad-805ea3205f29-trusted-ca-bundle\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:28:58.896807 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:28:58.896757    5060 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f7f000c-c637-4b51-b4ad-805ea3205f29-service-ca-bundle\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:28:58.896807 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:28:58.896766    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x7c87\" (UniqueName: \"kubernetes.io/projected/9f7f000c-c637-4b51-b4ad-805ea3205f29-kube-api-access-x7c87\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:28:58.896807 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:28:58.896777    5060 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/9f7f000c-c637-4b51-b4ad-805ea3205f29-archives-path\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:28:59.561825 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:28:59.561793    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-rrx9r-zvwfn"
Oct 31 12:28:59.561825 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:28:59.561809    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-rrx9r-zvwfn" event={"ID":"9f7f000c-c637-4b51-b4ad-805ea3205f29","Type":"ContainerDied","Data":"4f14c40ac2a6f12a28385a283a0f92f83512e34ca9413db9813bec0c36c6d237"}
Oct 31 12:28:59.562041 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:28:59.561860    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4f14c40ac2a6f12a28385a283a0f92f83512e34ca9413db9813bec0c36c6d237"
Oct 31 12:28:59.919291 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:28:59.919208    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7phb" containerName="extractor"
Oct 31 12:29:29.919864 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:29:29.919774    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7phb" containerName="extractor"
Oct 31 12:29:59.919417 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:29:59.919372    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7phb" containerName="extractor"
Oct 31 12:30:29.918573 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:30:29.918528    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7phb" containerName="extractor"
Oct 31 12:30:59.919902 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:30:59.919828    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7phb" containerName="extractor"
Oct 31 12:31:29.919642 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:31:29.919606    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7phb" containerName="extractor"
Oct 31 12:31:36.895793 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:36.895753    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644g8g7n"]
Oct 31 12:31:36.898213 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:36.896053    5060 cpu_manager.go:414] "containerMap: removing container" podUID="0db116b4-d971-4966-bb05-db28d077b6fe" containerName="registry"
Oct 31 12:31:36.898213 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:36.896064    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0db116b4-d971-4966-bb05-db28d077b6fe" containerName="registry"
Oct 31 12:31:36.898213 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:36.896075    5060 cpu_manager.go:414] "containerMap: removing container" podUID="9f7f000c-c637-4b51-b4ad-805ea3205f29" containerName="insights-gathering"
Oct 31 12:31:36.898213 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:36.896081    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f7f000c-c637-4b51-b4ad-805ea3205f29" containerName="insights-gathering"
Oct 31 12:31:36.898213 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:36.896099    5060 cpu_manager.go:414] "containerMap: removing container" podUID="9083c35a-9739-4f3e-8415-bad1e6433d40" containerName="registry"
Oct 31 12:31:36.898213 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:36.896104    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9083c35a-9739-4f3e-8415-bad1e6433d40" containerName="registry"
Oct 31 12:31:36.898213 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:36.896117    5060 cpu_manager.go:414] "containerMap: removing container" podUID="eb20e317-3ebe-4f5c-9b37-1de9ea5f063f" containerName="insights-gathering"
Oct 31 12:31:36.898213 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:36.896122    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb20e317-3ebe-4f5c-9b37-1de9ea5f063f" containerName="insights-gathering"
Oct 31 12:31:36.898213 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:36.896176    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="9083c35a-9739-4f3e-8415-bad1e6433d40" containerName="registry"
Oct 31 12:31:36.898213 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:36.896184    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb20e317-3ebe-4f5c-9b37-1de9ea5f063f" containerName="insights-gathering"
Oct 31 12:31:36.898213 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:36.896194    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="0db116b4-d971-4966-bb05-db28d077b6fe" containerName="registry"
Oct 31 12:31:36.898213 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:36.896199    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f7f000c-c637-4b51-b4ad-805ea3205f29" containerName="insights-gathering"
Oct 31 12:31:36.899153 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:36.899138    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644g8g7n"
Oct 31 12:31:36.905932 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:36.905911    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Oct 31 12:31:36.906824 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:36.906808    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Oct 31 12:31:36.906885 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:36.906810    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-m8fdg\""
Oct 31 12:31:36.912596 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:36.912568    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644g8g7n"]
Oct 31 12:31:37.051203 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:37.051167    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a610abd9-13b7-493f-84a5-e6cb1cbd1262-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644g8g7n\" (UID: \"a610abd9-13b7-493f-84a5-e6cb1cbd1262\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644g8g7n"
Oct 31 12:31:37.051203 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:37.051205    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a610abd9-13b7-493f-84a5-e6cb1cbd1262-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644g8g7n\" (UID: \"a610abd9-13b7-493f-84a5-e6cb1cbd1262\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644g8g7n"
Oct 31 12:31:37.051409 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:37.051241    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bsg9m\" (UniqueName: \"kubernetes.io/projected/a610abd9-13b7-493f-84a5-e6cb1cbd1262-kube-api-access-bsg9m\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644g8g7n\" (UID: \"a610abd9-13b7-493f-84a5-e6cb1cbd1262\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644g8g7n"
Oct 31 12:31:37.152101 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:37.152016    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a610abd9-13b7-493f-84a5-e6cb1cbd1262-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644g8g7n\" (UID: \"a610abd9-13b7-493f-84a5-e6cb1cbd1262\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644g8g7n"
Oct 31 12:31:37.152101 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:37.152056    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a610abd9-13b7-493f-84a5-e6cb1cbd1262-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644g8g7n\" (UID: \"a610abd9-13b7-493f-84a5-e6cb1cbd1262\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644g8g7n"
Oct 31 12:31:37.152284 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:37.152114    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsg9m\" (UniqueName: \"kubernetes.io/projected/a610abd9-13b7-493f-84a5-e6cb1cbd1262-kube-api-access-bsg9m\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644g8g7n\" (UID: \"a610abd9-13b7-493f-84a5-e6cb1cbd1262\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644g8g7n"
Oct 31 12:31:37.152432 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:37.152411    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a610abd9-13b7-493f-84a5-e6cb1cbd1262-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644g8g7n\" (UID: \"a610abd9-13b7-493f-84a5-e6cb1cbd1262\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644g8g7n"
Oct 31 12:31:37.152517 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:37.152475    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a610abd9-13b7-493f-84a5-e6cb1cbd1262-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644g8g7n\" (UID: \"a610abd9-13b7-493f-84a5-e6cb1cbd1262\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644g8g7n"
Oct 31 12:31:37.164715 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:37.164685    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bsg9m\" (UniqueName: \"kubernetes.io/projected/a610abd9-13b7-493f-84a5-e6cb1cbd1262-kube-api-access-bsg9m\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644g8g7n\" (UID: \"a610abd9-13b7-493f-84a5-e6cb1cbd1262\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644g8g7n"
Oct 31 12:31:37.215132 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:37.215096    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644g8g7n"
Oct 31 12:31:37.564939 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:37.564813    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644g8g7n"]
Oct 31 12:31:37.567627 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:31:37.567598    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda610abd9_13b7_493f_84a5_e6cb1cbd1262.slice/crio-bbfb1defee389366342e1641b692af20013c4a633f2eefcaa534a1b38dbd2d9c WatchSource:0}: Error finding container bbfb1defee389366342e1641b692af20013c4a633f2eefcaa534a1b38dbd2d9c: Status 404 returned error can't find the container with id bbfb1defee389366342e1641b692af20013c4a633f2eefcaa534a1b38dbd2d9c
Oct 31 12:31:37.775108 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:37.775069    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cw274p"]
Oct 31 12:31:37.779452 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:37.779429    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cw274p"
Oct 31 12:31:37.790142 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:37.790116    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-6mwtw\""
Oct 31 12:31:37.791274 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:37.791250    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cw274p"]
Oct 31 12:31:37.957433 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:37.957352    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5fmq\" (UniqueName: \"kubernetes.io/projected/00a1f4b5-4e4e-42b8-b539-f4da4db2a558-kube-api-access-f5fmq\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cw274p\" (UID: \"00a1f4b5-4e4e-42b8-b539-f4da4db2a558\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cw274p"
Oct 31 12:31:37.957433 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:37.957395    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/00a1f4b5-4e4e-42b8-b539-f4da4db2a558-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cw274p\" (UID: \"00a1f4b5-4e4e-42b8-b539-f4da4db2a558\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cw274p"
Oct 31 12:31:37.957841 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:37.957507    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/00a1f4b5-4e4e-42b8-b539-f4da4db2a558-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cw274p\" (UID: \"00a1f4b5-4e4e-42b8-b539-f4da4db2a558\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cw274p"
Oct 31 12:31:38.057944 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:38.057905    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5fmq\" (UniqueName: \"kubernetes.io/projected/00a1f4b5-4e4e-42b8-b539-f4da4db2a558-kube-api-access-f5fmq\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cw274p\" (UID: \"00a1f4b5-4e4e-42b8-b539-f4da4db2a558\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cw274p"
Oct 31 12:31:38.058120 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:38.057954    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/00a1f4b5-4e4e-42b8-b539-f4da4db2a558-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cw274p\" (UID: \"00a1f4b5-4e4e-42b8-b539-f4da4db2a558\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cw274p"
Oct 31 12:31:38.058120 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:38.058007    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/00a1f4b5-4e4e-42b8-b539-f4da4db2a558-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cw274p\" (UID: \"00a1f4b5-4e4e-42b8-b539-f4da4db2a558\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cw274p"
Oct 31 12:31:38.058428 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:38.058408    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/00a1f4b5-4e4e-42b8-b539-f4da4db2a558-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cw274p\" (UID: \"00a1f4b5-4e4e-42b8-b539-f4da4db2a558\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cw274p"
Oct 31 12:31:38.058485 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:38.058444    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/00a1f4b5-4e4e-42b8-b539-f4da4db2a558-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cw274p\" (UID: \"00a1f4b5-4e4e-42b8-b539-f4da4db2a558\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cw274p"
Oct 31 12:31:38.070917 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:38.070892    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5fmq\" (UniqueName: \"kubernetes.io/projected/00a1f4b5-4e4e-42b8-b539-f4da4db2a558-kube-api-access-f5fmq\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cw274p\" (UID: \"00a1f4b5-4e4e-42b8-b539-f4da4db2a558\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cw274p"
Oct 31 12:31:38.096632 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:38.096605    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cw274p"
Oct 31 12:31:38.167280 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:38.167245    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644g8g7n" event={"ID":"a610abd9-13b7-493f-84a5-e6cb1cbd1262","Type":"ContainerStarted","Data":"bbfb1defee389366342e1641b692af20013c4a633f2eefcaa534a1b38dbd2d9c"}
Oct 31 12:31:38.374222 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:38.374189    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aprjf2"]
Oct 31 12:31:38.378508 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:38.378468    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aprjf2"
Oct 31 12:31:38.380975 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:38.380951    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cw274p"]
Oct 31 12:31:38.386303 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:38.386265    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-s8knk\""
Oct 31 12:31:38.388706 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:38.388681    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aprjf2"]
Oct 31 12:31:38.470296 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:38.470259    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8pxmf"]
Oct 31 12:31:38.473793 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:38.473770    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8pxmf"
Oct 31 12:31:38.481243 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:38.481204    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-ph9jd\""
Oct 31 12:31:38.487917 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:38.487857    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8pxmf"]
Oct 31 12:31:38.563364 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:38.563331    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dzb6\" (UniqueName: \"kubernetes.io/projected/b6f12f5a-911b-4180-99f4-c6d70e670320-kube-api-access-9dzb6\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aprjf2\" (UID: \"b6f12f5a-911b-4180-99f4-c6d70e670320\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aprjf2"
Oct 31 12:31:38.563561 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:38.563383    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b6f12f5a-911b-4180-99f4-c6d70e670320-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aprjf2\" (UID: \"b6f12f5a-911b-4180-99f4-c6d70e670320\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aprjf2"
Oct 31 12:31:38.563561 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:38.563539    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b6f12f5a-911b-4180-99f4-c6d70e670320-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aprjf2\" (UID: \"b6f12f5a-911b-4180-99f4-c6d70e670320\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aprjf2"
Oct 31 12:31:38.580716 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:38.580673    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qr8zz"]
Oct 31 12:31:38.584468 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:38.584444    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qr8zz"
Oct 31 12:31:38.588961 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:38.588939    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-wk9fz\""
Oct 31 12:31:38.594124 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:38.594098    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qr8zz"]
Oct 31 12:31:38.664669 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:38.664579    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dzb6\" (UniqueName: \"kubernetes.io/projected/b6f12f5a-911b-4180-99f4-c6d70e670320-kube-api-access-9dzb6\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aprjf2\" (UID: \"b6f12f5a-911b-4180-99f4-c6d70e670320\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aprjf2"
Oct 31 12:31:38.664669 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:38.664637    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b6f12f5a-911b-4180-99f4-c6d70e670320-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aprjf2\" (UID: \"b6f12f5a-911b-4180-99f4-c6d70e670320\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aprjf2"
Oct 31 12:31:38.664891 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:38.664674    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5ba15213-2643-474d-84f3-50a443cd9088-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8pxmf\" (UID: \"5ba15213-2643-474d-84f3-50a443cd9088\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8pxmf"
Oct 31 12:31:38.664891 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:38.664696    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5ba15213-2643-474d-84f3-50a443cd9088-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8pxmf\" (UID: \"5ba15213-2643-474d-84f3-50a443cd9088\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8pxmf"
Oct 31 12:31:38.664891 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:38.664717    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lsr4f\" (UniqueName: \"kubernetes.io/projected/5ba15213-2643-474d-84f3-50a443cd9088-kube-api-access-lsr4f\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8pxmf\" (UID: \"5ba15213-2643-474d-84f3-50a443cd9088\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8pxmf"
Oct 31 12:31:38.664891 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:38.664806    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b6f12f5a-911b-4180-99f4-c6d70e670320-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aprjf2\" (UID: \"b6f12f5a-911b-4180-99f4-c6d70e670320\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aprjf2"
Oct 31 12:31:38.665265 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:38.665232    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b6f12f5a-911b-4180-99f4-c6d70e670320-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aprjf2\" (UID: \"b6f12f5a-911b-4180-99f4-c6d70e670320\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aprjf2"
Oct 31 12:31:38.665396 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:38.665271    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b6f12f5a-911b-4180-99f4-c6d70e670320-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aprjf2\" (UID: \"b6f12f5a-911b-4180-99f4-c6d70e670320\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aprjf2"
Oct 31 12:31:38.679368 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:38.679338    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dzb6\" (UniqueName: \"kubernetes.io/projected/b6f12f5a-911b-4180-99f4-c6d70e670320-kube-api-access-9dzb6\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aprjf2\" (UID: \"b6f12f5a-911b-4180-99f4-c6d70e670320\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aprjf2"
Oct 31 12:31:38.694877 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:38.694837    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mlqlx"]
Oct 31 12:31:38.698949 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:38.698925    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mlqlx"
Oct 31 12:31:38.703299 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:38.703273    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-ch26l\""
Oct 31 12:31:38.707237 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:38.707212    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aprjf2"
Oct 31 12:31:38.710518 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:38.710467    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mlqlx"]
Oct 31 12:31:38.767388 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:38.766667    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b4836388-df04-4576-80b8-4ecd90bbb1fe-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qr8zz\" (UID: \"b4836388-df04-4576-80b8-4ecd90bbb1fe\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qr8zz"
Oct 31 12:31:38.767388 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:38.766739    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b4836388-df04-4576-80b8-4ecd90bbb1fe-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qr8zz\" (UID: \"b4836388-df04-4576-80b8-4ecd90bbb1fe\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qr8zz"
Oct 31 12:31:38.767388 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:38.766769    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfdwz\" (UniqueName: \"kubernetes.io/projected/b4836388-df04-4576-80b8-4ecd90bbb1fe-kube-api-access-cfdwz\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qr8zz\" (UID: \"b4836388-df04-4576-80b8-4ecd90bbb1fe\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qr8zz"
Oct 31 12:31:38.767388 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:38.766815    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5ba15213-2643-474d-84f3-50a443cd9088-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8pxmf\" (UID: \"5ba15213-2643-474d-84f3-50a443cd9088\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8pxmf"
Oct 31 12:31:38.767388 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:38.766834    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5ba15213-2643-474d-84f3-50a443cd9088-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8pxmf\" (UID: \"5ba15213-2643-474d-84f3-50a443cd9088\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8pxmf"
Oct 31 12:31:38.767388 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:38.766850    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lsr4f\" (UniqueName: \"kubernetes.io/projected/5ba15213-2643-474d-84f3-50a443cd9088-kube-api-access-lsr4f\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8pxmf\" (UID: \"5ba15213-2643-474d-84f3-50a443cd9088\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8pxmf"
Oct 31 12:31:38.767388 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:38.767281    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5ba15213-2643-474d-84f3-50a443cd9088-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8pxmf\" (UID: \"5ba15213-2643-474d-84f3-50a443cd9088\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8pxmf"
Oct 31 12:31:38.767388 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:38.767351    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5ba15213-2643-474d-84f3-50a443cd9088-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8pxmf\" (UID: \"5ba15213-2643-474d-84f3-50a443cd9088\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8pxmf"
Oct 31 12:31:38.781381 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:38.781350    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsr4f\" (UniqueName: \"kubernetes.io/projected/5ba15213-2643-474d-84f3-50a443cd9088-kube-api-access-lsr4f\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8pxmf\" (UID: \"5ba15213-2643-474d-84f3-50a443cd9088\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8pxmf"
Oct 31 12:31:38.793677 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:38.792878    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8pxmf"
Oct 31 12:31:38.868181 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:38.868145    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfdwz\" (UniqueName: \"kubernetes.io/projected/b4836388-df04-4576-80b8-4ecd90bbb1fe-kube-api-access-cfdwz\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qr8zz\" (UID: \"b4836388-df04-4576-80b8-4ecd90bbb1fe\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qr8zz"
Oct 31 12:31:38.868319 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:38.868233    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1bc5f088-e98b-4353-9512-39a3b11b9aba-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mlqlx\" (UID: \"1bc5f088-e98b-4353-9512-39a3b11b9aba\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mlqlx"
Oct 31 12:31:38.868319 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:38.868276    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1bc5f088-e98b-4353-9512-39a3b11b9aba-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mlqlx\" (UID: \"1bc5f088-e98b-4353-9512-39a3b11b9aba\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mlqlx"
Oct 31 12:31:38.868319 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:38.868311    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b4836388-df04-4576-80b8-4ecd90bbb1fe-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qr8zz\" (UID: \"b4836388-df04-4576-80b8-4ecd90bbb1fe\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qr8zz"
Oct 31 12:31:38.868530 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:38.868344    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b4836388-df04-4576-80b8-4ecd90bbb1fe-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qr8zz\" (UID: \"b4836388-df04-4576-80b8-4ecd90bbb1fe\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qr8zz"
Oct 31 12:31:38.868530 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:38.868368    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pl69q\" (UniqueName: \"kubernetes.io/projected/1bc5f088-e98b-4353-9512-39a3b11b9aba-kube-api-access-pl69q\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mlqlx\" (UID: \"1bc5f088-e98b-4353-9512-39a3b11b9aba\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mlqlx"
Oct 31 12:31:38.869044 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:38.869019    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b4836388-df04-4576-80b8-4ecd90bbb1fe-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qr8zz\" (UID: \"b4836388-df04-4576-80b8-4ecd90bbb1fe\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qr8zz"
Oct 31 12:31:38.869253 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:38.869237    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b4836388-df04-4576-80b8-4ecd90bbb1fe-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qr8zz\" (UID: \"b4836388-df04-4576-80b8-4ecd90bbb1fe\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qr8zz"
Oct 31 12:31:38.882913 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:38.882887    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfdwz\" (UniqueName: \"kubernetes.io/projected/b4836388-df04-4576-80b8-4ecd90bbb1fe-kube-api-access-cfdwz\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qr8zz\" (UID: \"b4836388-df04-4576-80b8-4ecd90bbb1fe\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qr8zz"
Oct 31 12:31:38.903673 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:38.903637    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qr8zz"
Oct 31 12:31:38.973446 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:38.972320    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1bc5f088-e98b-4353-9512-39a3b11b9aba-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mlqlx\" (UID: \"1bc5f088-e98b-4353-9512-39a3b11b9aba\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mlqlx"
Oct 31 12:31:38.973446 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:38.972376    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1bc5f088-e98b-4353-9512-39a3b11b9aba-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mlqlx\" (UID: \"1bc5f088-e98b-4353-9512-39a3b11b9aba\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mlqlx"
Oct 31 12:31:38.973446 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:38.972453    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pl69q\" (UniqueName: \"kubernetes.io/projected/1bc5f088-e98b-4353-9512-39a3b11b9aba-kube-api-access-pl69q\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mlqlx\" (UID: \"1bc5f088-e98b-4353-9512-39a3b11b9aba\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mlqlx"
Oct 31 12:31:38.973446 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:38.973382    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1bc5f088-e98b-4353-9512-39a3b11b9aba-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mlqlx\" (UID: \"1bc5f088-e98b-4353-9512-39a3b11b9aba\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mlqlx"
Oct 31 12:31:38.974616 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:38.974592    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1bc5f088-e98b-4353-9512-39a3b11b9aba-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mlqlx\" (UID: \"1bc5f088-e98b-4353-9512-39a3b11b9aba\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mlqlx"
Oct 31 12:31:38.988615 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:38.988585    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pl69q\" (UniqueName: \"kubernetes.io/projected/1bc5f088-e98b-4353-9512-39a3b11b9aba-kube-api-access-pl69q\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mlqlx\" (UID: \"1bc5f088-e98b-4353-9512-39a3b11b9aba\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mlqlx"
Oct 31 12:31:38.995427 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:38.995396    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aprjf2"]
Oct 31 12:31:39.002327 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:31:39.002297    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6f12f5a_911b_4180_99f4_c6d70e670320.slice/crio-dfcda53187003d00f9dd74ec81886eb709ab48b10e2dffda542ef176e6dd5470 WatchSource:0}: Error finding container dfcda53187003d00f9dd74ec81886eb709ab48b10e2dffda542ef176e6dd5470: Status 404 returned error can't find the container with id dfcda53187003d00f9dd74ec81886eb709ab48b10e2dffda542ef176e6dd5470
Oct 31 12:31:39.023207 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:39.023172    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mlqlx"
Oct 31 12:31:39.087366 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:39.087332    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8pxmf"]
Oct 31 12:31:39.089354 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:31:39.089322    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ba15213_2643_474d_84f3_50a443cd9088.slice/crio-dfcb0b73999696833ef1a27ff38f2541988bb6dda2eb6de1d5815074b423dda4 WatchSource:0}: Error finding container dfcb0b73999696833ef1a27ff38f2541988bb6dda2eb6de1d5815074b423dda4: Status 404 returned error can't find the container with id dfcb0b73999696833ef1a27ff38f2541988bb6dda2eb6de1d5815074b423dda4
Oct 31 12:31:39.173807 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:39.173773    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aprjf2" event={"ID":"b6f12f5a-911b-4180-99f4-c6d70e670320","Type":"ContainerStarted","Data":"dfcda53187003d00f9dd74ec81886eb709ab48b10e2dffda542ef176e6dd5470"}
Oct 31 12:31:39.175730 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:39.175697    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8pxmf" event={"ID":"5ba15213-2643-474d-84f3-50a443cd9088","Type":"ContainerStarted","Data":"dfcb0b73999696833ef1a27ff38f2541988bb6dda2eb6de1d5815074b423dda4"}
Oct 31 12:31:39.177148 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:39.177118    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cw274p" event={"ID":"00a1f4b5-4e4e-42b8-b539-f4da4db2a558","Type":"ContainerStarted","Data":"e9b234b5f5bcbaf9879a72be48332e339568943b15fd276b74bcdee2ddb66c46"}
Oct 31 12:31:39.191920 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:39.191885    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qr8zz"]
Oct 31 12:31:39.196996 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:31:39.196966    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4836388_df04_4576_80b8_4ecd90bbb1fe.slice/crio-e04cbb2135b08e75762250c0a8c3e59c2342b925c2170a5ba15d8bb80b1aae9c WatchSource:0}: Error finding container e04cbb2135b08e75762250c0a8c3e59c2342b925c2170a5ba15d8bb80b1aae9c: Status 404 returned error can't find the container with id e04cbb2135b08e75762250c0a8c3e59c2342b925c2170a5ba15d8bb80b1aae9c
Oct 31 12:31:39.310260 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:39.310229    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mlqlx"]
Oct 31 12:31:39.313262 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:31:39.313229    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bc5f088_e98b_4353_9512_39a3b11b9aba.slice/crio-e27326091d5e35c3ff5815bd98775d6848d9bc4efb1d0824d3e183645a40f312 WatchSource:0}: Error finding container e27326091d5e35c3ff5815bd98775d6848d9bc4efb1d0824d3e183645a40f312: Status 404 returned error can't find the container with id e27326091d5e35c3ff5815bd98775d6848d9bc4efb1d0824d3e183645a40f312
Oct 31 12:31:40.186187 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:40.186092    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mlqlx" event={"ID":"1bc5f088-e98b-4353-9512-39a3b11b9aba","Type":"ContainerStarted","Data":"e27326091d5e35c3ff5815bd98775d6848d9bc4efb1d0824d3e183645a40f312"}
Oct 31 12:31:40.190527 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:40.190438    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qr8zz" event={"ID":"b4836388-df04-4576-80b8-4ecd90bbb1fe","Type":"ContainerStarted","Data":"e04cbb2135b08e75762250c0a8c3e59c2342b925c2170a5ba15d8bb80b1aae9c"}
Oct 31 12:31:41.670335 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:41.670288    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714vkpp"]
Oct 31 12:31:41.675231 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:41.674439    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714vkpp"
Oct 31 12:31:41.680739 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:41.679791    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-mvxms\""
Oct 31 12:31:41.693539 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:41.693377    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714vkpp"]
Oct 31 12:31:41.697132 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:41.696900    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/71e7abef-0f1f-4797-a4f6-1b6793083fb8-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714vkpp\" (UID: \"71e7abef-0f1f-4797-a4f6-1b6793083fb8\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714vkpp"
Oct 31 12:31:41.697132 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:41.696947    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zv8tf\" (UniqueName: \"kubernetes.io/projected/71e7abef-0f1f-4797-a4f6-1b6793083fb8-kube-api-access-zv8tf\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714vkpp\" (UID: \"71e7abef-0f1f-4797-a4f6-1b6793083fb8\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714vkpp"
Oct 31 12:31:41.697132 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:41.697016    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/71e7abef-0f1f-4797-a4f6-1b6793083fb8-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714vkpp\" (UID: \"71e7abef-0f1f-4797-a4f6-1b6793083fb8\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714vkpp"
Oct 31 12:31:41.798526 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:41.797610    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zv8tf\" (UniqueName: \"kubernetes.io/projected/71e7abef-0f1f-4797-a4f6-1b6793083fb8-kube-api-access-zv8tf\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714vkpp\" (UID: \"71e7abef-0f1f-4797-a4f6-1b6793083fb8\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714vkpp"
Oct 31 12:31:41.798526 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:41.797700    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/71e7abef-0f1f-4797-a4f6-1b6793083fb8-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714vkpp\" (UID: \"71e7abef-0f1f-4797-a4f6-1b6793083fb8\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714vkpp"
Oct 31 12:31:41.798526 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:41.797762    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/71e7abef-0f1f-4797-a4f6-1b6793083fb8-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714vkpp\" (UID: \"71e7abef-0f1f-4797-a4f6-1b6793083fb8\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714vkpp"
Oct 31 12:31:41.798526 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:41.798267    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/71e7abef-0f1f-4797-a4f6-1b6793083fb8-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714vkpp\" (UID: \"71e7abef-0f1f-4797-a4f6-1b6793083fb8\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714vkpp"
Oct 31 12:31:41.799277 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:41.799181    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/71e7abef-0f1f-4797-a4f6-1b6793083fb8-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714vkpp\" (UID: \"71e7abef-0f1f-4797-a4f6-1b6793083fb8\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714vkpp"
Oct 31 12:31:41.815522 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:41.815469    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zv8tf\" (UniqueName: \"kubernetes.io/projected/71e7abef-0f1f-4797-a4f6-1b6793083fb8-kube-api-access-zv8tf\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714vkpp\" (UID: \"71e7abef-0f1f-4797-a4f6-1b6793083fb8\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714vkpp"
Oct 31 12:31:42.002269 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:42.002174    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714vkpp"
Oct 31 12:31:42.326499 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:42.326269    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714vkpp"]
Oct 31 12:31:43.226806 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:43.226724    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714vkpp" event={"ID":"71e7abef-0f1f-4797-a4f6-1b6793083fb8","Type":"ContainerStarted","Data":"941e9e2e256549eee71db0df44ddaf4b3ae3fd0fa1b9520687eba63747867d3e"}
Oct 31 12:31:44.207696 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:44.207611    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cgdfz8"]
Oct 31 12:31:44.213378 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:44.213330    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cgdfz8"
Oct 31 12:31:44.233033 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:44.231650    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cgdfz8"]
Oct 31 12:31:44.234831 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:44.234650    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69pppl"]
Oct 31 12:31:44.238848 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:44.238829    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69pppl"
Oct 31 12:31:44.248813 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:44.248727    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69pppl"]
Oct 31 12:31:44.272424 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:44.272397    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xj7b9"]
Oct 31 12:31:44.277526 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:44.277479    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xj7b9"
Oct 31 12:31:44.286387 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:44.286324    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xj7b9"]
Oct 31 12:31:44.300209 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:44.300183    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7n5zk"]
Oct 31 12:31:44.307267 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:44.307245    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7n5zk"
Oct 31 12:31:44.317669 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:44.316435    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7n5zk"]
Oct 31 12:31:44.330581 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:44.330553    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c5fd411c-1234-4a0c-92ad-5904b87906c3-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69pppl\" (UID: \"c5fd411c-1234-4a0c-92ad-5904b87906c3\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69pppl"
Oct 31 12:31:44.330905 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:44.330864    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98vkl\" (UniqueName: \"kubernetes.io/projected/c0685425-0e8f-48ca-aa57-9a99616d1613-kube-api-access-98vkl\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7n5zk\" (UID: \"c0685425-0e8f-48ca-aa57-9a99616d1613\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7n5zk"
Oct 31 12:31:44.331089 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:44.331072    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4tnq\" (UniqueName: \"kubernetes.io/projected/198de6ba-59d4-4be8-96e9-f0c83b16c7dc-kube-api-access-r4tnq\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cgdfz8\" (UID: \"198de6ba-59d4-4be8-96e9-f0c83b16c7dc\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cgdfz8"
Oct 31 12:31:44.331197 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:44.331182    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/198de6ba-59d4-4be8-96e9-f0c83b16c7dc-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cgdfz8\" (UID: \"198de6ba-59d4-4be8-96e9-f0c83b16c7dc\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cgdfz8"
Oct 31 12:31:44.331302 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:44.331288    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c0685425-0e8f-48ca-aa57-9a99616d1613-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7n5zk\" (UID: \"c0685425-0e8f-48ca-aa57-9a99616d1613\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7n5zk"
Oct 31 12:31:44.331399 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:44.331387    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c5fd411c-1234-4a0c-92ad-5904b87906c3-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69pppl\" (UID: \"c5fd411c-1234-4a0c-92ad-5904b87906c3\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69pppl"
Oct 31 12:31:44.331695 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:44.331484    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c0685425-0e8f-48ca-aa57-9a99616d1613-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7n5zk\" (UID: \"c0685425-0e8f-48ca-aa57-9a99616d1613\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7n5zk"
Oct 31 12:31:44.331695 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:44.331551    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mhd8\" (UniqueName: \"kubernetes.io/projected/c5fd411c-1234-4a0c-92ad-5904b87906c3-kube-api-access-4mhd8\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69pppl\" (UID: \"c5fd411c-1234-4a0c-92ad-5904b87906c3\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69pppl"
Oct 31 12:31:44.331695 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:44.331594    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chv6c\" (UniqueName: \"kubernetes.io/projected/cce812aa-b1c4-4a27-95e5-51d5a36e5504-kube-api-access-chv6c\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xj7b9\" (UID: \"cce812aa-b1c4-4a27-95e5-51d5a36e5504\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xj7b9"
Oct 31 12:31:44.331695 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:44.331618    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cce812aa-b1c4-4a27-95e5-51d5a36e5504-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xj7b9\" (UID: \"cce812aa-b1c4-4a27-95e5-51d5a36e5504\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xj7b9"
Oct 31 12:31:44.331695 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:44.331647    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/198de6ba-59d4-4be8-96e9-f0c83b16c7dc-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cgdfz8\" (UID: \"198de6ba-59d4-4be8-96e9-f0c83b16c7dc\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cgdfz8"
Oct 31 12:31:44.331983 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:44.331795    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cce812aa-b1c4-4a27-95e5-51d5a36e5504-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xj7b9\" (UID: \"cce812aa-b1c4-4a27-95e5-51d5a36e5504\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xj7b9"
Oct 31 12:31:44.437205 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:44.433241    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c5fd411c-1234-4a0c-92ad-5904b87906c3-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69pppl\" (UID: \"c5fd411c-1234-4a0c-92ad-5904b87906c3\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69pppl"
Oct 31 12:31:44.437205 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:44.433296    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c0685425-0e8f-48ca-aa57-9a99616d1613-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7n5zk\" (UID: \"c0685425-0e8f-48ca-aa57-9a99616d1613\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7n5zk"
Oct 31 12:31:44.437205 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:44.433346    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mhd8\" (UniqueName: \"kubernetes.io/projected/c5fd411c-1234-4a0c-92ad-5904b87906c3-kube-api-access-4mhd8\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69pppl\" (UID: \"c5fd411c-1234-4a0c-92ad-5904b87906c3\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69pppl"
Oct 31 12:31:44.437205 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:44.433412    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chv6c\" (UniqueName: \"kubernetes.io/projected/cce812aa-b1c4-4a27-95e5-51d5a36e5504-kube-api-access-chv6c\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xj7b9\" (UID: \"cce812aa-b1c4-4a27-95e5-51d5a36e5504\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xj7b9"
Oct 31 12:31:44.437205 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:44.433440    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cce812aa-b1c4-4a27-95e5-51d5a36e5504-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xj7b9\" (UID: \"cce812aa-b1c4-4a27-95e5-51d5a36e5504\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xj7b9"
Oct 31 12:31:44.437205 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:44.433474    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/198de6ba-59d4-4be8-96e9-f0c83b16c7dc-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cgdfz8\" (UID: \"198de6ba-59d4-4be8-96e9-f0c83b16c7dc\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cgdfz8"
Oct 31 12:31:44.437205 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:44.433533    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cce812aa-b1c4-4a27-95e5-51d5a36e5504-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xj7b9\" (UID: \"cce812aa-b1c4-4a27-95e5-51d5a36e5504\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xj7b9"
Oct 31 12:31:44.437205 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:44.433580    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c5fd411c-1234-4a0c-92ad-5904b87906c3-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69pppl\" (UID: \"c5fd411c-1234-4a0c-92ad-5904b87906c3\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69pppl"
Oct 31 12:31:44.437205 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:44.433622    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98vkl\" (UniqueName: \"kubernetes.io/projected/c0685425-0e8f-48ca-aa57-9a99616d1613-kube-api-access-98vkl\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7n5zk\" (UID: \"c0685425-0e8f-48ca-aa57-9a99616d1613\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7n5zk"
Oct 31 12:31:44.437205 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:44.433676    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4tnq\" (UniqueName: \"kubernetes.io/projected/198de6ba-59d4-4be8-96e9-f0c83b16c7dc-kube-api-access-r4tnq\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cgdfz8\" (UID: \"198de6ba-59d4-4be8-96e9-f0c83b16c7dc\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cgdfz8"
Oct 31 12:31:44.437205 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:44.433716    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/198de6ba-59d4-4be8-96e9-f0c83b16c7dc-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cgdfz8\" (UID: \"198de6ba-59d4-4be8-96e9-f0c83b16c7dc\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cgdfz8"
Oct 31 12:31:44.437205 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:44.433741    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c0685425-0e8f-48ca-aa57-9a99616d1613-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7n5zk\" (UID: \"c0685425-0e8f-48ca-aa57-9a99616d1613\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7n5zk"
Oct 31 12:31:44.437205 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:44.434192    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c0685425-0e8f-48ca-aa57-9a99616d1613-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7n5zk\" (UID: \"c0685425-0e8f-48ca-aa57-9a99616d1613\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7n5zk"
Oct 31 12:31:44.437205 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:44.434536    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c5fd411c-1234-4a0c-92ad-5904b87906c3-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69pppl\" (UID: \"c5fd411c-1234-4a0c-92ad-5904b87906c3\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69pppl"
Oct 31 12:31:44.437205 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:44.434782    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c0685425-0e8f-48ca-aa57-9a99616d1613-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7n5zk\" (UID: \"c0685425-0e8f-48ca-aa57-9a99616d1613\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7n5zk"
Oct 31 12:31:44.437205 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:44.435703    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cce812aa-b1c4-4a27-95e5-51d5a36e5504-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xj7b9\" (UID: \"cce812aa-b1c4-4a27-95e5-51d5a36e5504\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xj7b9"
Oct 31 12:31:44.437205 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:44.436005    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/198de6ba-59d4-4be8-96e9-f0c83b16c7dc-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cgdfz8\" (UID: \"198de6ba-59d4-4be8-96e9-f0c83b16c7dc\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cgdfz8"
Oct 31 12:31:44.437205 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:44.436260    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cce812aa-b1c4-4a27-95e5-51d5a36e5504-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xj7b9\" (UID: \"cce812aa-b1c4-4a27-95e5-51d5a36e5504\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xj7b9"
Oct 31 12:31:44.437205 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:44.436513    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c5fd411c-1234-4a0c-92ad-5904b87906c3-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69pppl\" (UID: \"c5fd411c-1234-4a0c-92ad-5904b87906c3\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69pppl"
Oct 31 12:31:44.437205 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:44.437152    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/198de6ba-59d4-4be8-96e9-f0c83b16c7dc-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cgdfz8\" (UID: \"198de6ba-59d4-4be8-96e9-f0c83b16c7dc\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cgdfz8"
Oct 31 12:31:44.453274 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:44.453146    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mhd8\" (UniqueName: \"kubernetes.io/projected/c5fd411c-1234-4a0c-92ad-5904b87906c3-kube-api-access-4mhd8\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69pppl\" (UID: \"c5fd411c-1234-4a0c-92ad-5904b87906c3\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69pppl"
Oct 31 12:31:44.453715 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:44.453686    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chv6c\" (UniqueName: \"kubernetes.io/projected/cce812aa-b1c4-4a27-95e5-51d5a36e5504-kube-api-access-chv6c\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xj7b9\" (UID: \"cce812aa-b1c4-4a27-95e5-51d5a36e5504\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xj7b9"
Oct 31 12:31:44.454451 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:44.454385    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98vkl\" (UniqueName: \"kubernetes.io/projected/c0685425-0e8f-48ca-aa57-9a99616d1613-kube-api-access-98vkl\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7n5zk\" (UID: \"c0685425-0e8f-48ca-aa57-9a99616d1613\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7n5zk"
Oct 31 12:31:44.454971 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:44.454925    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4tnq\" (UniqueName: \"kubernetes.io/projected/198de6ba-59d4-4be8-96e9-f0c83b16c7dc-kube-api-access-r4tnq\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cgdfz8\" (UID: \"198de6ba-59d4-4be8-96e9-f0c83b16c7dc\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cgdfz8"
Oct 31 12:31:44.540573 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:44.540532    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cgdfz8"
Oct 31 12:31:44.563070 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:44.562841    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69pppl"
Oct 31 12:31:44.610526 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:44.606712    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xj7b9"
Oct 31 12:31:44.639211 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:44.639181    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7n5zk"
Oct 31 12:31:44.862056 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:44.862012    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69pppl"]
Oct 31 12:31:44.870833 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:44.868872    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cgdfz8"]
Oct 31 12:31:44.874726 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:31:44.874593    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5fd411c_1234_4a0c_92ad_5904b87906c3.slice/crio-be8033a1ab7edf14bb68690d4adb3fb0dbae075d5a366e463d04fbfc7362daaf WatchSource:0}: Error finding container be8033a1ab7edf14bb68690d4adb3fb0dbae075d5a366e463d04fbfc7362daaf: Status 404 returned error can't find the container with id be8033a1ab7edf14bb68690d4adb3fb0dbae075d5a366e463d04fbfc7362daaf
Oct 31 12:31:45.015631 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:45.015596    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7n5zk"]
Oct 31 12:31:45.022226 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:45.022197    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xj7b9"]
Oct 31 12:31:45.023356 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:31:45.023293    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0685425_0e8f_48ca_aa57_9a99616d1613.slice/crio-2255997d457f43ccddd7d1146bcdf484a12b67baa0201befef378c6d1be36461 WatchSource:0}: Error finding container 2255997d457f43ccddd7d1146bcdf484a12b67baa0201befef378c6d1be36461: Status 404 returned error can't find the container with id 2255997d457f43ccddd7d1146bcdf484a12b67baa0201befef378c6d1be36461
Oct 31 12:31:45.027560 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:31:45.027517    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcce812aa_b1c4_4a27_95e5_51d5a36e5504.slice/crio-c9fcf8ea13ecfe67d30785d4789e3b57540b73b7e33db02f399f9bc1e7947b7b WatchSource:0}: Error finding container c9fcf8ea13ecfe67d30785d4789e3b57540b73b7e33db02f399f9bc1e7947b7b: Status 404 returned error can't find the container with id c9fcf8ea13ecfe67d30785d4789e3b57540b73b7e33db02f399f9bc1e7947b7b
Oct 31 12:31:45.245738 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:45.245650    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cgdfz8" event={"ID":"198de6ba-59d4-4be8-96e9-f0c83b16c7dc","Type":"ContainerStarted","Data":"ea4eb76bd40aa4d9cd18c47450f4f345b829b77ce4fa0e540d5d86a87547d040"}
Oct 31 12:31:45.251613 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:45.251578    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7n5zk" event={"ID":"c0685425-0e8f-48ca-aa57-9a99616d1613","Type":"ContainerStarted","Data":"2255997d457f43ccddd7d1146bcdf484a12b67baa0201befef378c6d1be36461"}
Oct 31 12:31:45.255781 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:45.255748    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69pppl" event={"ID":"c5fd411c-1234-4a0c-92ad-5904b87906c3","Type":"ContainerStarted","Data":"be8033a1ab7edf14bb68690d4adb3fb0dbae075d5a366e463d04fbfc7362daaf"}
Oct 31 12:31:45.260121 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:45.260091    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xj7b9" event={"ID":"cce812aa-b1c4-4a27-95e5-51d5a36e5504","Type":"ContainerStarted","Data":"c9fcf8ea13ecfe67d30785d4789e3b57540b73b7e33db02f399f9bc1e7947b7b"}
Oct 31 12:31:54.408432 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:54.408380    5060 generic.go:356] "Generic (PLEG): container finished" podID="a610abd9-13b7-493f-84a5-e6cb1cbd1262" containerID="f1ff125f6945614a8cd0acfbedb487f72c39cb317715411f7d4393e956adedb4" exitCode=0
Oct 31 12:31:54.408978 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:54.408486    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644g8g7n" event={"ID":"a610abd9-13b7-493f-84a5-e6cb1cbd1262","Type":"ContainerDied","Data":"f1ff125f6945614a8cd0acfbedb487f72c39cb317715411f7d4393e956adedb4"}
Oct 31 12:31:56.432653 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:56.432582    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714vkpp" event={"ID":"71e7abef-0f1f-4797-a4f6-1b6793083fb8","Type":"ContainerStarted","Data":"bef5bb3de0da452807b914cc3b5a93dc0110ad8d04bc5fe5456fa2c8d73c487a"}
Oct 31 12:31:56.435974 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:56.435940    5060 generic.go:356] "Generic (PLEG): container finished" podID="00a1f4b5-4e4e-42b8-b539-f4da4db2a558" containerID="ca73cde7c165580f7c901968c84876934a2ddd0f1eb0872a4620890bca1ec64c" exitCode=0
Oct 31 12:31:56.436098 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:56.436031    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cw274p" event={"ID":"00a1f4b5-4e4e-42b8-b539-f4da4db2a558","Type":"ContainerDied","Data":"ca73cde7c165580f7c901968c84876934a2ddd0f1eb0872a4620890bca1ec64c"}
Oct 31 12:31:57.442425 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:57.442391    5060 generic.go:356] "Generic (PLEG): container finished" podID="b6f12f5a-911b-4180-99f4-c6d70e670320" containerID="2b098e97f270b9139d636c793f33f0f7f2f89f9d55d7519c41e4788ba0b1cd52" exitCode=0
Oct 31 12:31:57.442838 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:57.442468    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aprjf2" event={"ID":"b6f12f5a-911b-4180-99f4-c6d70e670320","Type":"ContainerDied","Data":"2b098e97f270b9139d636c793f33f0f7f2f89f9d55d7519c41e4788ba0b1cd52"}
Oct 31 12:31:57.444109 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:57.444081    5060 generic.go:356] "Generic (PLEG): container finished" podID="71e7abef-0f1f-4797-a4f6-1b6793083fb8" containerID="bef5bb3de0da452807b914cc3b5a93dc0110ad8d04bc5fe5456fa2c8d73c487a" exitCode=0
Oct 31 12:31:57.444219 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:57.444110    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714vkpp" event={"ID":"71e7abef-0f1f-4797-a4f6-1b6793083fb8","Type":"ContainerDied","Data":"bef5bb3de0da452807b914cc3b5a93dc0110ad8d04bc5fe5456fa2c8d73c487a"}
Oct 31 12:31:57.445821 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:57.445789    5060 generic.go:356] "Generic (PLEG): container finished" podID="b4836388-df04-4576-80b8-4ecd90bbb1fe" containerID="0c31c25edff47266d4878d29ec6f1ab45b01045b1ff76b37b73de928189065d1" exitCode=0
Oct 31 12:31:57.445918 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:57.445833    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qr8zz" event={"ID":"b4836388-df04-4576-80b8-4ecd90bbb1fe","Type":"ContainerDied","Data":"0c31c25edff47266d4878d29ec6f1ab45b01045b1ff76b37b73de928189065d1"}
Oct 31 12:31:58.455918 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:58.455880    5060 generic.go:356] "Generic (PLEG): container finished" podID="c0685425-0e8f-48ca-aa57-9a99616d1613" containerID="3d6e9dd53c2fccae5834a2c95ae0da45f543a669050b782bc856a7589f00aac2" exitCode=0
Oct 31 12:31:58.456367 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:58.455961    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7n5zk" event={"ID":"c0685425-0e8f-48ca-aa57-9a99616d1613","Type":"ContainerDied","Data":"3d6e9dd53c2fccae5834a2c95ae0da45f543a669050b782bc856a7589f00aac2"}
Oct 31 12:31:58.457961 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:58.457935    5060 generic.go:356] "Generic (PLEG): container finished" podID="c5fd411c-1234-4a0c-92ad-5904b87906c3" containerID="a17ee34b7d5070b0af98490775d8b85dfeb54ccb939e464829080aa594c44ef9" exitCode=0
Oct 31 12:31:58.458093 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:58.458028    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69pppl" event={"ID":"c5fd411c-1234-4a0c-92ad-5904b87906c3","Type":"ContainerDied","Data":"a17ee34b7d5070b0af98490775d8b85dfeb54ccb939e464829080aa594c44ef9"}
Oct 31 12:31:58.460683 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:58.460663    5060 generic.go:356] "Generic (PLEG): container finished" podID="71e7abef-0f1f-4797-a4f6-1b6793083fb8" containerID="7ed1088e090b02dda7ec7c6e324aba7da57312ddd664ebf5ff924791fe7b4530" exitCode=0
Oct 31 12:31:58.460777 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:58.460692    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714vkpp" event={"ID":"71e7abef-0f1f-4797-a4f6-1b6793083fb8","Type":"ContainerDied","Data":"7ed1088e090b02dda7ec7c6e324aba7da57312ddd664ebf5ff924791fe7b4530"}
Oct 31 12:31:58.462281 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:58.462233    5060 generic.go:356] "Generic (PLEG): container finished" podID="cce812aa-b1c4-4a27-95e5-51d5a36e5504" containerID="bbec2b2850d5fc87fe3bc9c717f545b4d54742ba93a8f2e7e633d0e422a137f6" exitCode=0
Oct 31 12:31:58.462342 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:58.462312    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xj7b9" event={"ID":"cce812aa-b1c4-4a27-95e5-51d5a36e5504","Type":"ContainerDied","Data":"bbec2b2850d5fc87fe3bc9c717f545b4d54742ba93a8f2e7e633d0e422a137f6"}
Oct 31 12:31:58.463968 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:58.463946    5060 generic.go:356] "Generic (PLEG): container finished" podID="5ba15213-2643-474d-84f3-50a443cd9088" containerID="ca46a3dc83de76c8660f54ec2c0f224c54fb355baa7cf76d2b6ac03d9c7a8c48" exitCode=0
Oct 31 12:31:58.464055 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:58.464020    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8pxmf" event={"ID":"5ba15213-2643-474d-84f3-50a443cd9088","Type":"ContainerDied","Data":"ca46a3dc83de76c8660f54ec2c0f224c54fb355baa7cf76d2b6ac03d9c7a8c48"}
Oct 31 12:31:58.465574 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:58.465553    5060 generic.go:356] "Generic (PLEG): container finished" podID="1bc5f088-e98b-4353-9512-39a3b11b9aba" containerID="4aec737c544866183676a8d2a715e3b881595c5f51e7bc4ebde2b83ea95690f6" exitCode=0
Oct 31 12:31:58.465674 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:58.465640    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mlqlx" event={"ID":"1bc5f088-e98b-4353-9512-39a3b11b9aba","Type":"ContainerDied","Data":"4aec737c544866183676a8d2a715e3b881595c5f51e7bc4ebde2b83ea95690f6"}
Oct 31 12:31:58.467551 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:58.467464    5060 generic.go:356] "Generic (PLEG): container finished" podID="198de6ba-59d4-4be8-96e9-f0c83b16c7dc" containerID="5c0a911512fca8d468b995a4255361e4a9e7779d7f12fe614de1c9f362f082f4" exitCode=0
Oct 31 12:31:58.467663 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:58.467551    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cgdfz8" event={"ID":"198de6ba-59d4-4be8-96e9-f0c83b16c7dc","Type":"ContainerDied","Data":"5c0a911512fca8d468b995a4255361e4a9e7779d7f12fe614de1c9f362f082f4"}
Oct 31 12:31:59.475486 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:59.475461    5060 generic.go:356] "Generic (PLEG): container finished" podID="b4836388-df04-4576-80b8-4ecd90bbb1fe" containerID="332e97ca1ed51d38231e15993c922a7b56182521369df2c4942b4faf155f703a" exitCode=0
Oct 31 12:31:59.475882 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:59.475523    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qr8zz" event={"ID":"b4836388-df04-4576-80b8-4ecd90bbb1fe","Type":"ContainerDied","Data":"332e97ca1ed51d38231e15993c922a7b56182521369df2c4942b4faf155f703a"}
Oct 31 12:31:59.477618 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:59.477597    5060 generic.go:356] "Generic (PLEG): container finished" podID="c5fd411c-1234-4a0c-92ad-5904b87906c3" containerID="2e07551496a0793577f094b3294b1296c24ad7fa72ef42f307c123bd3b81808d" exitCode=0
Oct 31 12:31:59.477716 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:59.477679    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69pppl" event={"ID":"c5fd411c-1234-4a0c-92ad-5904b87906c3","Type":"ContainerDied","Data":"2e07551496a0793577f094b3294b1296c24ad7fa72ef42f307c123bd3b81808d"}
Oct 31 12:31:59.479745 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:59.479726    5060 generic.go:356] "Generic (PLEG): container finished" podID="a610abd9-13b7-493f-84a5-e6cb1cbd1262" containerID="395074af3ad1064606a4e024682bb8baab06cbe8b4744f9d295415ae2c732a8c" exitCode=0
Oct 31 12:31:59.479828 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:31:59.479787    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644g8g7n" event={"ID":"a610abd9-13b7-493f-84a5-e6cb1cbd1262","Type":"ContainerDied","Data":"395074af3ad1064606a4e024682bb8baab06cbe8b4744f9d295415ae2c732a8c"}
Oct 31 12:31:59.936432 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:31:59.936376    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7phb" containerName="extractor"
Oct 31 12:32:00.490079 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:00.490036    5060 generic.go:356] "Generic (PLEG): container finished" podID="c0685425-0e8f-48ca-aa57-9a99616d1613" containerID="d0c497192b6bc208db6b154dd3a19e00232bf2e808c122ec817699439bc32695" exitCode=0
Oct 31 12:32:00.490561 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:00.490163    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7n5zk" event={"ID":"c0685425-0e8f-48ca-aa57-9a99616d1613","Type":"ContainerDied","Data":"d0c497192b6bc208db6b154dd3a19e00232bf2e808c122ec817699439bc32695"}
Oct 31 12:32:00.496310 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:00.496267    5060 generic.go:356] "Generic (PLEG): container finished" podID="b6f12f5a-911b-4180-99f4-c6d70e670320" containerID="63a7ad6c81d5cf8cb0aa911d23b7321f1357f7c31e3e66a8d5f39632779fa59f" exitCode=0
Oct 31 12:32:00.497082 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:00.496934    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aprjf2" event={"ID":"b6f12f5a-911b-4180-99f4-c6d70e670320","Type":"ContainerDied","Data":"63a7ad6c81d5cf8cb0aa911d23b7321f1357f7c31e3e66a8d5f39632779fa59f"}
Oct 31 12:32:00.501557 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:00.501531    5060 generic.go:356] "Generic (PLEG): container finished" podID="cce812aa-b1c4-4a27-95e5-51d5a36e5504" containerID="e4b5fb4f1bcfa53b5b08d549589b80d944eae01dd57b556332cc95faab5612a9" exitCode=0
Oct 31 12:32:00.501669 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:00.501621    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xj7b9" event={"ID":"cce812aa-b1c4-4a27-95e5-51d5a36e5504","Type":"ContainerDied","Data":"e4b5fb4f1bcfa53b5b08d549589b80d944eae01dd57b556332cc95faab5612a9"}
Oct 31 12:32:00.505832 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:00.505241    5060 generic.go:356] "Generic (PLEG): container finished" podID="5ba15213-2643-474d-84f3-50a443cd9088" containerID="3d941d06ce86f7e2fa3a6ae23e736a1e365cdd5b824c6d3b0c0319a5c9237636" exitCode=0
Oct 31 12:32:00.505832 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:00.505325    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8pxmf" event={"ID":"5ba15213-2643-474d-84f3-50a443cd9088","Type":"ContainerDied","Data":"3d941d06ce86f7e2fa3a6ae23e736a1e365cdd5b824c6d3b0c0319a5c9237636"}
Oct 31 12:32:00.509904 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:00.509879    5060 generic.go:356] "Generic (PLEG): container finished" podID="1bc5f088-e98b-4353-9512-39a3b11b9aba" containerID="c7339738d529be441cdb70d03919d011465d6caa8ff516569aac110cfddf19b3" exitCode=0
Oct 31 12:32:00.510006 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:00.509973    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mlqlx" event={"ID":"1bc5f088-e98b-4353-9512-39a3b11b9aba","Type":"ContainerDied","Data":"c7339738d529be441cdb70d03919d011465d6caa8ff516569aac110cfddf19b3"}
Oct 31 12:32:01.521614 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:01.520709    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cw274p" event={"ID":"00a1f4b5-4e4e-42b8-b539-f4da4db2a558","Type":"ContainerStarted","Data":"728fcd6e18b188c86ae9c7cdee45232a5be796f98e3e664de587143d5e3bd8d4"}
Oct 31 12:32:02.544537 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:02.544332    5060 generic.go:356] "Generic (PLEG): container finished" podID="00a1f4b5-4e4e-42b8-b539-f4da4db2a558" containerID="728fcd6e18b188c86ae9c7cdee45232a5be796f98e3e664de587143d5e3bd8d4" exitCode=0
Oct 31 12:32:02.544537 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:02.544404    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cw274p" event={"ID":"00a1f4b5-4e4e-42b8-b539-f4da4db2a558","Type":"ContainerDied","Data":"728fcd6e18b188c86ae9c7cdee45232a5be796f98e3e664de587143d5e3bd8d4"}
Oct 31 12:32:04.597541 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:04.597372    5060 generic.go:356] "Generic (PLEG): container finished" podID="198de6ba-59d4-4be8-96e9-f0c83b16c7dc" containerID="d58d637be5f64de05849a7e7f5691ac5a456fb6807cc5ffe3a2a293a410f2586" exitCode=0
Oct 31 12:32:04.598183 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:04.598113    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cgdfz8" event={"ID":"198de6ba-59d4-4be8-96e9-f0c83b16c7dc","Type":"ContainerDied","Data":"d58d637be5f64de05849a7e7f5691ac5a456fb6807cc5ffe3a2a293a410f2586"}
Oct 31 12:32:17.159353 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:17.159262    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pcvvm_f032909d-bc39-4c9e-99b7-f3d800842119/console-operator/2.log"
Oct 31 12:32:17.167021 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:17.166930    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pcvvm_f032909d-bc39-4c9e-99b7-f3d800842119/console-operator/2.log"
Oct 31 12:32:21.841530 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:21.841477    5060 generic.go:356] "Generic (PLEG): container finished" podID="c5fd411c-1234-4a0c-92ad-5904b87906c3" containerID="913802c56c42005f10689fd4e5954584e028a4fd503aada663c67a70b61f3ff2" exitCode=0
Oct 31 12:32:21.841983 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:21.841595    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69pppl" event={"ID":"c5fd411c-1234-4a0c-92ad-5904b87906c3","Type":"ContainerDied","Data":"913802c56c42005f10689fd4e5954584e028a4fd503aada663c67a70b61f3ff2"}
Oct 31 12:32:21.849416 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:21.849380    5060 generic.go:356] "Generic (PLEG): container finished" podID="71e7abef-0f1f-4797-a4f6-1b6793083fb8" containerID="df133e6be75a09a758ba3abd8ae6ade3a3bbf92910c3ab10c864a41f17382759" exitCode=0
Oct 31 12:32:21.849541 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:21.849469    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714vkpp" event={"ID":"71e7abef-0f1f-4797-a4f6-1b6793083fb8","Type":"ContainerDied","Data":"df133e6be75a09a758ba3abd8ae6ade3a3bbf92910c3ab10c864a41f17382759"}
Oct 31 12:32:22.859062 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:22.859029    5060 generic.go:356] "Generic (PLEG): container finished" podID="1bc5f088-e98b-4353-9512-39a3b11b9aba" containerID="88e4b1ec004cca5e3beceb456bd24f1824d0360e8563d443588ae31a87f3e51f" exitCode=0
Oct 31 12:32:22.859467 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:22.859128    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mlqlx" event={"ID":"1bc5f088-e98b-4353-9512-39a3b11b9aba","Type":"ContainerDied","Data":"88e4b1ec004cca5e3beceb456bd24f1824d0360e8563d443588ae31a87f3e51f"}
Oct 31 12:32:22.863577 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:22.863546    5060 generic.go:356] "Generic (PLEG): container finished" podID="b4836388-df04-4576-80b8-4ecd90bbb1fe" containerID="29c56dcbfac8ab8da288bfe6593f6b62fbfee6e9e3d3cdbe87e13bda8a9fa4fd" exitCode=0
Oct 31 12:32:22.863805 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:22.863594    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qr8zz" event={"ID":"b4836388-df04-4576-80b8-4ecd90bbb1fe","Type":"ContainerDied","Data":"29c56dcbfac8ab8da288bfe6593f6b62fbfee6e9e3d3cdbe87e13bda8a9fa4fd"}
Oct 31 12:32:22.867444 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:22.867166    5060 generic.go:356] "Generic (PLEG): container finished" podID="c0685425-0e8f-48ca-aa57-9a99616d1613" containerID="4a1eac4a2c26afe3cf7fdb696b6121628eb95e701817fea2a14cb7921bf17257" exitCode=0
Oct 31 12:32:22.867444 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:22.867269    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7n5zk" event={"ID":"c0685425-0e8f-48ca-aa57-9a99616d1613","Type":"ContainerDied","Data":"4a1eac4a2c26afe3cf7fdb696b6121628eb95e701817fea2a14cb7921bf17257"}
Oct 31 12:32:22.870681 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:22.870656    5060 generic.go:356] "Generic (PLEG): container finished" podID="a610abd9-13b7-493f-84a5-e6cb1cbd1262" containerID="618f88e7128ff35c1c842b7b1b23393dc7d23d265be0edcf1c7e33057d15c82c" exitCode=0
Oct 31 12:32:22.871105 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:22.871004    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644g8g7n" event={"ID":"a610abd9-13b7-493f-84a5-e6cb1cbd1262","Type":"ContainerDied","Data":"618f88e7128ff35c1c842b7b1b23393dc7d23d265be0edcf1c7e33057d15c82c"}
Oct 31 12:32:23.072119 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:23.071757    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714vkpp"
Oct 31 12:32:23.143298 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:23.143268    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69pppl"
Oct 31 12:32:23.225333 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:23.225290    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71e7abef-0f1f-4797-a4f6-1b6793083fb8-kube-api-access-zv8tf\" (UniqueName: \"kubernetes.io/projected/71e7abef-0f1f-4797-a4f6-1b6793083fb8-kube-api-access-zv8tf\") pod \"71e7abef-0f1f-4797-a4f6-1b6793083fb8\" (UID: \"71e7abef-0f1f-4797-a4f6-1b6793083fb8\") "
Oct 31 12:32:23.225463 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:23.225374    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5fd411c-1234-4a0c-92ad-5904b87906c3-util\" (UniqueName: \"kubernetes.io/empty-dir/c5fd411c-1234-4a0c-92ad-5904b87906c3-util\") pod \"c5fd411c-1234-4a0c-92ad-5904b87906c3\" (UID: \"c5fd411c-1234-4a0c-92ad-5904b87906c3\") "
Oct 31 12:32:23.225463 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:23.225428    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71e7abef-0f1f-4797-a4f6-1b6793083fb8-bundle\" (UniqueName: \"kubernetes.io/empty-dir/71e7abef-0f1f-4797-a4f6-1b6793083fb8-bundle\") pod \"71e7abef-0f1f-4797-a4f6-1b6793083fb8\" (UID: \"71e7abef-0f1f-4797-a4f6-1b6793083fb8\") "
Oct 31 12:32:23.225596 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:23.225473    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5fd411c-1234-4a0c-92ad-5904b87906c3-bundle\" (UniqueName: \"kubernetes.io/empty-dir/c5fd411c-1234-4a0c-92ad-5904b87906c3-bundle\") pod \"c5fd411c-1234-4a0c-92ad-5904b87906c3\" (UID: \"c5fd411c-1234-4a0c-92ad-5904b87906c3\") "
Oct 31 12:32:23.225596 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:23.225544    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5fd411c-1234-4a0c-92ad-5904b87906c3-kube-api-access-4mhd8\" (UniqueName: \"kubernetes.io/projected/c5fd411c-1234-4a0c-92ad-5904b87906c3-kube-api-access-4mhd8\") pod \"c5fd411c-1234-4a0c-92ad-5904b87906c3\" (UID: \"c5fd411c-1234-4a0c-92ad-5904b87906c3\") "
Oct 31 12:32:23.225596 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:23.225588    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71e7abef-0f1f-4797-a4f6-1b6793083fb8-util\" (UniqueName: \"kubernetes.io/empty-dir/71e7abef-0f1f-4797-a4f6-1b6793083fb8-util\") pod \"71e7abef-0f1f-4797-a4f6-1b6793083fb8\" (UID: \"71e7abef-0f1f-4797-a4f6-1b6793083fb8\") "
Oct 31 12:32:23.228484 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:23.226615    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5fd411c-1234-4a0c-92ad-5904b87906c3-bundle" pod "c5fd411c-1234-4a0c-92ad-5904b87906c3" (UID: "c5fd411c-1234-4a0c-92ad-5904b87906c3"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:32:23.228484 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:23.226778    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71e7abef-0f1f-4797-a4f6-1b6793083fb8-bundle" pod "71e7abef-0f1f-4797-a4f6-1b6793083fb8" (UID: "71e7abef-0f1f-4797-a4f6-1b6793083fb8"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:32:23.228827 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:23.228796    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71e7abef-0f1f-4797-a4f6-1b6793083fb8-kube-api-access-zv8tf" pod "71e7abef-0f1f-4797-a4f6-1b6793083fb8" (UID: "71e7abef-0f1f-4797-a4f6-1b6793083fb8"). InnerVolumeSpecName "kube-api-access-zv8tf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:32:23.233731 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:23.233706    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71e7abef-0f1f-4797-a4f6-1b6793083fb8-util" pod "71e7abef-0f1f-4797-a4f6-1b6793083fb8" (UID: "71e7abef-0f1f-4797-a4f6-1b6793083fb8"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:32:23.233832 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:23.233725    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5fd411c-1234-4a0c-92ad-5904b87906c3-util" pod "c5fd411c-1234-4a0c-92ad-5904b87906c3" (UID: "c5fd411c-1234-4a0c-92ad-5904b87906c3"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:32:23.236577 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:23.236547    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5fd411c-1234-4a0c-92ad-5904b87906c3-kube-api-access-4mhd8" pod "c5fd411c-1234-4a0c-92ad-5904b87906c3" (UID: "c5fd411c-1234-4a0c-92ad-5904b87906c3"). InnerVolumeSpecName "kube-api-access-4mhd8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:32:23.326592 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:23.326485    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zv8tf\" (UniqueName: \"kubernetes.io/projected/71e7abef-0f1f-4797-a4f6-1b6793083fb8-kube-api-access-zv8tf\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:32:23.326592 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:23.326526    5060 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c5fd411c-1234-4a0c-92ad-5904b87906c3-util\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:32:23.326592 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:23.326536    5060 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/71e7abef-0f1f-4797-a4f6-1b6793083fb8-bundle\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:32:23.326592 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:23.326545    5060 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c5fd411c-1234-4a0c-92ad-5904b87906c3-bundle\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:32:23.326592 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:23.326553    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4mhd8\" (UniqueName: \"kubernetes.io/projected/c5fd411c-1234-4a0c-92ad-5904b87906c3-kube-api-access-4mhd8\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:32:23.326592 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:23.326562    5060 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/71e7abef-0f1f-4797-a4f6-1b6793083fb8-util\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:32:23.879586 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:23.879531    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69pppl" event={"ID":"c5fd411c-1234-4a0c-92ad-5904b87906c3","Type":"ContainerDied","Data":"be8033a1ab7edf14bb68690d4adb3fb0dbae075d5a366e463d04fbfc7362daaf"}
Oct 31 12:32:23.879586 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:23.879573    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69pppl"
Oct 31 12:32:23.880110 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:23.879654    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="be8033a1ab7edf14bb68690d4adb3fb0dbae075d5a366e463d04fbfc7362daaf"
Oct 31 12:32:23.882328 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:23.882207    5060 generic.go:356] "Generic (PLEG): container finished" podID="b6f12f5a-911b-4180-99f4-c6d70e670320" containerID="e39c82ce90736183a76d1e3bcb9ff92b101648e81ca9eb0f2343f692222062c6" exitCode=0
Oct 31 12:32:23.882328 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:23.882278    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aprjf2" event={"ID":"b6f12f5a-911b-4180-99f4-c6d70e670320","Type":"ContainerDied","Data":"e39c82ce90736183a76d1e3bcb9ff92b101648e81ca9eb0f2343f692222062c6"}
Oct 31 12:32:23.888608 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:23.888560    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714vkpp" event={"ID":"71e7abef-0f1f-4797-a4f6-1b6793083fb8","Type":"ContainerDied","Data":"941e9e2e256549eee71db0df44ddaf4b3ae3fd0fa1b9520687eba63747867d3e"}
Oct 31 12:32:23.888779 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:23.888758    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="941e9e2e256549eee71db0df44ddaf4b3ae3fd0fa1b9520687eba63747867d3e"
Oct 31 12:32:23.888950 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:23.888780    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714vkpp"
Oct 31 12:32:23.891651 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:23.891620    5060 generic.go:356] "Generic (PLEG): container finished" podID="cce812aa-b1c4-4a27-95e5-51d5a36e5504" containerID="ec7b98aac51ced144e66bd972350981442958ef3f8fb781087331cefd87760d2" exitCode=0
Oct 31 12:32:23.891758 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:23.891691    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xj7b9" event={"ID":"cce812aa-b1c4-4a27-95e5-51d5a36e5504","Type":"ContainerDied","Data":"ec7b98aac51ced144e66bd972350981442958ef3f8fb781087331cefd87760d2"}
Oct 31 12:32:23.894395 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:23.894368    5060 generic.go:356] "Generic (PLEG): container finished" podID="5ba15213-2643-474d-84f3-50a443cd9088" containerID="2bb7a145977ea6f9a87325154e59ac8542f8f675b537be752c6d45371099185f" exitCode=0
Oct 31 12:32:23.894528 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:23.894448    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8pxmf" event={"ID":"5ba15213-2643-474d-84f3-50a443cd9088","Type":"ContainerDied","Data":"2bb7a145977ea6f9a87325154e59ac8542f8f675b537be752c6d45371099185f"}
Oct 31 12:32:24.073197 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:24.073168    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qr8zz"
Oct 31 12:32:24.101976 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:24.101949    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7n5zk"
Oct 31 12:32:24.166911 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:24.166882    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644g8g7n"
Oct 31 12:32:24.168955 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:24.168933    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mlqlx"
Oct 31 12:32:24.234101 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:24.234041    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0685425-0e8f-48ca-aa57-9a99616d1613-bundle\" (UniqueName: \"kubernetes.io/empty-dir/c0685425-0e8f-48ca-aa57-9a99616d1613-bundle\") pod \"c0685425-0e8f-48ca-aa57-9a99616d1613\" (UID: \"c0685425-0e8f-48ca-aa57-9a99616d1613\") "
Oct 31 12:32:24.234277 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:24.234121    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4836388-df04-4576-80b8-4ecd90bbb1fe-bundle\" (UniqueName: \"kubernetes.io/empty-dir/b4836388-df04-4576-80b8-4ecd90bbb1fe-bundle\") pod \"b4836388-df04-4576-80b8-4ecd90bbb1fe\" (UID: \"b4836388-df04-4576-80b8-4ecd90bbb1fe\") "
Oct 31 12:32:24.234277 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:24.234153    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4836388-df04-4576-80b8-4ecd90bbb1fe-kube-api-access-cfdwz\" (UniqueName: \"kubernetes.io/projected/b4836388-df04-4576-80b8-4ecd90bbb1fe-kube-api-access-cfdwz\") pod \"b4836388-df04-4576-80b8-4ecd90bbb1fe\" (UID: \"b4836388-df04-4576-80b8-4ecd90bbb1fe\") "
Oct 31 12:32:24.234277 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:24.234185    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0685425-0e8f-48ca-aa57-9a99616d1613-kube-api-access-98vkl\" (UniqueName: \"kubernetes.io/projected/c0685425-0e8f-48ca-aa57-9a99616d1613-kube-api-access-98vkl\") pod \"c0685425-0e8f-48ca-aa57-9a99616d1613\" (UID: \"c0685425-0e8f-48ca-aa57-9a99616d1613\") "
Oct 31 12:32:24.234594 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:24.234562    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0685425-0e8f-48ca-aa57-9a99616d1613-util\" (UniqueName: \"kubernetes.io/empty-dir/c0685425-0e8f-48ca-aa57-9a99616d1613-util\") pod \"c0685425-0e8f-48ca-aa57-9a99616d1613\" (UID: \"c0685425-0e8f-48ca-aa57-9a99616d1613\") "
Oct 31 12:32:24.234715 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:24.234677    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4836388-df04-4576-80b8-4ecd90bbb1fe-util\" (UniqueName: \"kubernetes.io/empty-dir/b4836388-df04-4576-80b8-4ecd90bbb1fe-util\") pod \"b4836388-df04-4576-80b8-4ecd90bbb1fe\" (UID: \"b4836388-df04-4576-80b8-4ecd90bbb1fe\") "
Oct 31 12:32:24.234774 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:24.234725    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4836388-df04-4576-80b8-4ecd90bbb1fe-bundle" pod "b4836388-df04-4576-80b8-4ecd90bbb1fe" (UID: "b4836388-df04-4576-80b8-4ecd90bbb1fe"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:32:24.234925 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:24.234898    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0685425-0e8f-48ca-aa57-9a99616d1613-bundle" pod "c0685425-0e8f-48ca-aa57-9a99616d1613" (UID: "c0685425-0e8f-48ca-aa57-9a99616d1613"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:32:24.234995 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:24.234969    5060 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b4836388-df04-4576-80b8-4ecd90bbb1fe-bundle\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:32:24.237710 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:24.237670    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0685425-0e8f-48ca-aa57-9a99616d1613-kube-api-access-98vkl" pod "c0685425-0e8f-48ca-aa57-9a99616d1613" (UID: "c0685425-0e8f-48ca-aa57-9a99616d1613"). InnerVolumeSpecName "kube-api-access-98vkl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:32:24.241076 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:24.240567    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4836388-df04-4576-80b8-4ecd90bbb1fe-util" pod "b4836388-df04-4576-80b8-4ecd90bbb1fe" (UID: "b4836388-df04-4576-80b8-4ecd90bbb1fe"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:32:24.242673 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:24.242636    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4836388-df04-4576-80b8-4ecd90bbb1fe-kube-api-access-cfdwz" pod "b4836388-df04-4576-80b8-4ecd90bbb1fe" (UID: "b4836388-df04-4576-80b8-4ecd90bbb1fe"). InnerVolumeSpecName "kube-api-access-cfdwz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:32:24.244319 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:24.244283    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0685425-0e8f-48ca-aa57-9a99616d1613-util" pod "c0685425-0e8f-48ca-aa57-9a99616d1613" (UID: "c0685425-0e8f-48ca-aa57-9a99616d1613"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:32:24.335882 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:24.335835    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a610abd9-13b7-493f-84a5-e6cb1cbd1262-kube-api-access-bsg9m\" (UniqueName: \"kubernetes.io/projected/a610abd9-13b7-493f-84a5-e6cb1cbd1262-kube-api-access-bsg9m\") pod \"a610abd9-13b7-493f-84a5-e6cb1cbd1262\" (UID: \"a610abd9-13b7-493f-84a5-e6cb1cbd1262\") "
Oct 31 12:32:24.336083 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:24.335920    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1bc5f088-e98b-4353-9512-39a3b11b9aba-kube-api-access-pl69q\" (UniqueName: \"kubernetes.io/projected/1bc5f088-e98b-4353-9512-39a3b11b9aba-kube-api-access-pl69q\") pod \"1bc5f088-e98b-4353-9512-39a3b11b9aba\" (UID: \"1bc5f088-e98b-4353-9512-39a3b11b9aba\") "
Oct 31 12:32:24.336083 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:24.335966    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bc5f088-e98b-4353-9512-39a3b11b9aba-bundle\" (UniqueName: \"kubernetes.io/empty-dir/1bc5f088-e98b-4353-9512-39a3b11b9aba-bundle\") pod \"1bc5f088-e98b-4353-9512-39a3b11b9aba\" (UID: \"1bc5f088-e98b-4353-9512-39a3b11b9aba\") "
Oct 31 12:32:24.336083 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:24.336012    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a610abd9-13b7-493f-84a5-e6cb1cbd1262-bundle\" (UniqueName: \"kubernetes.io/empty-dir/a610abd9-13b7-493f-84a5-e6cb1cbd1262-bundle\") pod \"a610abd9-13b7-493f-84a5-e6cb1cbd1262\" (UID: \"a610abd9-13b7-493f-84a5-e6cb1cbd1262\") "
Oct 31 12:32:24.336083 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:24.336036    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bc5f088-e98b-4353-9512-39a3b11b9aba-util\" (UniqueName: \"kubernetes.io/empty-dir/1bc5f088-e98b-4353-9512-39a3b11b9aba-util\") pod \"1bc5f088-e98b-4353-9512-39a3b11b9aba\" (UID: \"1bc5f088-e98b-4353-9512-39a3b11b9aba\") "
Oct 31 12:32:24.336083 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:24.336078    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a610abd9-13b7-493f-84a5-e6cb1cbd1262-util\" (UniqueName: \"kubernetes.io/empty-dir/a610abd9-13b7-493f-84a5-e6cb1cbd1262-util\") pod \"a610abd9-13b7-493f-84a5-e6cb1cbd1262\" (UID: \"a610abd9-13b7-493f-84a5-e6cb1cbd1262\") "
Oct 31 12:32:24.336339 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:24.336316    5060 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c0685425-0e8f-48ca-aa57-9a99616d1613-util\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:32:24.336395 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:24.336337    5060 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b4836388-df04-4576-80b8-4ecd90bbb1fe-util\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:32:24.336395 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:24.336351    5060 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c0685425-0e8f-48ca-aa57-9a99616d1613-bundle\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:32:24.336395 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:24.336369    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfdwz\" (UniqueName: \"kubernetes.io/projected/b4836388-df04-4576-80b8-4ecd90bbb1fe-kube-api-access-cfdwz\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:32:24.336395 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:24.336384    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98vkl\" (UniqueName: \"kubernetes.io/projected/c0685425-0e8f-48ca-aa57-9a99616d1613-kube-api-access-98vkl\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:32:24.336675 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:24.336643    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a610abd9-13b7-493f-84a5-e6cb1cbd1262-bundle" pod "a610abd9-13b7-493f-84a5-e6cb1cbd1262" (UID: "a610abd9-13b7-493f-84a5-e6cb1cbd1262"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:32:24.336756 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:24.336722    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bc5f088-e98b-4353-9512-39a3b11b9aba-bundle" pod "1bc5f088-e98b-4353-9512-39a3b11b9aba" (UID: "1bc5f088-e98b-4353-9512-39a3b11b9aba"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:32:24.338609 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:24.338579    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bc5f088-e98b-4353-9512-39a3b11b9aba-kube-api-access-pl69q" pod "1bc5f088-e98b-4353-9512-39a3b11b9aba" (UID: "1bc5f088-e98b-4353-9512-39a3b11b9aba"). InnerVolumeSpecName "kube-api-access-pl69q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:32:24.338779 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:24.338708    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a610abd9-13b7-493f-84a5-e6cb1cbd1262-kube-api-access-bsg9m" pod "a610abd9-13b7-493f-84a5-e6cb1cbd1262" (UID: "a610abd9-13b7-493f-84a5-e6cb1cbd1262"). InnerVolumeSpecName "kube-api-access-bsg9m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:32:24.340710 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:24.340667    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a610abd9-13b7-493f-84a5-e6cb1cbd1262-util" pod "a610abd9-13b7-493f-84a5-e6cb1cbd1262" (UID: "a610abd9-13b7-493f-84a5-e6cb1cbd1262"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:32:24.345475 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:24.345440    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bc5f088-e98b-4353-9512-39a3b11b9aba-util" pod "1bc5f088-e98b-4353-9512-39a3b11b9aba" (UID: "1bc5f088-e98b-4353-9512-39a3b11b9aba"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:32:24.437333 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:24.437257    5060 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1bc5f088-e98b-4353-9512-39a3b11b9aba-bundle\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:32:24.437333 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:24.437288    5060 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a610abd9-13b7-493f-84a5-e6cb1cbd1262-bundle\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:32:24.437333 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:24.437297    5060 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1bc5f088-e98b-4353-9512-39a3b11b9aba-util\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:32:24.437333 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:24.437305    5060 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a610abd9-13b7-493f-84a5-e6cb1cbd1262-util\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:32:24.437333 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:24.437316    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bsg9m\" (UniqueName: \"kubernetes.io/projected/a610abd9-13b7-493f-84a5-e6cb1cbd1262-kube-api-access-bsg9m\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:32:24.437333 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:24.437329    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pl69q\" (UniqueName: \"kubernetes.io/projected/1bc5f088-e98b-4353-9512-39a3b11b9aba-kube-api-access-pl69q\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:32:24.901645 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:24.901615    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7n5zk"
Oct 31 12:32:24.902105 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:24.901615    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7n5zk" event={"ID":"c0685425-0e8f-48ca-aa57-9a99616d1613","Type":"ContainerDied","Data":"2255997d457f43ccddd7d1146bcdf484a12b67baa0201befef378c6d1be36461"}
Oct 31 12:32:24.902105 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:24.901780    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2255997d457f43ccddd7d1146bcdf484a12b67baa0201befef378c6d1be36461"
Oct 31 12:32:24.903649 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:24.903625    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644g8g7n"
Oct 31 12:32:24.903794 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:24.903625    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644g8g7n" event={"ID":"a610abd9-13b7-493f-84a5-e6cb1cbd1262","Type":"ContainerDied","Data":"bbfb1defee389366342e1641b692af20013c4a633f2eefcaa534a1b38dbd2d9c"}
Oct 31 12:32:24.903794 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:24.903751    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bbfb1defee389366342e1641b692af20013c4a633f2eefcaa534a1b38dbd2d9c"
Oct 31 12:32:24.906019 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:24.905991    5060 generic.go:356] "Generic (PLEG): container finished" podID="00a1f4b5-4e4e-42b8-b539-f4da4db2a558" containerID="03ca005865c7aaa8e3bb76f22439c43afece0d238eb4856184ad36cf15db2409" exitCode=0
Oct 31 12:32:24.906142 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:24.906071    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cw274p" event={"ID":"00a1f4b5-4e4e-42b8-b539-f4da4db2a558","Type":"ContainerDied","Data":"03ca005865c7aaa8e3bb76f22439c43afece0d238eb4856184ad36cf15db2409"}
Oct 31 12:32:24.908274 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:24.908253    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mlqlx" event={"ID":"1bc5f088-e98b-4353-9512-39a3b11b9aba","Type":"ContainerDied","Data":"e27326091d5e35c3ff5815bd98775d6848d9bc4efb1d0824d3e183645a40f312"}
Oct 31 12:32:24.908382 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:24.908287    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e27326091d5e35c3ff5815bd98775d6848d9bc4efb1d0824d3e183645a40f312"
Oct 31 12:32:24.908382 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:24.908291    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mlqlx"
Oct 31 12:32:24.911623 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:24.911597    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qr8zz" event={"ID":"b4836388-df04-4576-80b8-4ecd90bbb1fe","Type":"ContainerDied","Data":"e04cbb2135b08e75762250c0a8c3e59c2342b925c2170a5ba15d8bb80b1aae9c"}
Oct 31 12:32:24.911623 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:24.911616    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qr8zz"
Oct 31 12:32:24.911757 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:24.911639    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e04cbb2135b08e75762250c0a8c3e59c2342b925c2170a5ba15d8bb80b1aae9c"
Oct 31 12:32:24.914522 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:24.914470    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cgdfz8" event={"ID":"198de6ba-59d4-4be8-96e9-f0c83b16c7dc","Type":"ContainerStarted","Data":"d6ea7604b2cfbd66a926baafb07604f9237c1fd314f9092a823e69be5db05950"}
Oct 31 12:32:25.059011 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:25.058985    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xj7b9"
Oct 31 12:32:25.073464 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:25.073432    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aprjf2"
Oct 31 12:32:25.081188 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:25.081129    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cgdfz8" podStartSLOduration=2.496982523 podStartE2EDuration="41.081109709s" totalImagesPullingTime="38.584127186s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:31:44 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:32:24.94871616 +0000 UTC m=+308.051783886" watchObservedRunningTime="2026-10-31 12:32:25.081109709 +0000 UTC m=+308.184177425"
Oct 31 12:32:25.106921 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:25.106890    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8pxmf"
Oct 31 12:32:25.144151 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:25.144118    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cce812aa-b1c4-4a27-95e5-51d5a36e5504-bundle\" (UniqueName: \"kubernetes.io/empty-dir/cce812aa-b1c4-4a27-95e5-51d5a36e5504-bundle\") pod \"cce812aa-b1c4-4a27-95e5-51d5a36e5504\" (UID: \"cce812aa-b1c4-4a27-95e5-51d5a36e5504\") "
Oct 31 12:32:25.144151 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:25.144151    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cce812aa-b1c4-4a27-95e5-51d5a36e5504-util\" (UniqueName: \"kubernetes.io/empty-dir/cce812aa-b1c4-4a27-95e5-51d5a36e5504-util\") pod \"cce812aa-b1c4-4a27-95e5-51d5a36e5504\" (UID: \"cce812aa-b1c4-4a27-95e5-51d5a36e5504\") "
Oct 31 12:32:25.144369 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:25.144181    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cce812aa-b1c4-4a27-95e5-51d5a36e5504-kube-api-access-chv6c\" (UniqueName: \"kubernetes.io/projected/cce812aa-b1c4-4a27-95e5-51d5a36e5504-kube-api-access-chv6c\") pod \"cce812aa-b1c4-4a27-95e5-51d5a36e5504\" (UID: \"cce812aa-b1c4-4a27-95e5-51d5a36e5504\") "
Oct 31 12:32:25.144746 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:25.144715    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cce812aa-b1c4-4a27-95e5-51d5a36e5504-bundle" pod "cce812aa-b1c4-4a27-95e5-51d5a36e5504" (UID: "cce812aa-b1c4-4a27-95e5-51d5a36e5504"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:32:25.146757 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:25.146732    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cce812aa-b1c4-4a27-95e5-51d5a36e5504-kube-api-access-chv6c" pod "cce812aa-b1c4-4a27-95e5-51d5a36e5504" (UID: "cce812aa-b1c4-4a27-95e5-51d5a36e5504"). InnerVolumeSpecName "kube-api-access-chv6c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:32:25.150409 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:25.150382    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cce812aa-b1c4-4a27-95e5-51d5a36e5504-util" pod "cce812aa-b1c4-4a27-95e5-51d5a36e5504" (UID: "cce812aa-b1c4-4a27-95e5-51d5a36e5504"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:32:25.244752 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:25.244721    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ba15213-2643-474d-84f3-50a443cd9088-kube-api-access-lsr4f\" (UniqueName: \"kubernetes.io/projected/5ba15213-2643-474d-84f3-50a443cd9088-kube-api-access-lsr4f\") pod \"5ba15213-2643-474d-84f3-50a443cd9088\" (UID: \"5ba15213-2643-474d-84f3-50a443cd9088\") "
Oct 31 12:32:25.244919 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:25.244801    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ba15213-2643-474d-84f3-50a443cd9088-bundle\" (UniqueName: \"kubernetes.io/empty-dir/5ba15213-2643-474d-84f3-50a443cd9088-bundle\") pod \"5ba15213-2643-474d-84f3-50a443cd9088\" (UID: \"5ba15213-2643-474d-84f3-50a443cd9088\") "
Oct 31 12:32:25.244919 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:25.244847    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6f12f5a-911b-4180-99f4-c6d70e670320-util\" (UniqueName: \"kubernetes.io/empty-dir/b6f12f5a-911b-4180-99f4-c6d70e670320-util\") pod \"b6f12f5a-911b-4180-99f4-c6d70e670320\" (UID: \"b6f12f5a-911b-4180-99f4-c6d70e670320\") "
Oct 31 12:32:25.244919 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:25.244872    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6f12f5a-911b-4180-99f4-c6d70e670320-kube-api-access-9dzb6\" (UniqueName: \"kubernetes.io/projected/b6f12f5a-911b-4180-99f4-c6d70e670320-kube-api-access-9dzb6\") pod \"b6f12f5a-911b-4180-99f4-c6d70e670320\" (UID: \"b6f12f5a-911b-4180-99f4-c6d70e670320\") "
Oct 31 12:32:25.244919 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:25.244917    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ba15213-2643-474d-84f3-50a443cd9088-util\" (UniqueName: \"kubernetes.io/empty-dir/5ba15213-2643-474d-84f3-50a443cd9088-util\") pod \"5ba15213-2643-474d-84f3-50a443cd9088\" (UID: \"5ba15213-2643-474d-84f3-50a443cd9088\") "
Oct 31 12:32:25.245103 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:25.244952    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6f12f5a-911b-4180-99f4-c6d70e670320-bundle\" (UniqueName: \"kubernetes.io/empty-dir/b6f12f5a-911b-4180-99f4-c6d70e670320-bundle\") pod \"b6f12f5a-911b-4180-99f4-c6d70e670320\" (UID: \"b6f12f5a-911b-4180-99f4-c6d70e670320\") "
Oct 31 12:32:25.245157 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:25.245121    5060 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cce812aa-b1c4-4a27-95e5-51d5a36e5504-bundle\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:32:25.245157 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:25.245143    5060 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cce812aa-b1c4-4a27-95e5-51d5a36e5504-util\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:32:25.245258 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:25.245157    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chv6c\" (UniqueName: \"kubernetes.io/projected/cce812aa-b1c4-4a27-95e5-51d5a36e5504-kube-api-access-chv6c\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:32:25.245479 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:25.245442    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ba15213-2643-474d-84f3-50a443cd9088-bundle" pod "5ba15213-2643-474d-84f3-50a443cd9088" (UID: "5ba15213-2643-474d-84f3-50a443cd9088"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:32:25.245739 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:25.245715    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6f12f5a-911b-4180-99f4-c6d70e670320-bundle" pod "b6f12f5a-911b-4180-99f4-c6d70e670320" (UID: "b6f12f5a-911b-4180-99f4-c6d70e670320"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:32:25.247442 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:25.247420    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6f12f5a-911b-4180-99f4-c6d70e670320-kube-api-access-9dzb6" pod "b6f12f5a-911b-4180-99f4-c6d70e670320" (UID: "b6f12f5a-911b-4180-99f4-c6d70e670320"). InnerVolumeSpecName "kube-api-access-9dzb6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:32:25.247562 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:25.247448    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ba15213-2643-474d-84f3-50a443cd9088-kube-api-access-lsr4f" pod "5ba15213-2643-474d-84f3-50a443cd9088" (UID: "5ba15213-2643-474d-84f3-50a443cd9088"). InnerVolumeSpecName "kube-api-access-lsr4f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:32:25.251395 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:25.251374    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6f12f5a-911b-4180-99f4-c6d70e670320-util" pod "b6f12f5a-911b-4180-99f4-c6d70e670320" (UID: "b6f12f5a-911b-4180-99f4-c6d70e670320"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:32:25.253483 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:25.253455    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ba15213-2643-474d-84f3-50a443cd9088-util" pod "5ba15213-2643-474d-84f3-50a443cd9088" (UID: "5ba15213-2643-474d-84f3-50a443cd9088"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:32:25.346116 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:25.346039    5060 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b6f12f5a-911b-4180-99f4-c6d70e670320-bundle\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:32:25.346116 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:25.346071    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lsr4f\" (UniqueName: \"kubernetes.io/projected/5ba15213-2643-474d-84f3-50a443cd9088-kube-api-access-lsr4f\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:32:25.346116 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:25.346083    5060 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5ba15213-2643-474d-84f3-50a443cd9088-bundle\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:32:25.346116 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:25.346091    5060 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b6f12f5a-911b-4180-99f4-c6d70e670320-util\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:32:25.346116 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:25.346099    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9dzb6\" (UniqueName: \"kubernetes.io/projected/b6f12f5a-911b-4180-99f4-c6d70e670320-kube-api-access-9dzb6\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:32:25.346116 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:25.346108    5060 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5ba15213-2643-474d-84f3-50a443cd9088-util\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:32:25.921239 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:25.921199    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aprjf2" event={"ID":"b6f12f5a-911b-4180-99f4-c6d70e670320","Type":"ContainerDied","Data":"dfcda53187003d00f9dd74ec81886eb709ab48b10e2dffda542ef176e6dd5470"}
Oct 31 12:32:25.921737 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:25.921285    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dfcda53187003d00f9dd74ec81886eb709ab48b10e2dffda542ef176e6dd5470"
Oct 31 12:32:25.921737 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:25.921209    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aprjf2"
Oct 31 12:32:25.923260 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:25.923236    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xj7b9" event={"ID":"cce812aa-b1c4-4a27-95e5-51d5a36e5504","Type":"ContainerDied","Data":"c9fcf8ea13ecfe67d30785d4789e3b57540b73b7e33db02f399f9bc1e7947b7b"}
Oct 31 12:32:25.923374 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:25.923279    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c9fcf8ea13ecfe67d30785d4789e3b57540b73b7e33db02f399f9bc1e7947b7b"
Oct 31 12:32:25.923374 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:25.923245    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xj7b9"
Oct 31 12:32:25.925525 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:25.925333    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8pxmf"
Oct 31 12:32:25.925525 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:25.925349    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8pxmf" event={"ID":"5ba15213-2643-474d-84f3-50a443cd9088","Type":"ContainerDied","Data":"dfcb0b73999696833ef1a27ff38f2541988bb6dda2eb6de1d5815074b423dda4"}
Oct 31 12:32:25.925525 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:25.925399    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dfcb0b73999696833ef1a27ff38f2541988bb6dda2eb6de1d5815074b423dda4"
Oct 31 12:32:25.927454 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:25.927428    5060 generic.go:356] "Generic (PLEG): container finished" podID="198de6ba-59d4-4be8-96e9-f0c83b16c7dc" containerID="d6ea7604b2cfbd66a926baafb07604f9237c1fd314f9092a823e69be5db05950" exitCode=0
Oct 31 12:32:25.927539 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:25.927512    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cgdfz8" event={"ID":"198de6ba-59d4-4be8-96e9-f0c83b16c7dc","Type":"ContainerDied","Data":"d6ea7604b2cfbd66a926baafb07604f9237c1fd314f9092a823e69be5db05950"}
Oct 31 12:32:26.056725 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:26.056701    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cw274p"
Oct 31 12:32:26.151991 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:26.151947    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00a1f4b5-4e4e-42b8-b539-f4da4db2a558-kube-api-access-f5fmq\" (UniqueName: \"kubernetes.io/projected/00a1f4b5-4e4e-42b8-b539-f4da4db2a558-kube-api-access-f5fmq\") pod \"00a1f4b5-4e4e-42b8-b539-f4da4db2a558\" (UID: \"00a1f4b5-4e4e-42b8-b539-f4da4db2a558\") "
Oct 31 12:32:26.152161 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:26.152001    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00a1f4b5-4e4e-42b8-b539-f4da4db2a558-bundle\" (UniqueName: \"kubernetes.io/empty-dir/00a1f4b5-4e4e-42b8-b539-f4da4db2a558-bundle\") pod \"00a1f4b5-4e4e-42b8-b539-f4da4db2a558\" (UID: \"00a1f4b5-4e4e-42b8-b539-f4da4db2a558\") "
Oct 31 12:32:26.152161 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:26.152053    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00a1f4b5-4e4e-42b8-b539-f4da4db2a558-util\" (UniqueName: \"kubernetes.io/empty-dir/00a1f4b5-4e4e-42b8-b539-f4da4db2a558-util\") pod \"00a1f4b5-4e4e-42b8-b539-f4da4db2a558\" (UID: \"00a1f4b5-4e4e-42b8-b539-f4da4db2a558\") "
Oct 31 12:32:26.152823 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:26.152794    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00a1f4b5-4e4e-42b8-b539-f4da4db2a558-bundle" pod "00a1f4b5-4e4e-42b8-b539-f4da4db2a558" (UID: "00a1f4b5-4e4e-42b8-b539-f4da4db2a558"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:32:26.154162 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:26.154135    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00a1f4b5-4e4e-42b8-b539-f4da4db2a558-kube-api-access-f5fmq" pod "00a1f4b5-4e4e-42b8-b539-f4da4db2a558" (UID: "00a1f4b5-4e4e-42b8-b539-f4da4db2a558"). InnerVolumeSpecName "kube-api-access-f5fmq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:32:26.158153 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:26.158124    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00a1f4b5-4e4e-42b8-b539-f4da4db2a558-util" pod "00a1f4b5-4e4e-42b8-b539-f4da4db2a558" (UID: "00a1f4b5-4e4e-42b8-b539-f4da4db2a558"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:32:26.253041 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:26.252954    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f5fmq\" (UniqueName: \"kubernetes.io/projected/00a1f4b5-4e4e-42b8-b539-f4da4db2a558-kube-api-access-f5fmq\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:32:26.253041 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:26.252986    5060 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/00a1f4b5-4e4e-42b8-b539-f4da4db2a558-bundle\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:32:26.253041 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:26.252999    5060 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/00a1f4b5-4e4e-42b8-b539-f4da4db2a558-util\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:32:26.933871 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:26.933828    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cw274p" event={"ID":"00a1f4b5-4e4e-42b8-b539-f4da4db2a558","Type":"ContainerDied","Data":"e9b234b5f5bcbaf9879a72be48332e339568943b15fd276b74bcdee2ddb66c46"}
Oct 31 12:32:26.933871 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:26.933860    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cw274p"
Oct 31 12:32:26.934338 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:26.933891    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e9b234b5f5bcbaf9879a72be48332e339568943b15fd276b74bcdee2ddb66c46"
Oct 31 12:32:27.060849 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:27.060824    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cgdfz8"
Oct 31 12:32:27.159217 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:27.159189    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/198de6ba-59d4-4be8-96e9-f0c83b16c7dc-kube-api-access-r4tnq\" (UniqueName: \"kubernetes.io/projected/198de6ba-59d4-4be8-96e9-f0c83b16c7dc-kube-api-access-r4tnq\") pod \"198de6ba-59d4-4be8-96e9-f0c83b16c7dc\" (UID: \"198de6ba-59d4-4be8-96e9-f0c83b16c7dc\") "
Oct 31 12:32:27.159391 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:27.159232    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/198de6ba-59d4-4be8-96e9-f0c83b16c7dc-bundle\" (UniqueName: \"kubernetes.io/empty-dir/198de6ba-59d4-4be8-96e9-f0c83b16c7dc-bundle\") pod \"198de6ba-59d4-4be8-96e9-f0c83b16c7dc\" (UID: \"198de6ba-59d4-4be8-96e9-f0c83b16c7dc\") "
Oct 31 12:32:27.159391 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:27.159269    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/198de6ba-59d4-4be8-96e9-f0c83b16c7dc-util\" (UniqueName: \"kubernetes.io/empty-dir/198de6ba-59d4-4be8-96e9-f0c83b16c7dc-util\") pod \"198de6ba-59d4-4be8-96e9-f0c83b16c7dc\" (UID: \"198de6ba-59d4-4be8-96e9-f0c83b16c7dc\") "
Oct 31 12:32:27.159932 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:27.159899    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/198de6ba-59d4-4be8-96e9-f0c83b16c7dc-bundle" pod "198de6ba-59d4-4be8-96e9-f0c83b16c7dc" (UID: "198de6ba-59d4-4be8-96e9-f0c83b16c7dc"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:32:27.161396 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:27.161363    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/198de6ba-59d4-4be8-96e9-f0c83b16c7dc-kube-api-access-r4tnq" pod "198de6ba-59d4-4be8-96e9-f0c83b16c7dc" (UID: "198de6ba-59d4-4be8-96e9-f0c83b16c7dc"). InnerVolumeSpecName "kube-api-access-r4tnq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:32:27.164922 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:27.164901    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/198de6ba-59d4-4be8-96e9-f0c83b16c7dc-util" pod "198de6ba-59d4-4be8-96e9-f0c83b16c7dc" (UID: "198de6ba-59d4-4be8-96e9-f0c83b16c7dc"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:32:27.260853 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:27.260769    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4tnq\" (UniqueName: \"kubernetes.io/projected/198de6ba-59d4-4be8-96e9-f0c83b16c7dc-kube-api-access-r4tnq\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:32:27.260853 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:27.260798    5060 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/198de6ba-59d4-4be8-96e9-f0c83b16c7dc-bundle\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:32:27.260853 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:27.260808    5060 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/198de6ba-59d4-4be8-96e9-f0c83b16c7dc-util\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:32:27.944956 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:27.944859    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cgdfz8" event={"ID":"198de6ba-59d4-4be8-96e9-f0c83b16c7dc","Type":"ContainerDied","Data":"ea4eb76bd40aa4d9cd18c47450f4f345b829b77ce4fa0e540d5d86a87547d040"}
Oct 31 12:32:27.945329 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:27.944886    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cgdfz8"
Oct 31 12:32:27.945329 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:27.944976    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ea4eb76bd40aa4d9cd18c47450f4f345b829b77ce4fa0e540d5d86a87547d040"
Oct 31 12:32:29.664977 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:29.664936    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-h45r8"]
Oct 31 12:32:29.665442 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:29.665403    5060 cpu_manager.go:414] "containerMap: removing container" podUID="00a1f4b5-4e4e-42b8-b539-f4da4db2a558" containerName="pull"
Oct 31 12:32:29.665442 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:29.665422    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00a1f4b5-4e4e-42b8-b539-f4da4db2a558" containerName="pull"
Oct 31 12:32:29.665442 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:29.665440    5060 cpu_manager.go:414] "containerMap: removing container" podUID="5ba15213-2643-474d-84f3-50a443cd9088" containerName="util"
Oct 31 12:32:29.665665 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:29.665449    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ba15213-2643-474d-84f3-50a443cd9088" containerName="util"
Oct 31 12:32:29.665665 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:29.665460    5060 cpu_manager.go:414] "containerMap: removing container" podUID="71e7abef-0f1f-4797-a4f6-1b6793083fb8" containerName="extract"
Oct 31 12:32:29.665665 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:29.665467    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71e7abef-0f1f-4797-a4f6-1b6793083fb8" containerName="extract"
Oct 31 12:32:29.665665 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:29.665479    5060 cpu_manager.go:414] "containerMap: removing container" podUID="a610abd9-13b7-493f-84a5-e6cb1cbd1262" containerName="extract"
Oct 31 12:32:29.665665 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:29.665486    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a610abd9-13b7-493f-84a5-e6cb1cbd1262" containerName="extract"
Oct 31 12:32:29.665665 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:29.665517    5060 cpu_manager.go:414] "containerMap: removing container" podUID="c0685425-0e8f-48ca-aa57-9a99616d1613" containerName="pull"
Oct 31 12:32:29.665665 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:29.665525    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0685425-0e8f-48ca-aa57-9a99616d1613" containerName="pull"
Oct 31 12:32:29.665665 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:29.665535    5060 cpu_manager.go:414] "containerMap: removing container" podUID="1bc5f088-e98b-4353-9512-39a3b11b9aba" containerName="pull"
Oct 31 12:32:29.665665 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:29.665542    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1bc5f088-e98b-4353-9512-39a3b11b9aba" containerName="pull"
Oct 31 12:32:29.665665 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:29.665552    5060 cpu_manager.go:414] "containerMap: removing container" podUID="c0685425-0e8f-48ca-aa57-9a99616d1613" containerName="extract"
Oct 31 12:32:29.665665 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:29.665559    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0685425-0e8f-48ca-aa57-9a99616d1613" containerName="extract"
Oct 31 12:32:29.665665 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:29.665568    5060 cpu_manager.go:414] "containerMap: removing container" podUID="b6f12f5a-911b-4180-99f4-c6d70e670320" containerName="extract"
Oct 31 12:32:29.665665 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:29.665575    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6f12f5a-911b-4180-99f4-c6d70e670320" containerName="extract"
Oct 31 12:32:29.665665 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:29.665591    5060 cpu_manager.go:414] "containerMap: removing container" podUID="00a1f4b5-4e4e-42b8-b539-f4da4db2a558" containerName="util"
Oct 31 12:32:29.665665 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:29.665599    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00a1f4b5-4e4e-42b8-b539-f4da4db2a558" containerName="util"
Oct 31 12:32:29.665665 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:29.665610    5060 cpu_manager.go:414] "containerMap: removing container" podUID="b4836388-df04-4576-80b8-4ecd90bbb1fe" containerName="pull"
Oct 31 12:32:29.665665 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:29.665617    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4836388-df04-4576-80b8-4ecd90bbb1fe" containerName="pull"
Oct 31 12:32:29.665665 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:29.665631    5060 cpu_manager.go:414] "containerMap: removing container" podUID="b4836388-df04-4576-80b8-4ecd90bbb1fe" containerName="extract"
Oct 31 12:32:29.665665 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:29.665639    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4836388-df04-4576-80b8-4ecd90bbb1fe" containerName="extract"
Oct 31 12:32:29.665665 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:29.665648    5060 cpu_manager.go:414] "containerMap: removing container" podUID="b6f12f5a-911b-4180-99f4-c6d70e670320" containerName="util"
Oct 31 12:32:29.665665 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:29.665656    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6f12f5a-911b-4180-99f4-c6d70e670320" containerName="util"
Oct 31 12:32:29.665665 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:29.665666    5060 cpu_manager.go:414] "containerMap: removing container" podUID="b4836388-df04-4576-80b8-4ecd90bbb1fe" containerName="util"
Oct 31 12:32:29.665665 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:29.665673    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4836388-df04-4576-80b8-4ecd90bbb1fe" containerName="util"
Oct 31 12:32:29.666721 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:29.665683    5060 cpu_manager.go:414] "containerMap: removing container" podUID="5ba15213-2643-474d-84f3-50a443cd9088" containerName="pull"
Oct 31 12:32:29.666721 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:29.665690    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ba15213-2643-474d-84f3-50a443cd9088" containerName="pull"
Oct 31 12:32:29.666721 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:29.665701    5060 cpu_manager.go:414] "containerMap: removing container" podUID="a610abd9-13b7-493f-84a5-e6cb1cbd1262" containerName="pull"
Oct 31 12:32:29.666721 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:29.665708    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a610abd9-13b7-493f-84a5-e6cb1cbd1262" containerName="pull"
Oct 31 12:32:29.666721 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:29.665719    5060 cpu_manager.go:414] "containerMap: removing container" podUID="b6f12f5a-911b-4180-99f4-c6d70e670320" containerName="pull"
Oct 31 12:32:29.666721 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:29.665725    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6f12f5a-911b-4180-99f4-c6d70e670320" containerName="pull"
Oct 31 12:32:29.666721 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:29.665735    5060 cpu_manager.go:414] "containerMap: removing container" podUID="cce812aa-b1c4-4a27-95e5-51d5a36e5504" containerName="pull"
Oct 31 12:32:29.666721 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:29.665743    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cce812aa-b1c4-4a27-95e5-51d5a36e5504" containerName="pull"
Oct 31 12:32:29.666721 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:29.665752    5060 cpu_manager.go:414] "containerMap: removing container" podUID="198de6ba-59d4-4be8-96e9-f0c83b16c7dc" containerName="pull"
Oct 31 12:32:29.666721 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:29.665759    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="198de6ba-59d4-4be8-96e9-f0c83b16c7dc" containerName="pull"
Oct 31 12:32:29.666721 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:29.665770    5060 cpu_manager.go:414] "containerMap: removing container" podUID="1bc5f088-e98b-4353-9512-39a3b11b9aba" containerName="util"
Oct 31 12:32:29.666721 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:29.665777    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1bc5f088-e98b-4353-9512-39a3b11b9aba" containerName="util"
Oct 31 12:32:29.666721 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:29.665786    5060 cpu_manager.go:414] "containerMap: removing container" podUID="5ba15213-2643-474d-84f3-50a443cd9088" containerName="extract"
Oct 31 12:32:29.666721 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:29.665792    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ba15213-2643-474d-84f3-50a443cd9088" containerName="extract"
Oct 31 12:32:29.666721 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:29.665803    5060 cpu_manager.go:414] "containerMap: removing container" podUID="00a1f4b5-4e4e-42b8-b539-f4da4db2a558" containerName="extract"
Oct 31 12:32:29.666721 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:29.665809    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00a1f4b5-4e4e-42b8-b539-f4da4db2a558" containerName="extract"
Oct 31 12:32:29.666721 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:29.665826    5060 cpu_manager.go:414] "containerMap: removing container" podUID="cce812aa-b1c4-4a27-95e5-51d5a36e5504" containerName="util"
Oct 31 12:32:29.666721 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:29.665834    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cce812aa-b1c4-4a27-95e5-51d5a36e5504" containerName="util"
Oct 31 12:32:29.666721 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:29.665844    5060 cpu_manager.go:414] "containerMap: removing container" podUID="c5fd411c-1234-4a0c-92ad-5904b87906c3" containerName="pull"
Oct 31 12:32:29.666721 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:29.665851    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5fd411c-1234-4a0c-92ad-5904b87906c3" containerName="pull"
Oct 31 12:32:29.666721 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:29.665864    5060 cpu_manager.go:414] "containerMap: removing container" podUID="c5fd411c-1234-4a0c-92ad-5904b87906c3" containerName="util"
Oct 31 12:32:29.666721 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:29.665871    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5fd411c-1234-4a0c-92ad-5904b87906c3" containerName="util"
Oct 31 12:32:29.666721 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:29.665880    5060 cpu_manager.go:414] "containerMap: removing container" podUID="cce812aa-b1c4-4a27-95e5-51d5a36e5504" containerName="extract"
Oct 31 12:32:29.666721 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:29.665888    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cce812aa-b1c4-4a27-95e5-51d5a36e5504" containerName="extract"
Oct 31 12:32:29.666721 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:29.665907    5060 cpu_manager.go:414] "containerMap: removing container" podUID="a610abd9-13b7-493f-84a5-e6cb1cbd1262" containerName="util"
Oct 31 12:32:29.666721 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:29.665914    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a610abd9-13b7-493f-84a5-e6cb1cbd1262" containerName="util"
Oct 31 12:32:29.666721 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:29.665928    5060 cpu_manager.go:414] "containerMap: removing container" podUID="198de6ba-59d4-4be8-96e9-f0c83b16c7dc" containerName="util"
Oct 31 12:32:29.666721 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:29.665937    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="198de6ba-59d4-4be8-96e9-f0c83b16c7dc" containerName="util"
Oct 31 12:32:29.666721 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:29.665948    5060 cpu_manager.go:414] "containerMap: removing container" podUID="71e7abef-0f1f-4797-a4f6-1b6793083fb8" containerName="util"
Oct 31 12:32:29.666721 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:29.665955    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71e7abef-0f1f-4797-a4f6-1b6793083fb8" containerName="util"
Oct 31 12:32:29.666721 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:29.665963    5060 cpu_manager.go:414] "containerMap: removing container" podUID="c5fd411c-1234-4a0c-92ad-5904b87906c3" containerName="extract"
Oct 31 12:32:29.666721 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:29.665970    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5fd411c-1234-4a0c-92ad-5904b87906c3" containerName="extract"
Oct 31 12:32:29.666721 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:29.665985    5060 cpu_manager.go:414] "containerMap: removing container" podUID="c0685425-0e8f-48ca-aa57-9a99616d1613" containerName="util"
Oct 31 12:32:29.666721 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:29.665992    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0685425-0e8f-48ca-aa57-9a99616d1613" containerName="util"
Oct 31 12:32:29.666721 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:29.666002    5060 cpu_manager.go:414] "containerMap: removing container" podUID="71e7abef-0f1f-4797-a4f6-1b6793083fb8" containerName="pull"
Oct 31 12:32:29.666721 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:29.666008    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71e7abef-0f1f-4797-a4f6-1b6793083fb8" containerName="pull"
Oct 31 12:32:29.666721 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:29.666017    5060 cpu_manager.go:414] "containerMap: removing container" podUID="1bc5f088-e98b-4353-9512-39a3b11b9aba" containerName="extract"
Oct 31 12:32:29.666721 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:29.666024    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1bc5f088-e98b-4353-9512-39a3b11b9aba" containerName="extract"
Oct 31 12:32:29.666721 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:29.666034    5060 cpu_manager.go:414] "containerMap: removing container" podUID="198de6ba-59d4-4be8-96e9-f0c83b16c7dc" containerName="extract"
Oct 31 12:32:29.666721 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:29.666042    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="198de6ba-59d4-4be8-96e9-f0c83b16c7dc" containerName="extract"
Oct 31 12:32:29.666721 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:29.666135    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="cce812aa-b1c4-4a27-95e5-51d5a36e5504" containerName="extract"
Oct 31 12:32:29.666721 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:29.666150    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5fd411c-1234-4a0c-92ad-5904b87906c3" containerName="extract"
Oct 31 12:32:29.666721 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:29.666160    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="1bc5f088-e98b-4353-9512-39a3b11b9aba" containerName="extract"
Oct 31 12:32:29.666721 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:29.666168    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="198de6ba-59d4-4be8-96e9-f0c83b16c7dc" containerName="extract"
Oct 31 12:32:29.666721 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:29.666178    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="71e7abef-0f1f-4797-a4f6-1b6793083fb8" containerName="extract"
Oct 31 12:32:29.666721 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:29.666187    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="a610abd9-13b7-493f-84a5-e6cb1cbd1262" containerName="extract"
Oct 31 12:32:29.666721 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:29.666196    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4836388-df04-4576-80b8-4ecd90bbb1fe" containerName="extract"
Oct 31 12:32:29.666721 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:29.666205    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6f12f5a-911b-4180-99f4-c6d70e670320" containerName="extract"
Oct 31 12:32:29.666721 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:29.666213    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0685425-0e8f-48ca-aa57-9a99616d1613" containerName="extract"
Oct 31 12:32:29.666721 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:29.666221    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ba15213-2643-474d-84f3-50a443cd9088" containerName="extract"
Oct 31 12:32:29.666721 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:29.666231    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="00a1f4b5-4e4e-42b8-b539-f4da4db2a558" containerName="extract"
Oct 31 12:32:29.668629 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:29.668235    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-h45r8"
Oct 31 12:32:29.673626 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:29.673599    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\""
Oct 31 12:32:29.676308 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:29.676281    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\""
Oct 31 12:32:29.677628 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:29.677598    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-operator-dockercfg-blhxm\""
Oct 31 12:32:29.679726 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:29.679704    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-h45r8"]
Oct 31 12:32:29.781180 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:29.781142    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ef57947a-5e0a-43df-aebb-641564a41208-tmp\") pod \"jobset-operator-674cd59dc5-h45r8\" (UID: \"ef57947a-5e0a-43df-aebb-641564a41208\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-h45r8"
Oct 31 12:32:29.781327 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:29.781271    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79fb5\" (UniqueName: \"kubernetes.io/projected/ef57947a-5e0a-43df-aebb-641564a41208-kube-api-access-79fb5\") pod \"jobset-operator-674cd59dc5-h45r8\" (UID: \"ef57947a-5e0a-43df-aebb-641564a41208\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-h45r8"
Oct 31 12:32:29.882246 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:29.882210    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79fb5\" (UniqueName: \"kubernetes.io/projected/ef57947a-5e0a-43df-aebb-641564a41208-kube-api-access-79fb5\") pod \"jobset-operator-674cd59dc5-h45r8\" (UID: \"ef57947a-5e0a-43df-aebb-641564a41208\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-h45r8"
Oct 31 12:32:29.882420 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:29.882262    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ef57947a-5e0a-43df-aebb-641564a41208-tmp\") pod \"jobset-operator-674cd59dc5-h45r8\" (UID: \"ef57947a-5e0a-43df-aebb-641564a41208\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-h45r8"
Oct 31 12:32:29.882642 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:29.882624    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ef57947a-5e0a-43df-aebb-641564a41208-tmp\") pod \"jobset-operator-674cd59dc5-h45r8\" (UID: \"ef57947a-5e0a-43df-aebb-641564a41208\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-h45r8"
Oct 31 12:32:29.894960 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:29.894934    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79fb5\" (UniqueName: \"kubernetes.io/projected/ef57947a-5e0a-43df-aebb-641564a41208-kube-api-access-79fb5\") pod \"jobset-operator-674cd59dc5-h45r8\" (UID: \"ef57947a-5e0a-43df-aebb-641564a41208\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-h45r8"
Oct 31 12:32:29.918314 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:32:29.918235    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7phb" containerName="extractor"
Oct 31 12:32:29.986814 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:29.986770    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-h45r8"
Oct 31 12:32:30.258947 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:30.258923    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-h45r8"]
Oct 31 12:32:30.261387 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:32:30.261359    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podef57947a_5e0a_43df_aebb_641564a41208.slice/crio-487090abd6e2b3b4cc8d524188063229bc3a009c31203622808cb80031761fe5 WatchSource:0}: Error finding container 487090abd6e2b3b4cc8d524188063229bc3a009c31203622808cb80031761fe5: Status 404 returned error can't find the container with id 487090abd6e2b3b4cc8d524188063229bc3a009c31203622808cb80031761fe5
Oct 31 12:32:30.262854 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:30.262831    5060 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 12:32:30.960037 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:30.959998    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-h45r8" event={"ID":"ef57947a-5e0a-43df-aebb-641564a41208","Type":"ContainerStarted","Data":"487090abd6e2b3b4cc8d524188063229bc3a009c31203622808cb80031761fe5"}
Oct 31 12:32:35.214198 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:35.214158    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5967ccc7d4-tbkp7"]
Oct 31 12:32:35.218830 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:35.218811    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5967ccc7d4-tbkp7"
Oct 31 12:32:35.230877 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:35.230854    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Oct 31 12:32:35.230877 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:35.230866    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Oct 31 12:32:35.231699 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:35.231683    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Oct 31 12:32:35.231830 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:35.231815    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Oct 31 12:32:35.231928 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:35.231916    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Oct 31 12:32:35.231994 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:35.231981    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-b82ch\""
Oct 31 12:32:35.251847 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:35.251822    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5967ccc7d4-tbkp7"]
Oct 31 12:32:35.328693 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:35.328650    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/9a2f809e-4e2d-4aa2-baf1-c9a4b24e9f96-manifests\") pod \"opendatahub-operator-controller-manager-5967ccc7d4-tbkp7\" (UID: \"9a2f809e-4e2d-4aa2-baf1-c9a4b24e9f96\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5967ccc7d4-tbkp7"
Oct 31 12:32:35.328887 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:35.328765    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/9a2f809e-4e2d-4aa2-baf1-c9a4b24e9f96-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5967ccc7d4-tbkp7\" (UID: \"9a2f809e-4e2d-4aa2-baf1-c9a4b24e9f96\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5967ccc7d4-tbkp7"
Oct 31 12:32:35.328942 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:35.328879    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9a2f809e-4e2d-4aa2-baf1-c9a4b24e9f96-tmp\") pod \"opendatahub-operator-controller-manager-5967ccc7d4-tbkp7\" (UID: \"9a2f809e-4e2d-4aa2-baf1-c9a4b24e9f96\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5967ccc7d4-tbkp7"
Oct 31 12:32:35.328942 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:35.328918    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9a2f809e-4e2d-4aa2-baf1-c9a4b24e9f96-webhook-cert\") pod \"opendatahub-operator-controller-manager-5967ccc7d4-tbkp7\" (UID: \"9a2f809e-4e2d-4aa2-baf1-c9a4b24e9f96\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5967ccc7d4-tbkp7"
Oct 31 12:32:35.329012 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:35.328949    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/9a2f809e-4e2d-4aa2-baf1-c9a4b24e9f96-metrics-cert\") pod \"opendatahub-operator-controller-manager-5967ccc7d4-tbkp7\" (UID: \"9a2f809e-4e2d-4aa2-baf1-c9a4b24e9f96\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5967ccc7d4-tbkp7"
Oct 31 12:32:35.329012 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:35.328976    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hdmp\" (UniqueName: \"kubernetes.io/projected/9a2f809e-4e2d-4aa2-baf1-c9a4b24e9f96-kube-api-access-6hdmp\") pod \"opendatahub-operator-controller-manager-5967ccc7d4-tbkp7\" (UID: \"9a2f809e-4e2d-4aa2-baf1-c9a4b24e9f96\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5967ccc7d4-tbkp7"
Oct 31 12:32:35.430083 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:35.430046    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hdmp\" (UniqueName: \"kubernetes.io/projected/9a2f809e-4e2d-4aa2-baf1-c9a4b24e9f96-kube-api-access-6hdmp\") pod \"opendatahub-operator-controller-manager-5967ccc7d4-tbkp7\" (UID: \"9a2f809e-4e2d-4aa2-baf1-c9a4b24e9f96\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5967ccc7d4-tbkp7"
Oct 31 12:32:35.430278 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:35.430104    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/9a2f809e-4e2d-4aa2-baf1-c9a4b24e9f96-manifests\") pod \"opendatahub-operator-controller-manager-5967ccc7d4-tbkp7\" (UID: \"9a2f809e-4e2d-4aa2-baf1-c9a4b24e9f96\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5967ccc7d4-tbkp7"
Oct 31 12:32:35.430278 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:35.430128    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/9a2f809e-4e2d-4aa2-baf1-c9a4b24e9f96-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5967ccc7d4-tbkp7\" (UID: \"9a2f809e-4e2d-4aa2-baf1-c9a4b24e9f96\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5967ccc7d4-tbkp7"
Oct 31 12:32:35.430278 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:35.430156    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9a2f809e-4e2d-4aa2-baf1-c9a4b24e9f96-tmp\") pod \"opendatahub-operator-controller-manager-5967ccc7d4-tbkp7\" (UID: \"9a2f809e-4e2d-4aa2-baf1-c9a4b24e9f96\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5967ccc7d4-tbkp7"
Oct 31 12:32:35.430278 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:35.430179    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9a2f809e-4e2d-4aa2-baf1-c9a4b24e9f96-webhook-cert\") pod \"opendatahub-operator-controller-manager-5967ccc7d4-tbkp7\" (UID: \"9a2f809e-4e2d-4aa2-baf1-c9a4b24e9f96\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5967ccc7d4-tbkp7"
Oct 31 12:32:35.430278 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:35.430224    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/9a2f809e-4e2d-4aa2-baf1-c9a4b24e9f96-metrics-cert\") pod \"opendatahub-operator-controller-manager-5967ccc7d4-tbkp7\" (UID: \"9a2f809e-4e2d-4aa2-baf1-c9a4b24e9f96\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5967ccc7d4-tbkp7"
Oct 31 12:32:35.430679 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:35.430657    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/9a2f809e-4e2d-4aa2-baf1-c9a4b24e9f96-manifests\") pod \"opendatahub-operator-controller-manager-5967ccc7d4-tbkp7\" (UID: \"9a2f809e-4e2d-4aa2-baf1-c9a4b24e9f96\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5967ccc7d4-tbkp7"
Oct 31 12:32:35.432934 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:35.432911    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/9a2f809e-4e2d-4aa2-baf1-c9a4b24e9f96-metrics-cert\") pod \"opendatahub-operator-controller-manager-5967ccc7d4-tbkp7\" (UID: \"9a2f809e-4e2d-4aa2-baf1-c9a4b24e9f96\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5967ccc7d4-tbkp7"
Oct 31 12:32:35.436097 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:35.436068    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9a2f809e-4e2d-4aa2-baf1-c9a4b24e9f96-webhook-cert\") pod \"opendatahub-operator-controller-manager-5967ccc7d4-tbkp7\" (UID: \"9a2f809e-4e2d-4aa2-baf1-c9a4b24e9f96\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5967ccc7d4-tbkp7"
Oct 31 12:32:35.436217 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:35.436077    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/9a2f809e-4e2d-4aa2-baf1-c9a4b24e9f96-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5967ccc7d4-tbkp7\" (UID: \"9a2f809e-4e2d-4aa2-baf1-c9a4b24e9f96\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5967ccc7d4-tbkp7"
Oct 31 12:32:35.437296 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:35.437268    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9a2f809e-4e2d-4aa2-baf1-c9a4b24e9f96-tmp\") pod \"opendatahub-operator-controller-manager-5967ccc7d4-tbkp7\" (UID: \"9a2f809e-4e2d-4aa2-baf1-c9a4b24e9f96\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5967ccc7d4-tbkp7"
Oct 31 12:32:35.442978 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:35.442956    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hdmp\" (UniqueName: \"kubernetes.io/projected/9a2f809e-4e2d-4aa2-baf1-c9a4b24e9f96-kube-api-access-6hdmp\") pod \"opendatahub-operator-controller-manager-5967ccc7d4-tbkp7\" (UID: \"9a2f809e-4e2d-4aa2-baf1-c9a4b24e9f96\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5967ccc7d4-tbkp7"
Oct 31 12:32:35.533928 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:35.533894    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5967ccc7d4-tbkp7"
Oct 31 12:32:35.836310 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:35.836274    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5967ccc7d4-tbkp7"]
Oct 31 12:32:35.836469 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:32:35.836286    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a2f809e_4e2d_4aa2_baf1_c9a4b24e9f96.slice/crio-35366bbba47ce3a2aaeda65bebfc86e1b20ba58764646159c76b65f511f4020d WatchSource:0}: Error finding container 35366bbba47ce3a2aaeda65bebfc86e1b20ba58764646159c76b65f511f4020d: Status 404 returned error can't find the container with id 35366bbba47ce3a2aaeda65bebfc86e1b20ba58764646159c76b65f511f4020d
Oct 31 12:32:35.986453 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:35.986373    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5967ccc7d4-tbkp7" event={"ID":"9a2f809e-4e2d-4aa2-baf1-c9a4b24e9f96","Type":"ContainerStarted","Data":"35366bbba47ce3a2aaeda65bebfc86e1b20ba58764646159c76b65f511f4020d"}
Oct 31 12:32:38.004916 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:38.004862    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-h45r8" event={"ID":"ef57947a-5e0a-43df-aebb-641564a41208","Type":"ContainerStarted","Data":"2f35565fd2b1e2e901ec8cbceb8f8cfab4d855586f7567c5cba4fef10498f68f"}
Oct 31 12:32:38.024288 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:38.024232    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-h45r8" podStartSLOduration=1.499143598 podStartE2EDuration="9.024217278s" totalImagesPullingTime="7.52507368s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:32:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:32:38.023276395 +0000 UTC m=+321.126344108" watchObservedRunningTime="2026-10-31 12:32:38.024217278 +0000 UTC m=+321.127284990"
Oct 31 12:32:39.973312 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:39.973271    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-kfgxr"]
Oct 31 12:32:39.976676 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:39.976659    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-kfgxr"
Oct 31 12:32:39.983433 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:39.983412    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Oct 31 12:32:39.983569 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:39.983460    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Oct 31 12:32:39.983569 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:39.983533    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-jll8w\""
Oct 31 12:32:39.995679 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:39.995646    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-kfgxr"]
Oct 31 12:32:40.018746 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:40.018654    5060 generic.go:356] "Generic (PLEG): container finished" podID="9a2f809e-4e2d-4aa2-baf1-c9a4b24e9f96" containerID="8b92207c2d7cdd7803a8dbc8e1284142b8cba0825df189d4c9314fe8cace3a38" exitCode=0
Oct 31 12:32:40.018900 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:40.018730    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5967ccc7d4-tbkp7" event={"ID":"9a2f809e-4e2d-4aa2-baf1-c9a4b24e9f96","Type":"ContainerDied","Data":"8b92207c2d7cdd7803a8dbc8e1284142b8cba0825df189d4c9314fe8cace3a38"}
Oct 31 12:32:40.070236 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:40.070193    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c55a7e00-969d-4e83-b84a-b9447e9c49d4-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-kfgxr\" (UID: \"c55a7e00-969d-4e83-b84a-b9447e9c49d4\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-kfgxr"
Oct 31 12:32:40.070418 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:40.070262    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vm54w\" (UniqueName: \"kubernetes.io/projected/c55a7e00-969d-4e83-b84a-b9447e9c49d4-kube-api-access-vm54w\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-kfgxr\" (UID: \"c55a7e00-969d-4e83-b84a-b9447e9c49d4\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-kfgxr"
Oct 31 12:32:40.171435 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:40.171392    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c55a7e00-969d-4e83-b84a-b9447e9c49d4-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-kfgxr\" (UID: \"c55a7e00-969d-4e83-b84a-b9447e9c49d4\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-kfgxr"
Oct 31 12:32:40.171638 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:40.171456    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vm54w\" (UniqueName: \"kubernetes.io/projected/c55a7e00-969d-4e83-b84a-b9447e9c49d4-kube-api-access-vm54w\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-kfgxr\" (UID: \"c55a7e00-969d-4e83-b84a-b9447e9c49d4\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-kfgxr"
Oct 31 12:32:40.171819 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:40.171791    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c55a7e00-969d-4e83-b84a-b9447e9c49d4-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-kfgxr\" (UID: \"c55a7e00-969d-4e83-b84a-b9447e9c49d4\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-kfgxr"
Oct 31 12:32:40.211136 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:40.211103    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vm54w\" (UniqueName: \"kubernetes.io/projected/c55a7e00-969d-4e83-b84a-b9447e9c49d4-kube-api-access-vm54w\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-kfgxr\" (UID: \"c55a7e00-969d-4e83-b84a-b9447e9c49d4\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-kfgxr"
Oct 31 12:32:40.287927 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:40.287893    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-kfgxr"
Oct 31 12:32:40.571317 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:40.571290    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-kfgxr"]
Oct 31 12:32:40.574090 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:32:40.574058    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc55a7e00_969d_4e83_b84a_b9447e9c49d4.slice/crio-49be540f9996fff7b8b6ea3f33a26ec441a3d79c0d0380e709d0095eef960c60 WatchSource:0}: Error finding container 49be540f9996fff7b8b6ea3f33a26ec441a3d79c0d0380e709d0095eef960c60: Status 404 returned error can't find the container with id 49be540f9996fff7b8b6ea3f33a26ec441a3d79c0d0380e709d0095eef960c60
Oct 31 12:32:41.025304 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:41.025266    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-kfgxr" event={"ID":"c55a7e00-969d-4e83-b84a-b9447e9c49d4","Type":"ContainerStarted","Data":"49be540f9996fff7b8b6ea3f33a26ec441a3d79c0d0380e709d0095eef960c60"}
Oct 31 12:32:41.027146 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:41.027119    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5967ccc7d4-tbkp7" event={"ID":"9a2f809e-4e2d-4aa2-baf1-c9a4b24e9f96","Type":"ContainerStarted","Data":"97e844e8336eb0efd874ffd92f1e3eadf9f127b40871c7bd8d10dc67ae99d350"}
Oct 31 12:32:41.027274 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:41.027254    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5967ccc7d4-tbkp7"
Oct 31 12:32:41.075612 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:41.075558    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5967ccc7d4-tbkp7" podStartSLOduration=1.833681004 podStartE2EDuration="6.075545277s" totalImagesPullingTime="3.766578323s" totalInitContainerRuntime="475.28595ms" isStatefulPod=false podCreationTimestamp="2026-10-31 12:32:35 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:32:41.072833429 +0000 UTC m=+324.175901134" watchObservedRunningTime="2026-10-31 12:32:41.075545277 +0000 UTC m=+324.178612988"
Oct 31 12:32:44.045354 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:44.045320    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-kfgxr" event={"ID":"c55a7e00-969d-4e83-b84a-b9447e9c49d4","Type":"ContainerStarted","Data":"8f0abdd7326613a0691c3ae2d1f6357117bc24eeca3a77e4090dd900c4cf7594"}
Oct 31 12:32:44.075370 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:44.075317    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-kfgxr" podStartSLOduration=2.272789299 podStartE2EDuration="5.075303422s" totalImagesPullingTime="2.802514123s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:32:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:32:44.073538632 +0000 UTC m=+327.176606348" watchObservedRunningTime="2026-10-31 12:32:44.075303422 +0000 UTC m=+327.178371133"
Oct 31 12:32:45.593327 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:45.593289    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-kgjh7"]
Oct 31 12:32:45.599590 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:45.599572    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-kgjh7"
Oct 31 12:32:45.616109 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:45.616085    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l55br\" (UniqueName: \"kubernetes.io/projected/4f8464cd-b924-44f9-821b-f4c683531f45-kube-api-access-l55br\") pod \"cert-manager-webhook-5dbbbf75cc-kgjh7\" (UID: \"4f8464cd-b924-44f9-821b-f4c683531f45\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-kgjh7"
Oct 31 12:32:45.616216 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:45.616145    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4f8464cd-b924-44f9-821b-f4c683531f45-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-kgjh7\" (UID: \"4f8464cd-b924-44f9-821b-f4c683531f45\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-kgjh7"
Oct 31 12:32:45.618751 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:45.618704    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Oct 31 12:32:45.618897 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:45.618737    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-xbwms\""
Oct 31 12:32:45.620066 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:45.620040    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-kgjh7"]
Oct 31 12:32:45.716633 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:45.716601    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Oct 31 12:32:45.716809 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:45.716693    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4f8464cd-b924-44f9-821b-f4c683531f45-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-kgjh7\" (UID: \"4f8464cd-b924-44f9-821b-f4c683531f45\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-kgjh7"
Oct 31 12:32:45.716809 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:45.716745    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l55br\" (UniqueName: \"kubernetes.io/projected/4f8464cd-b924-44f9-821b-f4c683531f45-kube-api-access-l55br\") pod \"cert-manager-webhook-5dbbbf75cc-kgjh7\" (UID: \"4f8464cd-b924-44f9-821b-f4c683531f45\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-kgjh7"
Oct 31 12:32:45.730194 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:45.730163    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4f8464cd-b924-44f9-821b-f4c683531f45-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-kgjh7\" (UID: \"4f8464cd-b924-44f9-821b-f4c683531f45\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-kgjh7"
Oct 31 12:32:45.730311 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:45.730237    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l55br\" (UniqueName: \"kubernetes.io/projected/4f8464cd-b924-44f9-821b-f4c683531f45-kube-api-access-l55br\") pod \"cert-manager-webhook-5dbbbf75cc-kgjh7\" (UID: \"4f8464cd-b924-44f9-821b-f4c683531f45\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-kgjh7"
Oct 31 12:32:45.911540 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:45.911436    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-kgjh7"
Oct 31 12:32:46.189123 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:46.185507    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-kgjh7"]
Oct 31 12:32:46.192345 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:32:46.192310    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4f8464cd_b924_44f9_821b_f4c683531f45.slice/crio-f451d165795a14f4c3b6179d9f56da61b30e7512f05a9579539931cafc7aa0a1 WatchSource:0}: Error finding container f451d165795a14f4c3b6179d9f56da61b30e7512f05a9579539931cafc7aa0a1: Status 404 returned error can't find the container with id f451d165795a14f4c3b6179d9f56da61b30e7512f05a9579539931cafc7aa0a1
Oct 31 12:32:47.059994 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:47.059959    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-kgjh7" event={"ID":"4f8464cd-b924-44f9-821b-f4c683531f45","Type":"ContainerStarted","Data":"f451d165795a14f4c3b6179d9f56da61b30e7512f05a9579539931cafc7aa0a1"}
Oct 31 12:32:49.985875 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:49.985837    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-rh8gs"]
Oct 31 12:32:49.989349 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:49.989327    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-rh8gs"
Oct 31 12:32:50.045306 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:50.045260    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Oct 31 12:32:50.045306 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:50.045287    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Oct 31 12:32:50.045306 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:50.045270    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-l2tnf\""
Oct 31 12:32:50.046854 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:50.046826    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-rh8gs"]
Oct 31 12:32:50.056641 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:50.056612    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxlwk\" (UniqueName: \"kubernetes.io/projected/3b9c999a-c93e-40fc-9404-2b8ad6ae8e92-kube-api-access-vxlwk\") pod \"kuadrant-operator-controller-manager-cf8469d7b-rh8gs\" (UID: \"3b9c999a-c93e-40fc-9404-2b8ad6ae8e92\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-rh8gs"
Oct 31 12:32:50.056769 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:50.056671    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3b9c999a-c93e-40fc-9404-2b8ad6ae8e92-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-rh8gs\" (UID: \"3b9c999a-c93e-40fc-9404-2b8ad6ae8e92\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-rh8gs"
Oct 31 12:32:50.076695 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:50.076657    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-kgjh7" event={"ID":"4f8464cd-b924-44f9-821b-f4c683531f45","Type":"ContainerStarted","Data":"fb57cd2f00a9f01ca99de20156ee45cd880c4a3a8f08db8add76534a15a88f68"}
Oct 31 12:32:50.076838 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:50.076808    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-kgjh7"
Oct 31 12:32:50.105171 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:50.105116    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-kgjh7" podStartSLOduration=1.421476465 podStartE2EDuration="5.105103344s" totalImagesPullingTime="3.683626879s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:32:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:32:50.103037026 +0000 UTC m=+333.206104740" watchObservedRunningTime="2026-10-31 12:32:50.105103344 +0000 UTC m=+333.208171055"
Oct 31 12:32:50.163765 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:50.158389    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3b9c999a-c93e-40fc-9404-2b8ad6ae8e92-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-rh8gs\" (UID: \"3b9c999a-c93e-40fc-9404-2b8ad6ae8e92\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-rh8gs"
Oct 31 12:32:50.163765 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:50.158574    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxlwk\" (UniqueName: \"kubernetes.io/projected/3b9c999a-c93e-40fc-9404-2b8ad6ae8e92-kube-api-access-vxlwk\") pod \"kuadrant-operator-controller-manager-cf8469d7b-rh8gs\" (UID: \"3b9c999a-c93e-40fc-9404-2b8ad6ae8e92\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-rh8gs"
Oct 31 12:32:50.163765 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:50.159538    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3b9c999a-c93e-40fc-9404-2b8ad6ae8e92-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-rh8gs\" (UID: \"3b9c999a-c93e-40fc-9404-2b8ad6ae8e92\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-rh8gs"
Oct 31 12:32:50.171852 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:50.171820    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxlwk\" (UniqueName: \"kubernetes.io/projected/3b9c999a-c93e-40fc-9404-2b8ad6ae8e92-kube-api-access-vxlwk\") pod \"kuadrant-operator-controller-manager-cf8469d7b-rh8gs\" (UID: \"3b9c999a-c93e-40fc-9404-2b8ad6ae8e92\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-rh8gs"
Oct 31 12:32:50.301761 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:50.301721    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-rh8gs"
Oct 31 12:32:50.605437 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:50.605342    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-rh8gs"]
Oct 31 12:32:50.608147 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:32:50.608111    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b9c999a_c93e_40fc_9404_2b8ad6ae8e92.slice/crio-802c0a86f8c674ae76454487b5f0965985209aa18c3b51a6783783c85a98e768 WatchSource:0}: Error finding container 802c0a86f8c674ae76454487b5f0965985209aa18c3b51a6783783c85a98e768: Status 404 returned error can't find the container with id 802c0a86f8c674ae76454487b5f0965985209aa18c3b51a6783783c85a98e768
Oct 31 12:32:51.083247 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:51.083206    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-rh8gs" event={"ID":"3b9c999a-c93e-40fc-9404-2b8ad6ae8e92","Type":"ContainerStarted","Data":"802c0a86f8c674ae76454487b5f0965985209aa18c3b51a6783783c85a98e768"}
Oct 31 12:32:52.035594 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:52.035558    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5967ccc7d4-tbkp7"
Oct 31 12:32:54.294153 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:54.294108    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-crpzx"]
Oct 31 12:32:54.297831 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:54.297808    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-crpzx"
Oct 31 12:32:54.312591 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:54.312562    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-p6nbl\""
Oct 31 12:32:54.318529 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:54.318502    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-crpzx"]
Oct 31 12:32:54.318699 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:54.318684    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Oct 31 12:32:54.394063 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:54.394027    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2xbr\" (UniqueName: \"kubernetes.io/projected/b674e577-791f-43ef-9ad8-81938802d946-kube-api-access-v2xbr\") pod \"dns-operator-controller-manager-8557cbdb84-crpzx\" (UID: \"b674e577-791f-43ef-9ad8-81938802d946\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-crpzx"
Oct 31 12:32:54.494827 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:54.494789    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2xbr\" (UniqueName: \"kubernetes.io/projected/b674e577-791f-43ef-9ad8-81938802d946-kube-api-access-v2xbr\") pod \"dns-operator-controller-manager-8557cbdb84-crpzx\" (UID: \"b674e577-791f-43ef-9ad8-81938802d946\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-crpzx"
Oct 31 12:32:54.510711 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:54.510683    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2xbr\" (UniqueName: \"kubernetes.io/projected/b674e577-791f-43ef-9ad8-81938802d946-kube-api-access-v2xbr\") pod \"dns-operator-controller-manager-8557cbdb84-crpzx\" (UID: \"b674e577-791f-43ef-9ad8-81938802d946\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-crpzx"
Oct 31 12:32:54.611648 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:54.611563    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-crpzx"
Oct 31 12:32:54.905455 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:54.905382    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-crpzx"]
Oct 31 12:32:54.908799 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:32:54.908763    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb674e577_791f_43ef_9ad8_81938802d946.slice/crio-b81a54cd1236f92c995d14bd5f630e809b347f07ddf321ceff176472f48d6afb WatchSource:0}: Error finding container b81a54cd1236f92c995d14bd5f630e809b347f07ddf321ceff176472f48d6afb: Status 404 returned error can't find the container with id b81a54cd1236f92c995d14bd5f630e809b347f07ddf321ceff176472f48d6afb
Oct 31 12:32:55.110862 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:55.110826    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-crpzx" event={"ID":"b674e577-791f-43ef-9ad8-81938802d946","Type":"ContainerStarted","Data":"b81a54cd1236f92c995d14bd5f630e809b347f07ddf321ceff176472f48d6afb"}
Oct 31 12:32:56.085721 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:56.085678    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-kgjh7"
Oct 31 12:32:57.123929 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:57.123889    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-crpzx" event={"ID":"b674e577-791f-43ef-9ad8-81938802d946","Type":"ContainerStarted","Data":"7e16c41d2a93cc60f6c3ccfe932c93e7d0eddf67947f4e22d07f49ef2ac70629"}
Oct 31 12:32:57.124350 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:57.124049    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-crpzx"
Oct 31 12:32:57.125506 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:57.125464    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-rh8gs" event={"ID":"3b9c999a-c93e-40fc-9404-2b8ad6ae8e92","Type":"ContainerStarted","Data":"bc1bd6d913edee0d7d54e85e494ac17643e61821a12e4706c64d99628daf0d2a"}
Oct 31 12:32:57.125618 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:57.125537    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-rh8gs"
Oct 31 12:32:57.142949 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:57.142873    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-crpzx" podStartSLOduration=1.174060209 podStartE2EDuration="3.142862667s" totalImagesPullingTime="1.968802458s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:32:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:32:57.142369593 +0000 UTC m=+340.245437305" watchObservedRunningTime="2026-10-31 12:32:57.142862667 +0000 UTC m=+340.245930378"
Oct 31 12:32:57.174053 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:57.174001    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-rh8gs" podStartSLOduration=1.907660923 podStartE2EDuration="8.173987682s" totalImagesPullingTime="6.266326759s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:32:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:32:57.17198768 +0000 UTC m=+340.275055391" watchObservedRunningTime="2026-10-31 12:32:57.173987682 +0000 UTC m=+340.277055394"
Oct 31 12:32:57.555438 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:57.555398    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-jbwh5"]
Oct 31 12:32:57.559543 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:57.559518    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-jbwh5"
Oct 31 12:32:57.570035 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:57.570014    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-kqk6n\""
Oct 31 12:32:57.580153 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:57.580131    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-jbwh5"]
Oct 31 12:32:57.621962 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:57.621916    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42h5q\" (UniqueName: \"kubernetes.io/projected/2eda55f0-194c-462f-b1d0-dc52a2b4a669-kube-api-access-42h5q\") pod \"authorino-operator-7b7bb4bc77-jbwh5\" (UID: \"2eda55f0-194c-462f-b1d0-dc52a2b4a669\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-jbwh5"
Oct 31 12:32:57.723223 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:57.723188    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42h5q\" (UniqueName: \"kubernetes.io/projected/2eda55f0-194c-462f-b1d0-dc52a2b4a669-kube-api-access-42h5q\") pod \"authorino-operator-7b7bb4bc77-jbwh5\" (UID: \"2eda55f0-194c-462f-b1d0-dc52a2b4a669\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-jbwh5"
Oct 31 12:32:57.736117 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:57.736090    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42h5q\" (UniqueName: \"kubernetes.io/projected/2eda55f0-194c-462f-b1d0-dc52a2b4a669-kube-api-access-42h5q\") pod \"authorino-operator-7b7bb4bc77-jbwh5\" (UID: \"2eda55f0-194c-462f-b1d0-dc52a2b4a669\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-jbwh5"
Oct 31 12:32:57.872588 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:57.872485    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-jbwh5"
Oct 31 12:32:58.152710 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:58.152682    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-jbwh5"]
Oct 31 12:32:58.154566 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:32:58.154520    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2eda55f0_194c_462f_b1d0_dc52a2b4a669.slice/crio-7c28c2224e25bf0f507075a865e9660aab2fb0fb9a25c016fc7b051f22cb3e81 WatchSource:0}: Error finding container 7c28c2224e25bf0f507075a865e9660aab2fb0fb9a25c016fc7b051f22cb3e81: Status 404 returned error can't find the container with id 7c28c2224e25bf0f507075a865e9660aab2fb0fb9a25c016fc7b051f22cb3e81
Oct 31 12:32:59.137084 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:32:59.137046    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-jbwh5" event={"ID":"2eda55f0-194c-462f-b1d0-dc52a2b4a669","Type":"ContainerStarted","Data":"7c28c2224e25bf0f507075a865e9660aab2fb0fb9a25c016fc7b051f22cb3e81"}
Oct 31 12:32:59.923768 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:32:59.923732    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7phb" containerName="extractor"
Oct 31 12:33:00.143610 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:33:00.143522    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-jbwh5" event={"ID":"2eda55f0-194c-462f-b1d0-dc52a2b4a669","Type":"ContainerStarted","Data":"1c939e370faefba382500d71d183c4af62fb9e281ee7c9a276e2df995ef505aa"}
Oct 31 12:33:00.143610 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:33:00.143592    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-jbwh5"
Oct 31 12:33:00.162325 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:33:00.162265    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-7b7bb4bc77-jbwh5" podStartSLOduration=1.458574646 podStartE2EDuration="3.162251796s" totalImagesPullingTime="1.70367715s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:32:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:33:00.161550263 +0000 UTC m=+343.264617976" watchObservedRunningTime="2026-10-31 12:33:00.162251796 +0000 UTC m=+343.265319504"
Oct 31 12:33:00.649522 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:33:00.649471    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-vhq87"]
Oct 31 12:33:00.653266 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:33:00.653242    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-vhq87"
Oct 31 12:33:00.663351 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:33:00.663328    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-25kkp\""
Oct 31 12:33:00.666086 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:33:00.666056    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-vhq87"]
Oct 31 12:33:00.748347 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:33:00.748306    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxtpf\" (UniqueName: \"kubernetes.io/projected/e3209cd3-5f31-4d69-95d8-7ec4cf348ddf-kube-api-access-xxtpf\") pod \"limitador-operator-controller-manager-5778bb4449-vhq87\" (UID: \"e3209cd3-5f31-4d69-95d8-7ec4cf348ddf\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-vhq87"
Oct 31 12:33:00.849670 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:33:00.849632    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxtpf\" (UniqueName: \"kubernetes.io/projected/e3209cd3-5f31-4d69-95d8-7ec4cf348ddf-kube-api-access-xxtpf\") pod \"limitador-operator-controller-manager-5778bb4449-vhq87\" (UID: \"e3209cd3-5f31-4d69-95d8-7ec4cf348ddf\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-vhq87"
Oct 31 12:33:00.867785 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:33:00.867758    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxtpf\" (UniqueName: \"kubernetes.io/projected/e3209cd3-5f31-4d69-95d8-7ec4cf348ddf-kube-api-access-xxtpf\") pod \"limitador-operator-controller-manager-5778bb4449-vhq87\" (UID: \"e3209cd3-5f31-4d69-95d8-7ec4cf348ddf\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-vhq87"
Oct 31 12:33:00.966586 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:33:00.966480    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-vhq87"
Oct 31 12:33:01.246912 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:33:01.246884    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-vhq87"]
Oct 31 12:33:01.248470 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:33:01.248432    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3209cd3_5f31_4d69_95d8_7ec4cf348ddf.slice/crio-3033e24e636d0947ef62c0f0ce0be4c15089b698590634b7bb77cf0753c58690 WatchSource:0}: Error finding container 3033e24e636d0947ef62c0f0ce0be4c15089b698590634b7bb77cf0753c58690: Status 404 returned error can't find the container with id 3033e24e636d0947ef62c0f0ce0be4c15089b698590634b7bb77cf0753c58690
Oct 31 12:33:02.155555 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:33:02.155519    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-vhq87" event={"ID":"e3209cd3-5f31-4d69-95d8-7ec4cf348ddf","Type":"ContainerStarted","Data":"3033e24e636d0947ef62c0f0ce0be4c15089b698590634b7bb77cf0753c58690"}
Oct 31 12:33:04.168299 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:33:04.168269    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-vhq87" event={"ID":"e3209cd3-5f31-4d69-95d8-7ec4cf348ddf","Type":"ContainerStarted","Data":"a07f36c9d1aeb47ac04196dfd5e1a2f5ee21c6868a4c3c451ed44a3c09a0328b"}
Oct 31 12:33:04.168712 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:33:04.168333    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-vhq87"
Oct 31 12:33:04.188834 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:33:04.188785    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-vhq87" podStartSLOduration=1.965871835 podStartE2EDuration="4.1887728s" totalImagesPullingTime="2.222900965s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:33:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:33:04.186345918 +0000 UTC m=+347.289413630" watchObservedRunningTime="2026-10-31 12:33:04.1887728 +0000 UTC m=+347.291840513"
Oct 31 12:33:08.133623 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:33:08.133584    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-rh8gs"
Oct 31 12:33:08.134200 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:33:08.133685    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-crpzx"
Oct 31 12:33:09.154585 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:33:09.154550    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-tcnsn"]
Oct 31 12:33:09.158243 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:33:09.158224    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-tcnsn"
Oct 31 12:33:09.168331 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:33:09.168308    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-vcwt2\""
Oct 31 12:33:09.171578 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:33:09.171551    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-tcnsn"]
Oct 31 12:33:09.222829 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:33:09.222795    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvtxk\" (UniqueName: \"kubernetes.io/projected/e24808b7-d1c6-41bb-b5bf-9a6b628623bd-kube-api-access-rvtxk\") pod \"authorino-57f7f8659f-tcnsn\" (UID: \"e24808b7-d1c6-41bb-b5bf-9a6b628623bd\") " pod="kuadrant-system/authorino-57f7f8659f-tcnsn"
Oct 31 12:33:09.324251 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:33:09.324212    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvtxk\" (UniqueName: \"kubernetes.io/projected/e24808b7-d1c6-41bb-b5bf-9a6b628623bd-kube-api-access-rvtxk\") pod \"authorino-57f7f8659f-tcnsn\" (UID: \"e24808b7-d1c6-41bb-b5bf-9a6b628623bd\") " pod="kuadrant-system/authorino-57f7f8659f-tcnsn"
Oct 31 12:33:09.340064 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:33:09.340030    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvtxk\" (UniqueName: \"kubernetes.io/projected/e24808b7-d1c6-41bb-b5bf-9a6b628623bd-kube-api-access-rvtxk\") pod \"authorino-57f7f8659f-tcnsn\" (UID: \"e24808b7-d1c6-41bb-b5bf-9a6b628623bd\") " pod="kuadrant-system/authorino-57f7f8659f-tcnsn"
Oct 31 12:33:09.469884 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:33:09.469805    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-tcnsn"
Oct 31 12:33:09.759305 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:33:09.759277    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-tcnsn"]
Oct 31 12:33:09.760855 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:33:09.760826    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode24808b7_d1c6_41bb_b5bf_9a6b628623bd.slice/crio-2f08563f7e9c68a5287e7e137c25c1fe50cba85b97afe9e39eda2785ebdc8cbe WatchSource:0}: Error finding container 2f08563f7e9c68a5287e7e137c25c1fe50cba85b97afe9e39eda2785ebdc8cbe: Status 404 returned error can't find the container with id 2f08563f7e9c68a5287e7e137c25c1fe50cba85b97afe9e39eda2785ebdc8cbe
Oct 31 12:33:10.202688 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:33:10.202654    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-tcnsn" event={"ID":"e24808b7-d1c6-41bb-b5bf-9a6b628623bd","Type":"ContainerStarted","Data":"2f08563f7e9c68a5287e7e137c25c1fe50cba85b97afe9e39eda2785ebdc8cbe"}
Oct 31 12:33:11.151735 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:33:11.151703    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-jbwh5"
Oct 31 12:33:11.238225 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:33:11.238192    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7pf4p"]
Oct 31 12:33:11.241719 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:33:11.241694    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7pf4p"
Oct 31 12:33:11.257094 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:33:11.257067    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7pf4p"]
Oct 31 12:33:11.341217 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:33:11.341181    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0f6d8e05-a27e-49b1-ba33-cfea30956f83-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-7pf4p\" (UID: \"0f6d8e05-a27e-49b1-ba33-cfea30956f83\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7pf4p"
Oct 31 12:33:11.341378 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:33:11.341290    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47b6x\" (UniqueName: \"kubernetes.io/projected/0f6d8e05-a27e-49b1-ba33-cfea30956f83-kube-api-access-47b6x\") pod \"kuadrant-operator-controller-manager-678d87766f-7pf4p\" (UID: \"0f6d8e05-a27e-49b1-ba33-cfea30956f83\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7pf4p"
Oct 31 12:33:11.442425 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:33:11.442341    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0f6d8e05-a27e-49b1-ba33-cfea30956f83-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-7pf4p\" (UID: \"0f6d8e05-a27e-49b1-ba33-cfea30956f83\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7pf4p"
Oct 31 12:33:11.442601 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:33:11.442425    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47b6x\" (UniqueName: \"kubernetes.io/projected/0f6d8e05-a27e-49b1-ba33-cfea30956f83-kube-api-access-47b6x\") pod \"kuadrant-operator-controller-manager-678d87766f-7pf4p\" (UID: \"0f6d8e05-a27e-49b1-ba33-cfea30956f83\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7pf4p"
Oct 31 12:33:11.442833 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:33:11.442811    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0f6d8e05-a27e-49b1-ba33-cfea30956f83-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-7pf4p\" (UID: \"0f6d8e05-a27e-49b1-ba33-cfea30956f83\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7pf4p"
Oct 31 12:33:11.456586 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:33:11.456560    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47b6x\" (UniqueName: \"kubernetes.io/projected/0f6d8e05-a27e-49b1-ba33-cfea30956f83-kube-api-access-47b6x\") pod \"kuadrant-operator-controller-manager-678d87766f-7pf4p\" (UID: \"0f6d8e05-a27e-49b1-ba33-cfea30956f83\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7pf4p"
Oct 31 12:33:11.556208 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:33:11.556175    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7pf4p"
Oct 31 12:33:11.869977 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:33:11.869944    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7pf4p"]
Oct 31 12:33:11.869977 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:33:11.869968    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f6d8e05_a27e_49b1_ba33_cfea30956f83.slice/crio-c89d7fccb9320216f02f56118726d48d40dc7932db2b55817e3deb8310a81edd WatchSource:0}: Error finding container c89d7fccb9320216f02f56118726d48d40dc7932db2b55817e3deb8310a81edd: Status 404 returned error can't find the container with id c89d7fccb9320216f02f56118726d48d40dc7932db2b55817e3deb8310a81edd
Oct 31 12:33:12.215986 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:33:12.215893    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7pf4p" event={"ID":"0f6d8e05-a27e-49b1-ba33-cfea30956f83","Type":"ContainerStarted","Data":"ccdaf64db1370f052353f5b067fc2aa20246e7e773bb32a9850de4065f1db8ff"}
Oct 31 12:33:12.215986 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:33:12.215962    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7pf4p" event={"ID":"0f6d8e05-a27e-49b1-ba33-cfea30956f83","Type":"ContainerStarted","Data":"c89d7fccb9320216f02f56118726d48d40dc7932db2b55817e3deb8310a81edd"}
Oct 31 12:33:12.216204 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:33:12.216094    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7pf4p"
Oct 31 12:33:12.268030 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:33:12.267975    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7pf4p" podStartSLOduration=1.267961068 podStartE2EDuration="1.267961068s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:33:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:33:12.2676413 +0000 UTC m=+355.370709013" watchObservedRunningTime="2026-10-31 12:33:12.267961068 +0000 UTC m=+355.371028779"
Oct 31 12:33:14.228196 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:33:14.228105    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-tcnsn" event={"ID":"e24808b7-d1c6-41bb-b5bf-9a6b628623bd","Type":"ContainerStarted","Data":"67471da2c5ffb8fb78901faef39af8b415a3a761bef418f9c19ae2e2a115b3df"}
Oct 31 12:33:14.252148 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:33:14.252097    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-tcnsn" podStartSLOduration=1.231711929 podStartE2EDuration="5.252081952s" totalImagesPullingTime="4.020370023s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:33:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:33:14.250055625 +0000 UTC m=+357.353123350" watchObservedRunningTime="2026-10-31 12:33:14.252081952 +0000 UTC m=+357.355149697"
Oct 31 12:33:15.175873 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:33:15.175844    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-vhq87"
Oct 31 12:33:23.224465 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:33:23.224427    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7pf4p"
Oct 31 12:33:23.297746 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:33:23.297706    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-rh8gs"]
Oct 31 12:33:23.297958 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:33:23.297918    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-rh8gs" podUID="3b9c999a-c93e-40fc-9404-2b8ad6ae8e92" containerName="manager" containerID="cri-o://bc1bd6d913edee0d7d54e85e494ac17643e61821a12e4706c64d99628daf0d2a" gracePeriod=10
Oct 31 12:33:23.536625 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:33:23.536600    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-rh8gs"
Oct 31 12:33:23.643559 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:33:23.643526    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b9c999a-c93e-40fc-9404-2b8ad6ae8e92-kube-api-access-vxlwk\" (UniqueName: \"kubernetes.io/projected/3b9c999a-c93e-40fc-9404-2b8ad6ae8e92-kube-api-access-vxlwk\") pod \"3b9c999a-c93e-40fc-9404-2b8ad6ae8e92\" (UID: \"3b9c999a-c93e-40fc-9404-2b8ad6ae8e92\") "
Oct 31 12:33:23.643725 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:33:23.643567    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b9c999a-c93e-40fc-9404-2b8ad6ae8e92-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3b9c999a-c93e-40fc-9404-2b8ad6ae8e92-extensions-socket-volume\") pod \"3b9c999a-c93e-40fc-9404-2b8ad6ae8e92\" (UID: \"3b9c999a-c93e-40fc-9404-2b8ad6ae8e92\") "
Oct 31 12:33:23.643941 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:33:23.643920    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b9c999a-c93e-40fc-9404-2b8ad6ae8e92-extensions-socket-volume" pod "3b9c999a-c93e-40fc-9404-2b8ad6ae8e92" (UID: "3b9c999a-c93e-40fc-9404-2b8ad6ae8e92"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:33:23.645775 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:33:23.645755    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b9c999a-c93e-40fc-9404-2b8ad6ae8e92-kube-api-access-vxlwk" pod "3b9c999a-c93e-40fc-9404-2b8ad6ae8e92" (UID: "3b9c999a-c93e-40fc-9404-2b8ad6ae8e92"). InnerVolumeSpecName "kube-api-access-vxlwk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:33:23.745091 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:33:23.745051    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vxlwk\" (UniqueName: \"kubernetes.io/projected/3b9c999a-c93e-40fc-9404-2b8ad6ae8e92-kube-api-access-vxlwk\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:33:23.745091 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:33:23.745085    5060 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3b9c999a-c93e-40fc-9404-2b8ad6ae8e92-extensions-socket-volume\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:33:24.280174 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:33:24.280131    5060 generic.go:356] "Generic (PLEG): container finished" podID="3b9c999a-c93e-40fc-9404-2b8ad6ae8e92" containerID="bc1bd6d913edee0d7d54e85e494ac17643e61821a12e4706c64d99628daf0d2a" exitCode=0
Oct 31 12:33:24.280664 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:33:24.280197    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-rh8gs"
Oct 31 12:33:24.280664 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:33:24.280202    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-rh8gs" event={"ID":"3b9c999a-c93e-40fc-9404-2b8ad6ae8e92","Type":"ContainerDied","Data":"bc1bd6d913edee0d7d54e85e494ac17643e61821a12e4706c64d99628daf0d2a"}
Oct 31 12:33:24.280664 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:33:24.280281    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-rh8gs" event={"ID":"3b9c999a-c93e-40fc-9404-2b8ad6ae8e92","Type":"ContainerDied","Data":"802c0a86f8c674ae76454487b5f0965985209aa18c3b51a6783783c85a98e768"}
Oct 31 12:33:24.280664 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:33:24.280303    5060 scope.go:122] "RemoveContainer" containerID="bc1bd6d913edee0d7d54e85e494ac17643e61821a12e4706c64d99628daf0d2a"
Oct 31 12:33:24.291704 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:33:24.291687    5060 scope.go:122] "RemoveContainer" containerID="bc1bd6d913edee0d7d54e85e494ac17643e61821a12e4706c64d99628daf0d2a"
Oct 31 12:33:24.291988 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:33:24.291965    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc1bd6d913edee0d7d54e85e494ac17643e61821a12e4706c64d99628daf0d2a\": container with ID starting with bc1bd6d913edee0d7d54e85e494ac17643e61821a12e4706c64d99628daf0d2a not found: ID does not exist" containerID="bc1bd6d913edee0d7d54e85e494ac17643e61821a12e4706c64d99628daf0d2a"
Oct 31 12:33:24.292039 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:33:24.292001    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc1bd6d913edee0d7d54e85e494ac17643e61821a12e4706c64d99628daf0d2a"} err="failed to get container status \"bc1bd6d913edee0d7d54e85e494ac17643e61821a12e4706c64d99628daf0d2a\": rpc error: code = NotFound desc = could not find container \"bc1bd6d913edee0d7d54e85e494ac17643e61821a12e4706c64d99628daf0d2a\": container with ID starting with bc1bd6d913edee0d7d54e85e494ac17643e61821a12e4706c64d99628daf0d2a not found: ID does not exist"
Oct 31 12:33:24.304745 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:33:24.304721    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-rh8gs"]
Oct 31 12:33:24.319868 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:33:24.319835    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-rh8gs"]
Oct 31 12:33:25.046602 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:33:25.046570    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b9c999a-c93e-40fc-9404-2b8ad6ae8e92" path="/var/lib/kubelet/pods/3b9c999a-c93e-40fc-9404-2b8ad6ae8e92/volumes"
Oct 31 12:33:29.921241 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:33:29.921208    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7phb" containerName="extractor"
Oct 31 12:33:32.366448 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:33:32.366414    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-6698b9df8-qxvfh"]
Oct 31 12:33:32.366876 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:33:32.366772    5060 cpu_manager.go:414] "containerMap: removing container" podUID="3b9c999a-c93e-40fc-9404-2b8ad6ae8e92" containerName="manager"
Oct 31 12:33:32.366876 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:33:32.366785    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b9c999a-c93e-40fc-9404-2b8ad6ae8e92" containerName="manager"
Oct 31 12:33:32.366876 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:33:32.366873    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b9c999a-c93e-40fc-9404-2b8ad6ae8e92" containerName="manager"
Oct 31 12:33:32.394504 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:33:32.394414    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-6698b9df8-qxvfh"]
Oct 31 12:33:32.394651 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:33:32.394536    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-6698b9df8-qxvfh"
Oct 31 12:33:32.398500 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:33:32.398462    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\""
Oct 31 12:33:32.399633 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:33:32.399610    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\""
Oct 31 12:33:32.399728 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:33:32.399691    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-vxnb8\""
Oct 31 12:33:32.399728 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:33:32.399694    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\""
Oct 31 12:33:32.522677 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:33:32.522638    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6a410bf-ca1f-4204-ac07-31fa2cb086a4-metrics-certs\") pod \"jobset-controller-manager-6698b9df8-qxvfh\" (UID: \"f6a410bf-ca1f-4204-ac07-31fa2cb086a4\") " pod="openshift-jobset-operator/jobset-controller-manager-6698b9df8-qxvfh"
Oct 31 12:33:32.522677 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:33:32.522689    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dswf4\" (UniqueName: \"kubernetes.io/projected/f6a410bf-ca1f-4204-ac07-31fa2cb086a4-kube-api-access-dswf4\") pod \"jobset-controller-manager-6698b9df8-qxvfh\" (UID: \"f6a410bf-ca1f-4204-ac07-31fa2cb086a4\") " pod="openshift-jobset-operator/jobset-controller-manager-6698b9df8-qxvfh"
Oct 31 12:33:32.522926 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:33:32.522772    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/f6a410bf-ca1f-4204-ac07-31fa2cb086a4-manager-config\") pod \"jobset-controller-manager-6698b9df8-qxvfh\" (UID: \"f6a410bf-ca1f-4204-ac07-31fa2cb086a4\") " pod="openshift-jobset-operator/jobset-controller-manager-6698b9df8-qxvfh"
Oct 31 12:33:32.522926 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:33:32.522805    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f6a410bf-ca1f-4204-ac07-31fa2cb086a4-cert\") pod \"jobset-controller-manager-6698b9df8-qxvfh\" (UID: \"f6a410bf-ca1f-4204-ac07-31fa2cb086a4\") " pod="openshift-jobset-operator/jobset-controller-manager-6698b9df8-qxvfh"
Oct 31 12:33:32.623824 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:33:32.623738    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6a410bf-ca1f-4204-ac07-31fa2cb086a4-metrics-certs\") pod \"jobset-controller-manager-6698b9df8-qxvfh\" (UID: \"f6a410bf-ca1f-4204-ac07-31fa2cb086a4\") " pod="openshift-jobset-operator/jobset-controller-manager-6698b9df8-qxvfh"
Oct 31 12:33:32.623824 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:33:32.623779    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dswf4\" (UniqueName: \"kubernetes.io/projected/f6a410bf-ca1f-4204-ac07-31fa2cb086a4-kube-api-access-dswf4\") pod \"jobset-controller-manager-6698b9df8-qxvfh\" (UID: \"f6a410bf-ca1f-4204-ac07-31fa2cb086a4\") " pod="openshift-jobset-operator/jobset-controller-manager-6698b9df8-qxvfh"
Oct 31 12:33:32.623824 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:33:32.623809    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/f6a410bf-ca1f-4204-ac07-31fa2cb086a4-manager-config\") pod \"jobset-controller-manager-6698b9df8-qxvfh\" (UID: \"f6a410bf-ca1f-4204-ac07-31fa2cb086a4\") " pod="openshift-jobset-operator/jobset-controller-manager-6698b9df8-qxvfh"
Oct 31 12:33:32.624117 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:33:32.623833    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f6a410bf-ca1f-4204-ac07-31fa2cb086a4-cert\") pod \"jobset-controller-manager-6698b9df8-qxvfh\" (UID: \"f6a410bf-ca1f-4204-ac07-31fa2cb086a4\") " pod="openshift-jobset-operator/jobset-controller-manager-6698b9df8-qxvfh"
Oct 31 12:33:32.624678 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:33:32.624651    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/f6a410bf-ca1f-4204-ac07-31fa2cb086a4-manager-config\") pod \"jobset-controller-manager-6698b9df8-qxvfh\" (UID: \"f6a410bf-ca1f-4204-ac07-31fa2cb086a4\") " pod="openshift-jobset-operator/jobset-controller-manager-6698b9df8-qxvfh"
Oct 31 12:33:32.626442 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:33:32.626415    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f6a410bf-ca1f-4204-ac07-31fa2cb086a4-cert\") pod \"jobset-controller-manager-6698b9df8-qxvfh\" (UID: \"f6a410bf-ca1f-4204-ac07-31fa2cb086a4\") " pod="openshift-jobset-operator/jobset-controller-manager-6698b9df8-qxvfh"
Oct 31 12:33:32.626551 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:33:32.626446    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6a410bf-ca1f-4204-ac07-31fa2cb086a4-metrics-certs\") pod \"jobset-controller-manager-6698b9df8-qxvfh\" (UID: \"f6a410bf-ca1f-4204-ac07-31fa2cb086a4\") " pod="openshift-jobset-operator/jobset-controller-manager-6698b9df8-qxvfh"
Oct 31 12:33:32.635996 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:33:32.635968    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dswf4\" (UniqueName: \"kubernetes.io/projected/f6a410bf-ca1f-4204-ac07-31fa2cb086a4-kube-api-access-dswf4\") pod \"jobset-controller-manager-6698b9df8-qxvfh\" (UID: \"f6a410bf-ca1f-4204-ac07-31fa2cb086a4\") " pod="openshift-jobset-operator/jobset-controller-manager-6698b9df8-qxvfh"
Oct 31 12:33:32.722843 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:33:32.722799    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-6698b9df8-qxvfh"
Oct 31 12:33:33.015885 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:33:33.015852    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-6698b9df8-qxvfh"]
Oct 31 12:33:33.017558 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:33:33.017520    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6a410bf_ca1f_4204_ac07_31fa2cb086a4.slice/crio-d246a44fa04041179adcbcee6aaee3fdb5f81818c0c7a498337568b98e720911 WatchSource:0}: Error finding container d246a44fa04041179adcbcee6aaee3fdb5f81818c0c7a498337568b98e720911: Status 404 returned error can't find the container with id d246a44fa04041179adcbcee6aaee3fdb5f81818c0c7a498337568b98e720911
Oct 31 12:33:33.330824 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:33:33.330788    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-6698b9df8-qxvfh" event={"ID":"f6a410bf-ca1f-4204-ac07-31fa2cb086a4","Type":"ContainerStarted","Data":"d246a44fa04041179adcbcee6aaee3fdb5f81818c0c7a498337568b98e720911"}
Oct 31 12:33:36.353035 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:33:36.352997    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-6698b9df8-qxvfh" event={"ID":"f6a410bf-ca1f-4204-ac07-31fa2cb086a4","Type":"ContainerStarted","Data":"ec36f004f672304c950f45abfb97f317984555109c6164f2ed0bba83888998df"}
Oct 31 12:33:36.353547 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:33:36.353105    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-6698b9df8-qxvfh"
Oct 31 12:33:36.376114 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:33:36.376069    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-controller-manager-6698b9df8-qxvfh" podStartSLOduration=1.504622796 podStartE2EDuration="4.376055236s" totalImagesPullingTime="2.87143244s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:33:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:33:36.374392483 +0000 UTC m=+379.477460194" watchObservedRunningTime="2026-10-31 12:33:36.376055236 +0000 UTC m=+379.479122947"
Oct 31 12:33:37.107573 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:33:37.107540    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-c5hhm"]
Oct 31 12:33:37.111114 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:33:37.111095    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-c5hhm"
Oct 31 12:33:37.118824 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:33:37.118393    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-c5hhm"]
Oct 31 12:33:37.164642 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:33:37.164609    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lqrk\" (UniqueName: \"kubernetes.io/projected/79a9d422-eb6c-416c-a6a9-575fcb2299cb-kube-api-access-9lqrk\") pod \"authorino-56c6c9ddd5-c5hhm\" (UID: \"79a9d422-eb6c-416c-a6a9-575fcb2299cb\") " pod="kuadrant-system/authorino-56c6c9ddd5-c5hhm"
Oct 31 12:33:37.265099 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:33:37.265068    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lqrk\" (UniqueName: \"kubernetes.io/projected/79a9d422-eb6c-416c-a6a9-575fcb2299cb-kube-api-access-9lqrk\") pod \"authorino-56c6c9ddd5-c5hhm\" (UID: \"79a9d422-eb6c-416c-a6a9-575fcb2299cb\") " pod="kuadrant-system/authorino-56c6c9ddd5-c5hhm"
Oct 31 12:33:37.277659 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:33:37.277627    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lqrk\" (UniqueName: \"kubernetes.io/projected/79a9d422-eb6c-416c-a6a9-575fcb2299cb-kube-api-access-9lqrk\") pod \"authorino-56c6c9ddd5-c5hhm\" (UID: \"79a9d422-eb6c-416c-a6a9-575fcb2299cb\") " pod="kuadrant-system/authorino-56c6c9ddd5-c5hhm"
Oct 31 12:33:37.424861 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:33:37.424773    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-c5hhm"
Oct 31 12:33:37.703092 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:33:37.703059    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-c5hhm"]
Oct 31 12:33:37.705326 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:33:37.705296    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod79a9d422_eb6c_416c_a6a9_575fcb2299cb.slice/crio-2ff9abc07270047277a15bda4a816f36122304e05dfb87063971dd317134a90a WatchSource:0}: Error finding container 2ff9abc07270047277a15bda4a816f36122304e05dfb87063971dd317134a90a: Status 404 returned error can't find the container with id 2ff9abc07270047277a15bda4a816f36122304e05dfb87063971dd317134a90a
Oct 31 12:33:38.364270 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:33:38.364239    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-c5hhm" event={"ID":"79a9d422-eb6c-416c-a6a9-575fcb2299cb","Type":"ContainerStarted","Data":"2ff9abc07270047277a15bda4a816f36122304e05dfb87063971dd317134a90a"}
Oct 31 12:33:39.372110 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:33:39.372067    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-c5hhm" event={"ID":"79a9d422-eb6c-416c-a6a9-575fcb2299cb","Type":"ContainerStarted","Data":"8a9e5d51a4df7e9be3e75400270f55c1f3b016c5872e8b18505ffa946720598e"}
Oct 31 12:33:39.391834 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:33:39.391786    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-c5hhm" podStartSLOduration=1.5298494740000002 podStartE2EDuration="2.391772513s" totalImagesPullingTime="861.923039ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:33:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:33:39.39097449 +0000 UTC m=+382.494042200" watchObservedRunningTime="2026-10-31 12:33:39.391772513 +0000 UTC m=+382.494840256"
Oct 31 12:33:39.421109 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:33:39.421072    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-tcnsn"]
Oct 31 12:33:39.421334 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:33:39.421308    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-tcnsn" podUID="e24808b7-d1c6-41bb-b5bf-9a6b628623bd" containerName="authorino" containerID="cri-o://67471da2c5ffb8fb78901faef39af8b415a3a761bef418f9c19ae2e2a115b3df" gracePeriod=30
Oct 31 12:33:39.676759 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:33:39.676731    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-tcnsn"
Oct 31 12:33:39.686058 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:33:39.686034    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e24808b7-d1c6-41bb-b5bf-9a6b628623bd-kube-api-access-rvtxk\" (UniqueName: \"kubernetes.io/projected/e24808b7-d1c6-41bb-b5bf-9a6b628623bd-kube-api-access-rvtxk\") pod \"e24808b7-d1c6-41bb-b5bf-9a6b628623bd\" (UID: \"e24808b7-d1c6-41bb-b5bf-9a6b628623bd\") "
Oct 31 12:33:39.688033 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:33:39.688002    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e24808b7-d1c6-41bb-b5bf-9a6b628623bd-kube-api-access-rvtxk" pod "e24808b7-d1c6-41bb-b5bf-9a6b628623bd" (UID: "e24808b7-d1c6-41bb-b5bf-9a6b628623bd"). InnerVolumeSpecName "kube-api-access-rvtxk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:33:39.787363 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:33:39.787324    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvtxk\" (UniqueName: \"kubernetes.io/projected/e24808b7-d1c6-41bb-b5bf-9a6b628623bd-kube-api-access-rvtxk\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:33:40.379748 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:33:40.379713    5060 generic.go:356] "Generic (PLEG): container finished" podID="e24808b7-d1c6-41bb-b5bf-9a6b628623bd" containerID="67471da2c5ffb8fb78901faef39af8b415a3a761bef418f9c19ae2e2a115b3df" exitCode=0
Oct 31 12:33:40.380229 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:33:40.379758    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-tcnsn"
Oct 31 12:33:40.380229 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:33:40.379790    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-tcnsn" event={"ID":"e24808b7-d1c6-41bb-b5bf-9a6b628623bd","Type":"ContainerDied","Data":"67471da2c5ffb8fb78901faef39af8b415a3a761bef418f9c19ae2e2a115b3df"}
Oct 31 12:33:40.380229 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:33:40.379834    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-tcnsn" event={"ID":"e24808b7-d1c6-41bb-b5bf-9a6b628623bd","Type":"ContainerDied","Data":"2f08563f7e9c68a5287e7e137c25c1fe50cba85b97afe9e39eda2785ebdc8cbe"}
Oct 31 12:33:40.380229 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:33:40.379851    5060 scope.go:122] "RemoveContainer" containerID="67471da2c5ffb8fb78901faef39af8b415a3a761bef418f9c19ae2e2a115b3df"
Oct 31 12:33:40.389286 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:33:40.389267    5060 scope.go:122] "RemoveContainer" containerID="67471da2c5ffb8fb78901faef39af8b415a3a761bef418f9c19ae2e2a115b3df"
Oct 31 12:33:40.389686 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:33:40.389668    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67471da2c5ffb8fb78901faef39af8b415a3a761bef418f9c19ae2e2a115b3df\": container with ID starting with 67471da2c5ffb8fb78901faef39af8b415a3a761bef418f9c19ae2e2a115b3df not found: ID does not exist" containerID="67471da2c5ffb8fb78901faef39af8b415a3a761bef418f9c19ae2e2a115b3df"
Oct 31 12:33:40.389744 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:33:40.389696    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67471da2c5ffb8fb78901faef39af8b415a3a761bef418f9c19ae2e2a115b3df"} err="failed to get container status \"67471da2c5ffb8fb78901faef39af8b415a3a761bef418f9c19ae2e2a115b3df\": rpc error: code = NotFound desc = could not find container \"67471da2c5ffb8fb78901faef39af8b415a3a761bef418f9c19ae2e2a115b3df\": container with ID starting with 67471da2c5ffb8fb78901faef39af8b415a3a761bef418f9c19ae2e2a115b3df not found: ID does not exist"
Oct 31 12:33:40.401165 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:33:40.401137    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-tcnsn"]
Oct 31 12:33:40.405259 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:33:40.405229    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-tcnsn"]
Oct 31 12:33:41.046409 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:33:41.046376    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e24808b7-d1c6-41bb-b5bf-9a6b628623bd" path="/var/lib/kubelet/pods/e24808b7-d1c6-41bb-b5bf-9a6b628623bd/volumes"
Oct 31 12:33:47.363342 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:33:47.363302    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-6698b9df8-qxvfh"
Oct 31 12:33:59.920073 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:33:59.919994    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7phb" containerName="extractor"
Oct 31 12:34:29.919430 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:34:29.919380    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7phb" containerName="extractor"
Oct 31 12:34:59.920587 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:34:59.920546    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7phb" containerName="extractor"
Oct 31 12:35:02.915373 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:35:02.915337    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6dd76875b5-kbssv"]
Oct 31 12:35:02.915789 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:35:02.915722    5060 cpu_manager.go:414] "containerMap: removing container" podUID="e24808b7-d1c6-41bb-b5bf-9a6b628623bd" containerName="authorino"
Oct 31 12:35:02.915789 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:35:02.915735    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e24808b7-d1c6-41bb-b5bf-9a6b628623bd" containerName="authorino"
Oct 31 12:35:02.915860 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:35:02.915816    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="e24808b7-d1c6-41bb-b5bf-9a6b628623bd" containerName="authorino"
Oct 31 12:35:02.918900 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:35:02.918876    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6dd76875b5-kbssv"
Oct 31 12:35:02.928918 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:35:02.928895    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Oct 31 12:35:02.930642 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:35:02.930615    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6dd76875b5-kbssv"]
Oct 31 12:35:02.957793 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:35:02.957760    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6dd76875b5-kbssv"]
Oct 31 12:35:02.957966 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:35:02.957948    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-smkms tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6dd76875b5-kbssv" podUID="f1cc3494-09f3-42ac-9f45-63a614597910"
Oct 31 12:35:02.981593 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:35:02.981566    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-2gcff"]
Oct 31 12:35:02.985076 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:35:02.985058    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-2gcff"
Oct 31 12:35:02.994885 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:35:02.994863    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-2gcff"]
Oct 31 12:35:03.012444 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:35:03.012417    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f1cc3494-09f3-42ac-9f45-63a614597910-tls-cert\") pod \"authorino-6dd76875b5-kbssv\" (UID: \"f1cc3494-09f3-42ac-9f45-63a614597910\") " pod="kuadrant-system/authorino-6dd76875b5-kbssv"
Oct 31 12:35:03.012599 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:35:03.012509    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smkms\" (UniqueName: \"kubernetes.io/projected/f1cc3494-09f3-42ac-9f45-63a614597910-kube-api-access-smkms\") pod \"authorino-6dd76875b5-kbssv\" (UID: \"f1cc3494-09f3-42ac-9f45-63a614597910\") " pod="kuadrant-system/authorino-6dd76875b5-kbssv"
Oct 31 12:35:03.113447 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:35:03.113413    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhpgw\" (UniqueName: \"kubernetes.io/projected/eb4e9a6f-6a1b-411a-a445-c77d44ac6d9d-kube-api-access-dhpgw\") pod \"authorino-6889cbd8c4-2gcff\" (UID: \"eb4e9a6f-6a1b-411a-a445-c77d44ac6d9d\") " pod="kuadrant-system/authorino-6889cbd8c4-2gcff"
Oct 31 12:35:03.113627 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:35:03.113453    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f1cc3494-09f3-42ac-9f45-63a614597910-tls-cert\") pod \"authorino-6dd76875b5-kbssv\" (UID: \"f1cc3494-09f3-42ac-9f45-63a614597910\") " pod="kuadrant-system/authorino-6dd76875b5-kbssv"
Oct 31 12:35:03.113688 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:35:03.113629    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/eb4e9a6f-6a1b-411a-a445-c77d44ac6d9d-tls-cert\") pod \"authorino-6889cbd8c4-2gcff\" (UID: \"eb4e9a6f-6a1b-411a-a445-c77d44ac6d9d\") " pod="kuadrant-system/authorino-6889cbd8c4-2gcff"
Oct 31 12:35:03.113777 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:35:03.113741    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smkms\" (UniqueName: \"kubernetes.io/projected/f1cc3494-09f3-42ac-9f45-63a614597910-kube-api-access-smkms\") pod \"authorino-6dd76875b5-kbssv\" (UID: \"f1cc3494-09f3-42ac-9f45-63a614597910\") " pod="kuadrant-system/authorino-6dd76875b5-kbssv"
Oct 31 12:35:03.116085 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:35:03.116061    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f1cc3494-09f3-42ac-9f45-63a614597910-tls-cert\") pod \"authorino-6dd76875b5-kbssv\" (UID: \"f1cc3494-09f3-42ac-9f45-63a614597910\") " pod="kuadrant-system/authorino-6dd76875b5-kbssv"
Oct 31 12:35:03.131188 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:35:03.131158    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-smkms\" (UniqueName: \"kubernetes.io/projected/f1cc3494-09f3-42ac-9f45-63a614597910-kube-api-access-smkms\") pod \"authorino-6dd76875b5-kbssv\" (UID: \"f1cc3494-09f3-42ac-9f45-63a614597910\") " pod="kuadrant-system/authorino-6dd76875b5-kbssv"
Oct 31 12:35:03.214356 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:35:03.214266    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/eb4e9a6f-6a1b-411a-a445-c77d44ac6d9d-tls-cert\") pod \"authorino-6889cbd8c4-2gcff\" (UID: \"eb4e9a6f-6a1b-411a-a445-c77d44ac6d9d\") " pod="kuadrant-system/authorino-6889cbd8c4-2gcff"
Oct 31 12:35:03.214356 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:35:03.214355    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhpgw\" (UniqueName: \"kubernetes.io/projected/eb4e9a6f-6a1b-411a-a445-c77d44ac6d9d-kube-api-access-dhpgw\") pod \"authorino-6889cbd8c4-2gcff\" (UID: \"eb4e9a6f-6a1b-411a-a445-c77d44ac6d9d\") " pod="kuadrant-system/authorino-6889cbd8c4-2gcff"
Oct 31 12:35:03.216785 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:35:03.216763    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/eb4e9a6f-6a1b-411a-a445-c77d44ac6d9d-tls-cert\") pod \"authorino-6889cbd8c4-2gcff\" (UID: \"eb4e9a6f-6a1b-411a-a445-c77d44ac6d9d\") " pod="kuadrant-system/authorino-6889cbd8c4-2gcff"
Oct 31 12:35:03.227367 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:35:03.227342    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhpgw\" (UniqueName: \"kubernetes.io/projected/eb4e9a6f-6a1b-411a-a445-c77d44ac6d9d-kube-api-access-dhpgw\") pod \"authorino-6889cbd8c4-2gcff\" (UID: \"eb4e9a6f-6a1b-411a-a445-c77d44ac6d9d\") " pod="kuadrant-system/authorino-6889cbd8c4-2gcff"
Oct 31 12:35:03.295007 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:35:03.294973    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-2gcff"]
Oct 31 12:35:03.295179 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:35:03.295168    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-2gcff"
Oct 31 12:35:03.325399 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:35:03.325366    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5b68ffb5fd-6s2ss"]
Oct 31 12:35:03.329082 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:35:03.329058    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5b68ffb5fd-6s2ss"
Oct 31 12:35:03.341625 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:35:03.341595    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5b68ffb5fd-6s2ss"]
Oct 31 12:35:03.416176 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:35:03.416141    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mh2tz\" (UniqueName: \"kubernetes.io/projected/bfc6317b-c496-4193-96fc-730f641a6b53-kube-api-access-mh2tz\") pod \"authorino-5b68ffb5fd-6s2ss\" (UID: \"bfc6317b-c496-4193-96fc-730f641a6b53\") " pod="kuadrant-system/authorino-5b68ffb5fd-6s2ss"
Oct 31 12:35:03.416347 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:35:03.416301    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/bfc6317b-c496-4193-96fc-730f641a6b53-tls-cert\") pod \"authorino-5b68ffb5fd-6s2ss\" (UID: \"bfc6317b-c496-4193-96fc-730f641a6b53\") " pod="kuadrant-system/authorino-5b68ffb5fd-6s2ss"
Oct 31 12:35:03.517779 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:35:03.517694    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/bfc6317b-c496-4193-96fc-730f641a6b53-tls-cert\") pod \"authorino-5b68ffb5fd-6s2ss\" (UID: \"bfc6317b-c496-4193-96fc-730f641a6b53\") " pod="kuadrant-system/authorino-5b68ffb5fd-6s2ss"
Oct 31 12:35:03.517779 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:35:03.517736    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mh2tz\" (UniqueName: \"kubernetes.io/projected/bfc6317b-c496-4193-96fc-730f641a6b53-kube-api-access-mh2tz\") pod \"authorino-5b68ffb5fd-6s2ss\" (UID: \"bfc6317b-c496-4193-96fc-730f641a6b53\") " pod="kuadrant-system/authorino-5b68ffb5fd-6s2ss"
Oct 31 12:35:03.520121 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:35:03.520091    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/bfc6317b-c496-4193-96fc-730f641a6b53-tls-cert\") pod \"authorino-5b68ffb5fd-6s2ss\" (UID: \"bfc6317b-c496-4193-96fc-730f641a6b53\") " pod="kuadrant-system/authorino-5b68ffb5fd-6s2ss"
Oct 31 12:35:03.530444 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:35:03.530420    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mh2tz\" (UniqueName: \"kubernetes.io/projected/bfc6317b-c496-4193-96fc-730f641a6b53-kube-api-access-mh2tz\") pod \"authorino-5b68ffb5fd-6s2ss\" (UID: \"bfc6317b-c496-4193-96fc-730f641a6b53\") " pod="kuadrant-system/authorino-5b68ffb5fd-6s2ss"
Oct 31 12:35:03.572932 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:35:03.572900    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-2gcff"]
Oct 31 12:35:03.576150 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:35:03.576124    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeb4e9a6f_6a1b_411a_a445_c77d44ac6d9d.slice/crio-dc0eb65c7502fed5e16a8772fb7018362dbf6fbf3cc06ba92223d8ca7296ee5e WatchSource:0}: Error finding container dc0eb65c7502fed5e16a8772fb7018362dbf6fbf3cc06ba92223d8ca7296ee5e: Status 404 returned error can't find the container with id dc0eb65c7502fed5e16a8772fb7018362dbf6fbf3cc06ba92223d8ca7296ee5e
Oct 31 12:35:03.642822 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:35:03.642792    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5b68ffb5fd-6s2ss"
Oct 31 12:35:03.825565 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:35:03.825528    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6889cbd8c4-2gcff" event={"ID":"eb4e9a6f-6a1b-411a-a445-c77d44ac6d9d","Type":"ContainerStarted","Data":"dc0eb65c7502fed5e16a8772fb7018362dbf6fbf3cc06ba92223d8ca7296ee5e"}
Oct 31 12:35:03.825736 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:35:03.825578    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6dd76875b5-kbssv"
Oct 31 12:35:03.832649 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:35:03.832628    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6dd76875b5-kbssv"
Oct 31 12:35:03.915534 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:35:03.915486    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5b68ffb5fd-6s2ss"]
Oct 31 12:35:03.920299 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:35:03.920256    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f1cc3494-09f3-42ac-9f45-63a614597910-tls-cert\" (UniqueName: \"kubernetes.io/secret/f1cc3494-09f3-42ac-9f45-63a614597910-tls-cert\") pod \"f1cc3494-09f3-42ac-9f45-63a614597910\" (UID: \"f1cc3494-09f3-42ac-9f45-63a614597910\") "
Oct 31 12:35:03.920414 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:35:03.920305    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1cc3494-09f3-42ac-9f45-63a614597910-kube-api-access-smkms\" (UniqueName: \"kubernetes.io/projected/f1cc3494-09f3-42ac-9f45-63a614597910-kube-api-access-smkms\") pod \"f1cc3494-09f3-42ac-9f45-63a614597910\" (UID: \"f1cc3494-09f3-42ac-9f45-63a614597910\") "
Oct 31 12:35:03.920477 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:35:03.920434    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbfc6317b_c496_4193_96fc_730f641a6b53.slice/crio-f7f75ca6ad8e67d5a8143c261ad566852f31f92984875783823a68b392649d95 WatchSource:0}: Error finding container f7f75ca6ad8e67d5a8143c261ad566852f31f92984875783823a68b392649d95: Status 404 returned error can't find the container with id f7f75ca6ad8e67d5a8143c261ad566852f31f92984875783823a68b392649d95
Oct 31 12:35:03.922597 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:35:03.922572    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1cc3494-09f3-42ac-9f45-63a614597910-tls-cert" pod "f1cc3494-09f3-42ac-9f45-63a614597910" (UID: "f1cc3494-09f3-42ac-9f45-63a614597910"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:35:03.922691 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:35:03.922662    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1cc3494-09f3-42ac-9f45-63a614597910-kube-api-access-smkms" pod "f1cc3494-09f3-42ac-9f45-63a614597910" (UID: "f1cc3494-09f3-42ac-9f45-63a614597910"). InnerVolumeSpecName "kube-api-access-smkms". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:35:04.021409 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:35:04.021373    5060 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f1cc3494-09f3-42ac-9f45-63a614597910-tls-cert\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:35:04.021409 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:35:04.021402    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smkms\" (UniqueName: \"kubernetes.io/projected/f1cc3494-09f3-42ac-9f45-63a614597910-kube-api-access-smkms\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:35:04.831856 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:35:04.831817    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5b68ffb5fd-6s2ss" event={"ID":"bfc6317b-c496-4193-96fc-730f641a6b53","Type":"ContainerStarted","Data":"f7f75ca6ad8e67d5a8143c261ad566852f31f92984875783823a68b392649d95"}
Oct 31 12:35:04.832031 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:35:04.831896    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6dd76875b5-kbssv"
Oct 31 12:35:04.867515 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:35:04.867303    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6dd76875b5-kbssv"]
Oct 31 12:35:04.869616 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:35:04.869591    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6dd76875b5-kbssv"]
Oct 31 12:35:05.057799 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:35:05.057760    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1cc3494-09f3-42ac-9f45-63a614597910" path="/var/lib/kubelet/pods/f1cc3494-09f3-42ac-9f45-63a614597910/volumes"
Oct 31 12:35:05.840076 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:35:05.840038    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6889cbd8c4-2gcff" event={"ID":"eb4e9a6f-6a1b-411a-a445-c77d44ac6d9d","Type":"ContainerStarted","Data":"3afdd90c12113009e3c064d59693604277b268bbac8724aa68f73510dc1189fe"}
Oct 31 12:35:05.840279 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:35:05.840088    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-6889cbd8c4-2gcff" podUID="eb4e9a6f-6a1b-411a-a445-c77d44ac6d9d" containerName="authorino" containerID="cri-o://3afdd90c12113009e3c064d59693604277b268bbac8724aa68f73510dc1189fe" gracePeriod=30
Oct 31 12:35:05.841836 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:35:05.841798    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5b68ffb5fd-6s2ss" event={"ID":"bfc6317b-c496-4193-96fc-730f641a6b53","Type":"ContainerStarted","Data":"6203b03c38abd072f086eb730ebb7e3a963464efd323a65d5095b0ae875f5197"}
Oct 31 12:35:05.858396 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:35:05.858347    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6889cbd8c4-2gcff" podStartSLOduration=2.477084429 podStartE2EDuration="3.858334357s" totalImagesPullingTime="1.381249928s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:35:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:35:05.857035494 +0000 UTC m=+468.960103208" watchObservedRunningTime="2026-10-31 12:35:05.858334357 +0000 UTC m=+468.961402069"
Oct 31 12:35:05.874397 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:35:05.874353    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-5b68ffb5fd-6s2ss" podStartSLOduration=1.910441959 podStartE2EDuration="2.87434042s" totalImagesPullingTime="963.898461ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:35:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:35:05.873772511 +0000 UTC m=+468.976840217" watchObservedRunningTime="2026-10-31 12:35:05.87434042 +0000 UTC m=+468.977408133"
Oct 31 12:35:05.905571 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:35:05.905537    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-c5hhm"]
Oct 31 12:35:05.905794 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:35:05.905766    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-c5hhm" podUID="79a9d422-eb6c-416c-a6a9-575fcb2299cb" containerName="authorino" containerID="cri-o://8a9e5d51a4df7e9be3e75400270f55c1f3b016c5872e8b18505ffa946720598e" gracePeriod=30
Oct 31 12:35:06.163649 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:35:06.163616    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-2gcff"
Oct 31 12:35:06.190792 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:35:06.190764    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-c5hhm"
Oct 31 12:35:06.243060 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:35:06.243026    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb4e9a6f-6a1b-411a-a445-c77d44ac6d9d-kube-api-access-dhpgw\" (UniqueName: \"kubernetes.io/projected/eb4e9a6f-6a1b-411a-a445-c77d44ac6d9d-kube-api-access-dhpgw\") pod \"eb4e9a6f-6a1b-411a-a445-c77d44ac6d9d\" (UID: \"eb4e9a6f-6a1b-411a-a445-c77d44ac6d9d\") "
Oct 31 12:35:06.243206 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:35:06.243065    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb4e9a6f-6a1b-411a-a445-c77d44ac6d9d-tls-cert\" (UniqueName: \"kubernetes.io/secret/eb4e9a6f-6a1b-411a-a445-c77d44ac6d9d-tls-cert\") pod \"eb4e9a6f-6a1b-411a-a445-c77d44ac6d9d\" (UID: \"eb4e9a6f-6a1b-411a-a445-c77d44ac6d9d\") "
Oct 31 12:35:06.245174 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:35:06.245144    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb4e9a6f-6a1b-411a-a445-c77d44ac6d9d-kube-api-access-dhpgw" pod "eb4e9a6f-6a1b-411a-a445-c77d44ac6d9d" (UID: "eb4e9a6f-6a1b-411a-a445-c77d44ac6d9d"). InnerVolumeSpecName "kube-api-access-dhpgw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:35:06.253344 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:35:06.253321    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb4e9a6f-6a1b-411a-a445-c77d44ac6d9d-tls-cert" pod "eb4e9a6f-6a1b-411a-a445-c77d44ac6d9d" (UID: "eb4e9a6f-6a1b-411a-a445-c77d44ac6d9d"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:35:06.344198 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:35:06.344172    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79a9d422-eb6c-416c-a6a9-575fcb2299cb-kube-api-access-9lqrk\" (UniqueName: \"kubernetes.io/projected/79a9d422-eb6c-416c-a6a9-575fcb2299cb-kube-api-access-9lqrk\") pod \"79a9d422-eb6c-416c-a6a9-575fcb2299cb\" (UID: \"79a9d422-eb6c-416c-a6a9-575fcb2299cb\") "
Oct 31 12:35:06.344334 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:35:06.344320    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhpgw\" (UniqueName: \"kubernetes.io/projected/eb4e9a6f-6a1b-411a-a445-c77d44ac6d9d-kube-api-access-dhpgw\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:35:06.344377 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:35:06.344336    5060 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/eb4e9a6f-6a1b-411a-a445-c77d44ac6d9d-tls-cert\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:35:06.345954 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:35:06.345927    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79a9d422-eb6c-416c-a6a9-575fcb2299cb-kube-api-access-9lqrk" pod "79a9d422-eb6c-416c-a6a9-575fcb2299cb" (UID: "79a9d422-eb6c-416c-a6a9-575fcb2299cb"). InnerVolumeSpecName "kube-api-access-9lqrk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:35:06.445296 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:35:06.445214    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9lqrk\" (UniqueName: \"kubernetes.io/projected/79a9d422-eb6c-416c-a6a9-575fcb2299cb-kube-api-access-9lqrk\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:35:06.848152 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:35:06.848114    5060 generic.go:356] "Generic (PLEG): container finished" podID="eb4e9a6f-6a1b-411a-a445-c77d44ac6d9d" containerID="3afdd90c12113009e3c064d59693604277b268bbac8724aa68f73510dc1189fe" exitCode=0
Oct 31 12:35:06.848329 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:35:06.848162    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-2gcff"
Oct 31 12:35:06.848329 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:35:06.848195    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6889cbd8c4-2gcff" event={"ID":"eb4e9a6f-6a1b-411a-a445-c77d44ac6d9d","Type":"ContainerDied","Data":"3afdd90c12113009e3c064d59693604277b268bbac8724aa68f73510dc1189fe"}
Oct 31 12:35:06.848329 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:35:06.848243    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6889cbd8c4-2gcff" event={"ID":"eb4e9a6f-6a1b-411a-a445-c77d44ac6d9d","Type":"ContainerDied","Data":"dc0eb65c7502fed5e16a8772fb7018362dbf6fbf3cc06ba92223d8ca7296ee5e"}
Oct 31 12:35:06.848329 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:35:06.848262    5060 scope.go:122] "RemoveContainer" containerID="3afdd90c12113009e3c064d59693604277b268bbac8724aa68f73510dc1189fe"
Oct 31 12:35:06.849600 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:35:06.849579    5060 generic.go:356] "Generic (PLEG): container finished" podID="79a9d422-eb6c-416c-a6a9-575fcb2299cb" containerID="8a9e5d51a4df7e9be3e75400270f55c1f3b016c5872e8b18505ffa946720598e" exitCode=0
Oct 31 12:35:06.849765 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:35:06.849611    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-c5hhm" event={"ID":"79a9d422-eb6c-416c-a6a9-575fcb2299cb","Type":"ContainerDied","Data":"8a9e5d51a4df7e9be3e75400270f55c1f3b016c5872e8b18505ffa946720598e"}
Oct 31 12:35:06.849765 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:35:06.849627    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-c5hhm"
Oct 31 12:35:06.849765 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:35:06.849650    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-c5hhm" event={"ID":"79a9d422-eb6c-416c-a6a9-575fcb2299cb","Type":"ContainerDied","Data":"2ff9abc07270047277a15bda4a816f36122304e05dfb87063971dd317134a90a"}
Oct 31 12:35:06.859338 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:35:06.859318    5060 scope.go:122] "RemoveContainer" containerID="3afdd90c12113009e3c064d59693604277b268bbac8724aa68f73510dc1189fe"
Oct 31 12:35:06.859669 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:35:06.859649    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3afdd90c12113009e3c064d59693604277b268bbac8724aa68f73510dc1189fe\": container with ID starting with 3afdd90c12113009e3c064d59693604277b268bbac8724aa68f73510dc1189fe not found: ID does not exist" containerID="3afdd90c12113009e3c064d59693604277b268bbac8724aa68f73510dc1189fe"
Oct 31 12:35:06.859745 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:35:06.859678    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3afdd90c12113009e3c064d59693604277b268bbac8724aa68f73510dc1189fe"} err="failed to get container status \"3afdd90c12113009e3c064d59693604277b268bbac8724aa68f73510dc1189fe\": rpc error: code = NotFound desc = could not find container \"3afdd90c12113009e3c064d59693604277b268bbac8724aa68f73510dc1189fe\": container with ID starting with 3afdd90c12113009e3c064d59693604277b268bbac8724aa68f73510dc1189fe not found: ID does not exist"
Oct 31 12:35:06.859745 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:35:06.859695    5060 scope.go:122] "RemoveContainer" containerID="8a9e5d51a4df7e9be3e75400270f55c1f3b016c5872e8b18505ffa946720598e"
Oct 31 12:35:06.867521 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:35:06.867486    5060 scope.go:122] "RemoveContainer" containerID="8a9e5d51a4df7e9be3e75400270f55c1f3b016c5872e8b18505ffa946720598e"
Oct 31 12:35:06.867753 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:35:06.867735    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a9e5d51a4df7e9be3e75400270f55c1f3b016c5872e8b18505ffa946720598e\": container with ID starting with 8a9e5d51a4df7e9be3e75400270f55c1f3b016c5872e8b18505ffa946720598e not found: ID does not exist" containerID="8a9e5d51a4df7e9be3e75400270f55c1f3b016c5872e8b18505ffa946720598e"
Oct 31 12:35:06.867791 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:35:06.867760    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a9e5d51a4df7e9be3e75400270f55c1f3b016c5872e8b18505ffa946720598e"} err="failed to get container status \"8a9e5d51a4df7e9be3e75400270f55c1f3b016c5872e8b18505ffa946720598e\": rpc error: code = NotFound desc = could not find container \"8a9e5d51a4df7e9be3e75400270f55c1f3b016c5872e8b18505ffa946720598e\": container with ID starting with 8a9e5d51a4df7e9be3e75400270f55c1f3b016c5872e8b18505ffa946720598e not found: ID does not exist"
Oct 31 12:35:06.873713 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:35:06.873690    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-2gcff"]
Oct 31 12:35:06.881021 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:35:06.880998    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-2gcff"]
Oct 31 12:35:06.893034 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:35:06.893012    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-c5hhm"]
Oct 31 12:35:06.898299 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:35:06.898278    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-c5hhm"]
Oct 31 12:35:07.046968 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:35:07.046940    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79a9d422-eb6c-416c-a6a9-575fcb2299cb" path="/var/lib/kubelet/pods/79a9d422-eb6c-416c-a6a9-575fcb2299cb/volumes"
Oct 31 12:35:07.047293 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:35:07.047281    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb4e9a6f-6a1b-411a-a445-c77d44ac6d9d" path="/var/lib/kubelet/pods/eb4e9a6f-6a1b-411a-a445-c77d44ac6d9d/volumes"
Oct 31 12:35:29.920017 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:35:29.919940    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7phb" containerName="extractor"
Oct 31 12:35:57.581424 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:35:57.581389    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-rbn8q"]
Oct 31 12:35:57.581907 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:35:57.581784    5060 cpu_manager.go:414] "containerMap: removing container" podUID="79a9d422-eb6c-416c-a6a9-575fcb2299cb" containerName="authorino"
Oct 31 12:35:57.581907 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:35:57.581796    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79a9d422-eb6c-416c-a6a9-575fcb2299cb" containerName="authorino"
Oct 31 12:35:57.581907 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:35:57.581810    5060 cpu_manager.go:414] "containerMap: removing container" podUID="eb4e9a6f-6a1b-411a-a445-c77d44ac6d9d" containerName="authorino"
Oct 31 12:35:57.581907 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:35:57.581815    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb4e9a6f-6a1b-411a-a445-c77d44ac6d9d" containerName="authorino"
Oct 31 12:35:57.581907 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:35:57.581884    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="79a9d422-eb6c-416c-a6a9-575fcb2299cb" containerName="authorino"
Oct 31 12:35:57.581907 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:35:57.581892    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb4e9a6f-6a1b-411a-a445-c77d44ac6d9d" containerName="authorino"
Oct 31 12:35:57.585216 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:35:57.585196    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-rbn8q"
Oct 31 12:35:57.589722 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:35:57.589699    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Oct 31 12:35:57.589722 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:35:57.589712    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Oct 31 12:35:57.603977 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:35:57.603954    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-rbn8q"]
Oct 31 12:35:57.647161 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:35:57.647131    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-2xp25\""
Oct 31 12:35:57.647295 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:35:57.647169    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"echo-simulator-kserve-self-signed-certs\""
Oct 31 12:35:57.682979 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:35:57.682953    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dea929f7-61c3-4921-9bdb-106f763e067e-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-rbn8q\" (UID: \"dea929f7-61c3-4921-9bdb-106f763e067e\") " pod="llm/echo-simulator-kserve-64b6f8f96c-rbn8q"
Oct 31 12:35:57.683122 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:35:57.682987    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6twxs\" (UniqueName: \"kubernetes.io/projected/dea929f7-61c3-4921-9bdb-106f763e067e-kube-api-access-6twxs\") pod \"echo-simulator-kserve-64b6f8f96c-rbn8q\" (UID: \"dea929f7-61c3-4921-9bdb-106f763e067e\") " pod="llm/echo-simulator-kserve-64b6f8f96c-rbn8q"
Oct 31 12:35:57.683122 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:35:57.683020    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dea929f7-61c3-4921-9bdb-106f763e067e-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-rbn8q\" (UID: \"dea929f7-61c3-4921-9bdb-106f763e067e\") " pod="llm/echo-simulator-kserve-64b6f8f96c-rbn8q"
Oct 31 12:35:57.683122 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:35:57.683068    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dea929f7-61c3-4921-9bdb-106f763e067e-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-rbn8q\" (UID: \"dea929f7-61c3-4921-9bdb-106f763e067e\") " pod="llm/echo-simulator-kserve-64b6f8f96c-rbn8q"
Oct 31 12:35:57.683122 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:35:57.683112    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dea929f7-61c3-4921-9bdb-106f763e067e-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-rbn8q\" (UID: \"dea929f7-61c3-4921-9bdb-106f763e067e\") " pod="llm/echo-simulator-kserve-64b6f8f96c-rbn8q"
Oct 31 12:35:57.683275 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:35:57.683133    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dea929f7-61c3-4921-9bdb-106f763e067e-home\") pod \"echo-simulator-kserve-64b6f8f96c-rbn8q\" (UID: \"dea929f7-61c3-4921-9bdb-106f763e067e\") " pod="llm/echo-simulator-kserve-64b6f8f96c-rbn8q"
Oct 31 12:35:57.783778 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:35:57.783749    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dea929f7-61c3-4921-9bdb-106f763e067e-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-rbn8q\" (UID: \"dea929f7-61c3-4921-9bdb-106f763e067e\") " pod="llm/echo-simulator-kserve-64b6f8f96c-rbn8q"
Oct 31 12:35:57.783952 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:35:57.783785    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dea929f7-61c3-4921-9bdb-106f763e067e-home\") pod \"echo-simulator-kserve-64b6f8f96c-rbn8q\" (UID: \"dea929f7-61c3-4921-9bdb-106f763e067e\") " pod="llm/echo-simulator-kserve-64b6f8f96c-rbn8q"
Oct 31 12:35:57.783952 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:35:57.783830    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dea929f7-61c3-4921-9bdb-106f763e067e-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-rbn8q\" (UID: \"dea929f7-61c3-4921-9bdb-106f763e067e\") " pod="llm/echo-simulator-kserve-64b6f8f96c-rbn8q"
Oct 31 12:35:57.783952 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:35:57.783848    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6twxs\" (UniqueName: \"kubernetes.io/projected/dea929f7-61c3-4921-9bdb-106f763e067e-kube-api-access-6twxs\") pod \"echo-simulator-kserve-64b6f8f96c-rbn8q\" (UID: \"dea929f7-61c3-4921-9bdb-106f763e067e\") " pod="llm/echo-simulator-kserve-64b6f8f96c-rbn8q"
Oct 31 12:35:57.783952 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:35:57.783879    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dea929f7-61c3-4921-9bdb-106f763e067e-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-rbn8q\" (UID: \"dea929f7-61c3-4921-9bdb-106f763e067e\") " pod="llm/echo-simulator-kserve-64b6f8f96c-rbn8q"
Oct 31 12:35:57.783952 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:35:57.783905    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dea929f7-61c3-4921-9bdb-106f763e067e-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-rbn8q\" (UID: \"dea929f7-61c3-4921-9bdb-106f763e067e\") " pod="llm/echo-simulator-kserve-64b6f8f96c-rbn8q"
Oct 31 12:35:57.784232 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:35:57.784179    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dea929f7-61c3-4921-9bdb-106f763e067e-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-rbn8q\" (UID: \"dea929f7-61c3-4921-9bdb-106f763e067e\") " pod="llm/echo-simulator-kserve-64b6f8f96c-rbn8q"
Oct 31 12:35:57.784294 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:35:57.784251    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dea929f7-61c3-4921-9bdb-106f763e067e-home\") pod \"echo-simulator-kserve-64b6f8f96c-rbn8q\" (UID: \"dea929f7-61c3-4921-9bdb-106f763e067e\") " pod="llm/echo-simulator-kserve-64b6f8f96c-rbn8q"
Oct 31 12:35:57.784294 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:35:57.784263    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dea929f7-61c3-4921-9bdb-106f763e067e-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-rbn8q\" (UID: \"dea929f7-61c3-4921-9bdb-106f763e067e\") " pod="llm/echo-simulator-kserve-64b6f8f96c-rbn8q"
Oct 31 12:35:57.786117 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:35:57.786091    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dea929f7-61c3-4921-9bdb-106f763e067e-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-rbn8q\" (UID: \"dea929f7-61c3-4921-9bdb-106f763e067e\") " pod="llm/echo-simulator-kserve-64b6f8f96c-rbn8q"
Oct 31 12:35:57.786533 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:35:57.786515    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dea929f7-61c3-4921-9bdb-106f763e067e-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-rbn8q\" (UID: \"dea929f7-61c3-4921-9bdb-106f763e067e\") " pod="llm/echo-simulator-kserve-64b6f8f96c-rbn8q"
Oct 31 12:35:57.798365 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:35:57.798338    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6twxs\" (UniqueName: \"kubernetes.io/projected/dea929f7-61c3-4921-9bdb-106f763e067e-kube-api-access-6twxs\") pod \"echo-simulator-kserve-64b6f8f96c-rbn8q\" (UID: \"dea929f7-61c3-4921-9bdb-106f763e067e\") " pod="llm/echo-simulator-kserve-64b6f8f96c-rbn8q"
Oct 31 12:35:57.897722 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:35:57.897644    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-rbn8q"
Oct 31 12:35:58.178262 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:35:58.178185    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-rbn8q"]
Oct 31 12:35:59.130133 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:35:59.130093    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-rbn8q" event={"ID":"dea929f7-61c3-4921-9bdb-106f763e067e","Type":"ContainerStarted","Data":"ca229d5bbc1d4846850bd3b2380afbc5b9e00a6c67169157f6ab0ac257c3bd6f"}
Oct 31 12:35:59.927459 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:35:59.927424    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7phb" containerName="extractor"
Oct 31 12:36:00.167169 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:00.167063    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-rbn8q" event={"ID":"dea929f7-61c3-4921-9bdb-106f763e067e","Type":"ContainerStarted","Data":"f6f02a07a447e3fdd7c8476382c7c77fdc8dee8ab3e98bbd9bc2b792cf409c12"}
Oct 31 12:36:00.190366 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:00.190308    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/echo-simulator-kserve-64b6f8f96c-rbn8q" podStartSLOduration=1.466422279 podStartE2EDuration="3.190295868s" totalImagesPullingTime="1.723873589s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:35:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:36:00.187927477 +0000 UTC m=+523.290995189" watchObservedRunningTime="2026-10-31 12:36:00.190295868 +0000 UTC m=+523.293363580"
Oct 31 12:36:00.898202 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:00.898167    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-rbn8q"
Oct 31 12:36:01.033546 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:01.033510    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x2qzv"]
Oct 31 12:36:01.037309 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:01.037286    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x2qzv"
Oct 31 12:36:01.042225 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:01.042200    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Oct 31 12:36:01.051886 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:01.051860    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x2qzv"]
Oct 31 12:36:01.122365 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:01.122330    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c247ed2a-aa90-424b-9b7e-4f2936d67e3f-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-x2qzv\" (UID: \"c247ed2a-aa90-424b-9b7e-4f2936d67e3f\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x2qzv"
Oct 31 12:36:01.122575 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:01.122392    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcmlp\" (UniqueName: \"kubernetes.io/projected/c247ed2a-aa90-424b-9b7e-4f2936d67e3f-kube-api-access-kcmlp\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-x2qzv\" (UID: \"c247ed2a-aa90-424b-9b7e-4f2936d67e3f\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x2qzv"
Oct 31 12:36:01.122575 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:01.122465    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c247ed2a-aa90-424b-9b7e-4f2936d67e3f-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-x2qzv\" (UID: \"c247ed2a-aa90-424b-9b7e-4f2936d67e3f\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x2qzv"
Oct 31 12:36:01.122575 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:01.122527    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c247ed2a-aa90-424b-9b7e-4f2936d67e3f-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-x2qzv\" (UID: \"c247ed2a-aa90-424b-9b7e-4f2936d67e3f\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x2qzv"
Oct 31 12:36:01.122764 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:01.122663    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c247ed2a-aa90-424b-9b7e-4f2936d67e3f-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-x2qzv\" (UID: \"c247ed2a-aa90-424b-9b7e-4f2936d67e3f\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x2qzv"
Oct 31 12:36:01.122764 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:01.122704    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c247ed2a-aa90-424b-9b7e-4f2936d67e3f-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-x2qzv\" (UID: \"c247ed2a-aa90-424b-9b7e-4f2936d67e3f\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x2qzv"
Oct 31 12:36:01.223622 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:01.223537    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c247ed2a-aa90-424b-9b7e-4f2936d67e3f-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-x2qzv\" (UID: \"c247ed2a-aa90-424b-9b7e-4f2936d67e3f\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x2qzv"
Oct 31 12:36:01.223622 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:01.223597    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c247ed2a-aa90-424b-9b7e-4f2936d67e3f-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-x2qzv\" (UID: \"c247ed2a-aa90-424b-9b7e-4f2936d67e3f\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x2qzv"
Oct 31 12:36:01.224120 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:01.223674    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c247ed2a-aa90-424b-9b7e-4f2936d67e3f-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-x2qzv\" (UID: \"c247ed2a-aa90-424b-9b7e-4f2936d67e3f\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x2qzv"
Oct 31 12:36:01.224120 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:01.223784    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcmlp\" (UniqueName: \"kubernetes.io/projected/c247ed2a-aa90-424b-9b7e-4f2936d67e3f-kube-api-access-kcmlp\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-x2qzv\" (UID: \"c247ed2a-aa90-424b-9b7e-4f2936d67e3f\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x2qzv"
Oct 31 12:36:01.224120 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:01.223849    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c247ed2a-aa90-424b-9b7e-4f2936d67e3f-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-x2qzv\" (UID: \"c247ed2a-aa90-424b-9b7e-4f2936d67e3f\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x2qzv"
Oct 31 12:36:01.224120 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:01.223904    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c247ed2a-aa90-424b-9b7e-4f2936d67e3f-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-x2qzv\" (UID: \"c247ed2a-aa90-424b-9b7e-4f2936d67e3f\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x2qzv"
Oct 31 12:36:01.224120 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:01.223916    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c247ed2a-aa90-424b-9b7e-4f2936d67e3f-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-x2qzv\" (UID: \"c247ed2a-aa90-424b-9b7e-4f2936d67e3f\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x2qzv"
Oct 31 12:36:01.224120 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:01.223972    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c247ed2a-aa90-424b-9b7e-4f2936d67e3f-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-x2qzv\" (UID: \"c247ed2a-aa90-424b-9b7e-4f2936d67e3f\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x2qzv"
Oct 31 12:36:01.224414 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:01.224383    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c247ed2a-aa90-424b-9b7e-4f2936d67e3f-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-x2qzv\" (UID: \"c247ed2a-aa90-424b-9b7e-4f2936d67e3f\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x2qzv"
Oct 31 12:36:01.226101 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:01.226077    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c247ed2a-aa90-424b-9b7e-4f2936d67e3f-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-x2qzv\" (UID: \"c247ed2a-aa90-424b-9b7e-4f2936d67e3f\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x2qzv"
Oct 31 12:36:01.226475 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:01.226452    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c247ed2a-aa90-424b-9b7e-4f2936d67e3f-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-x2qzv\" (UID: \"c247ed2a-aa90-424b-9b7e-4f2936d67e3f\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x2qzv"
Oct 31 12:36:01.241303 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:01.241279    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcmlp\" (UniqueName: \"kubernetes.io/projected/c247ed2a-aa90-424b-9b7e-4f2936d67e3f-kube-api-access-kcmlp\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-x2qzv\" (UID: \"c247ed2a-aa90-424b-9b7e-4f2936d67e3f\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x2qzv"
Oct 31 12:36:01.350651 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:01.350601    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x2qzv"
Oct 31 12:36:01.629520 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:01.629481    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x2qzv"]
Oct 31 12:36:01.631597 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:36:01.631556    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc247ed2a_aa90_424b_9b7e_4f2936d67e3f.slice/crio-d7055289c49583fd1d8b37c2c18cb38ad44ad019b050e22ccab9ac17eab5063c WatchSource:0}: Error finding container d7055289c49583fd1d8b37c2c18cb38ad44ad019b050e22ccab9ac17eab5063c: Status 404 returned error can't find the container with id d7055289c49583fd1d8b37c2c18cb38ad44ad019b050e22ccab9ac17eab5063c
Oct 31 12:36:02.183281 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:02.183239    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x2qzv" event={"ID":"c247ed2a-aa90-424b-9b7e-4f2936d67e3f","Type":"ContainerStarted","Data":"d7055289c49583fd1d8b37c2c18cb38ad44ad019b050e22ccab9ac17eab5063c"}
Oct 31 12:36:03.191237 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:03.191144    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x2qzv" event={"ID":"c247ed2a-aa90-424b-9b7e-4f2936d67e3f","Type":"ContainerStarted","Data":"fe18fc50fc45dfebd8aa81625402db0c8b0ecba462faff98eeebad879532eff0"}
Oct 31 12:36:03.211862 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:03.211809    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x2qzv" podStartSLOduration=0.91709825 podStartE2EDuration="2.211796465s" totalImagesPullingTime="1.294698215s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:36:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:36:03.211440352 +0000 UTC m=+526.314508066" watchObservedRunningTime="2026-10-31 12:36:03.211796465 +0000 UTC m=+526.314864177"
Oct 31 12:36:03.351017 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:03.350975    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x2qzv"
Oct 31 12:36:07.490578 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:07.490540    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-vmzbr"]
Oct 31 12:36:07.494992 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:07.494966    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vmzbr"
Oct 31 12:36:07.499795 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:07.499766    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Oct 31 12:36:07.507043 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:07.507018    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-vmzbr"]
Oct 31 12:36:07.594438 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:07.594401    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7049f597-e139-4f0e-afbd-ec6ea0effab9-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-vmzbr\" (UID: \"7049f597-e139-4f0e-afbd-ec6ea0effab9\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vmzbr"
Oct 31 12:36:07.594637 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:07.594452    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7049f597-e139-4f0e-afbd-ec6ea0effab9-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-vmzbr\" (UID: \"7049f597-e139-4f0e-afbd-ec6ea0effab9\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vmzbr"
Oct 31 12:36:07.594637 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:07.594469    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7049f597-e139-4f0e-afbd-ec6ea0effab9-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-vmzbr\" (UID: \"7049f597-e139-4f0e-afbd-ec6ea0effab9\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vmzbr"
Oct 31 12:36:07.594637 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:07.594598    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7049f597-e139-4f0e-afbd-ec6ea0effab9-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-vmzbr\" (UID: \"7049f597-e139-4f0e-afbd-ec6ea0effab9\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vmzbr"
Oct 31 12:36:07.594794 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:07.594651    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7049f597-e139-4f0e-afbd-ec6ea0effab9-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-vmzbr\" (UID: \"7049f597-e139-4f0e-afbd-ec6ea0effab9\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vmzbr"
Oct 31 12:36:07.594794 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:07.594668    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chctm\" (UniqueName: \"kubernetes.io/projected/7049f597-e139-4f0e-afbd-ec6ea0effab9-kube-api-access-chctm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-vmzbr\" (UID: \"7049f597-e139-4f0e-afbd-ec6ea0effab9\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vmzbr"
Oct 31 12:36:07.696143 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:07.696087    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7049f597-e139-4f0e-afbd-ec6ea0effab9-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-vmzbr\" (UID: \"7049f597-e139-4f0e-afbd-ec6ea0effab9\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vmzbr"
Oct 31 12:36:07.696143 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:07.696152    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chctm\" (UniqueName: \"kubernetes.io/projected/7049f597-e139-4f0e-afbd-ec6ea0effab9-kube-api-access-chctm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-vmzbr\" (UID: \"7049f597-e139-4f0e-afbd-ec6ea0effab9\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vmzbr"
Oct 31 12:36:07.696403 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:07.696214    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7049f597-e139-4f0e-afbd-ec6ea0effab9-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-vmzbr\" (UID: \"7049f597-e139-4f0e-afbd-ec6ea0effab9\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vmzbr"
Oct 31 12:36:07.696403 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:07.696266    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7049f597-e139-4f0e-afbd-ec6ea0effab9-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-vmzbr\" (UID: \"7049f597-e139-4f0e-afbd-ec6ea0effab9\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vmzbr"
Oct 31 12:36:07.696403 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:07.696290    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7049f597-e139-4f0e-afbd-ec6ea0effab9-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-vmzbr\" (UID: \"7049f597-e139-4f0e-afbd-ec6ea0effab9\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vmzbr"
Oct 31 12:36:07.696403 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:07.696341    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7049f597-e139-4f0e-afbd-ec6ea0effab9-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-vmzbr\" (UID: \"7049f597-e139-4f0e-afbd-ec6ea0effab9\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vmzbr"
Oct 31 12:36:07.696659 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:07.696638    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7049f597-e139-4f0e-afbd-ec6ea0effab9-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-vmzbr\" (UID: \"7049f597-e139-4f0e-afbd-ec6ea0effab9\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vmzbr"
Oct 31 12:36:07.696707 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:07.696653    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7049f597-e139-4f0e-afbd-ec6ea0effab9-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-vmzbr\" (UID: \"7049f597-e139-4f0e-afbd-ec6ea0effab9\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vmzbr"
Oct 31 12:36:07.696760 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:07.696740    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7049f597-e139-4f0e-afbd-ec6ea0effab9-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-vmzbr\" (UID: \"7049f597-e139-4f0e-afbd-ec6ea0effab9\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vmzbr"
Oct 31 12:36:07.698622 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:07.698598    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7049f597-e139-4f0e-afbd-ec6ea0effab9-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-vmzbr\" (UID: \"7049f597-e139-4f0e-afbd-ec6ea0effab9\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vmzbr"
Oct 31 12:36:07.698835 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:07.698807    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7049f597-e139-4f0e-afbd-ec6ea0effab9-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-vmzbr\" (UID: \"7049f597-e139-4f0e-afbd-ec6ea0effab9\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vmzbr"
Oct 31 12:36:07.708608 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:07.708586    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chctm\" (UniqueName: \"kubernetes.io/projected/7049f597-e139-4f0e-afbd-ec6ea0effab9-kube-api-access-chctm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-vmzbr\" (UID: \"7049f597-e139-4f0e-afbd-ec6ea0effab9\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vmzbr"
Oct 31 12:36:07.812617 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:07.812574    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vmzbr"
Oct 31 12:36:07.898222 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:07.897996    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/echo-simulator-kserve-64b6f8f96c-rbn8q"
Oct 31 12:36:07.912445 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:07.912417    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/echo-simulator-kserve-64b6f8f96c-rbn8q"
Oct 31 12:36:08.092545 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:08.092519    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-vmzbr"]
Oct 31 12:36:08.221954 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:08.221914    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vmzbr" event={"ID":"7049f597-e139-4f0e-afbd-ec6ea0effab9","Type":"ContainerStarted","Data":"981fbae94898e6b54cadaf3657965172894a5db87b27563b6e044f3f7df48a21"}
Oct 31 12:36:08.233090 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:08.233067    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-rbn8q"
Oct 31 12:36:09.229680 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:09.229640    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vmzbr" event={"ID":"7049f597-e139-4f0e-afbd-ec6ea0effab9","Type":"ContainerStarted","Data":"7cf3b364831673f91d8baf9e8bc3e681923ae87f0c8499c8d126e1c6d2b9350c"}
Oct 31 12:36:09.254052 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:09.254002    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vmzbr" podStartSLOduration=1.971629944 podStartE2EDuration="2.253987438s" totalImagesPullingTime="282.357494ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:36:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:36:09.252356534 +0000 UTC m=+532.355424244" watchObservedRunningTime="2026-10-31 12:36:09.253987438 +0000 UTC m=+532.357055150"
Oct 31 12:36:09.537740 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:09.537706    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-rm7cl"]
Oct 31 12:36:09.541431 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:09.541399    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-rm7cl"
Oct 31 12:36:09.554326 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:09.554303    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Oct 31 12:36:09.562861 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:09.562834    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-rm7cl"]
Oct 31 12:36:09.611955 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:09.611919    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8292eaa8-0a02-46be-8654-3c4136ff604f-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-rm7cl\" (UID: \"8292eaa8-0a02-46be-8654-3c4136ff604f\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-rm7cl"
Oct 31 12:36:09.611955 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:09.611962    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8292eaa8-0a02-46be-8654-3c4136ff604f-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-rm7cl\" (UID: \"8292eaa8-0a02-46be-8654-3c4136ff604f\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-rm7cl"
Oct 31 12:36:09.612174 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:09.612073    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8292eaa8-0a02-46be-8654-3c4136ff604f-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-rm7cl\" (UID: \"8292eaa8-0a02-46be-8654-3c4136ff604f\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-rm7cl"
Oct 31 12:36:09.612174 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:09.612101    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8292eaa8-0a02-46be-8654-3c4136ff604f-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-rm7cl\" (UID: \"8292eaa8-0a02-46be-8654-3c4136ff604f\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-rm7cl"
Oct 31 12:36:09.612174 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:09.612120    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22956\" (UniqueName: \"kubernetes.io/projected/8292eaa8-0a02-46be-8654-3c4136ff604f-kube-api-access-22956\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-rm7cl\" (UID: \"8292eaa8-0a02-46be-8654-3c4136ff604f\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-rm7cl"
Oct 31 12:36:09.612282 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:09.612252    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8292eaa8-0a02-46be-8654-3c4136ff604f-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-rm7cl\" (UID: \"8292eaa8-0a02-46be-8654-3c4136ff604f\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-rm7cl"
Oct 31 12:36:09.713342 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:09.713301    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8292eaa8-0a02-46be-8654-3c4136ff604f-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-rm7cl\" (UID: \"8292eaa8-0a02-46be-8654-3c4136ff604f\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-rm7cl"
Oct 31 12:36:09.713342 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:09.713346    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8292eaa8-0a02-46be-8654-3c4136ff604f-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-rm7cl\" (UID: \"8292eaa8-0a02-46be-8654-3c4136ff604f\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-rm7cl"
Oct 31 12:36:09.713640 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:09.713394    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8292eaa8-0a02-46be-8654-3c4136ff604f-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-rm7cl\" (UID: \"8292eaa8-0a02-46be-8654-3c4136ff604f\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-rm7cl"
Oct 31 12:36:09.713640 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:09.713415    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8292eaa8-0a02-46be-8654-3c4136ff604f-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-rm7cl\" (UID: \"8292eaa8-0a02-46be-8654-3c4136ff604f\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-rm7cl"
Oct 31 12:36:09.713640 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:09.713441    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22956\" (UniqueName: \"kubernetes.io/projected/8292eaa8-0a02-46be-8654-3c4136ff604f-kube-api-access-22956\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-rm7cl\" (UID: \"8292eaa8-0a02-46be-8654-3c4136ff604f\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-rm7cl"
Oct 31 12:36:09.713640 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:09.713524    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8292eaa8-0a02-46be-8654-3c4136ff604f-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-rm7cl\" (UID: \"8292eaa8-0a02-46be-8654-3c4136ff604f\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-rm7cl"
Oct 31 12:36:09.713970 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:09.713943    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8292eaa8-0a02-46be-8654-3c4136ff604f-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-rm7cl\" (UID: \"8292eaa8-0a02-46be-8654-3c4136ff604f\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-rm7cl"
Oct 31 12:36:09.714270 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:09.714243    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8292eaa8-0a02-46be-8654-3c4136ff604f-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-rm7cl\" (UID: \"8292eaa8-0a02-46be-8654-3c4136ff604f\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-rm7cl"
Oct 31 12:36:09.714387 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:09.714371    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8292eaa8-0a02-46be-8654-3c4136ff604f-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-rm7cl\" (UID: \"8292eaa8-0a02-46be-8654-3c4136ff604f\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-rm7cl"
Oct 31 12:36:09.716446 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:09.716418    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8292eaa8-0a02-46be-8654-3c4136ff604f-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-rm7cl\" (UID: \"8292eaa8-0a02-46be-8654-3c4136ff604f\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-rm7cl"
Oct 31 12:36:09.717238 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:09.717216    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8292eaa8-0a02-46be-8654-3c4136ff604f-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-rm7cl\" (UID: \"8292eaa8-0a02-46be-8654-3c4136ff604f\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-rm7cl"
Oct 31 12:36:09.726533 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:09.726462    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22956\" (UniqueName: \"kubernetes.io/projected/8292eaa8-0a02-46be-8654-3c4136ff604f-kube-api-access-22956\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-rm7cl\" (UID: \"8292eaa8-0a02-46be-8654-3c4136ff604f\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-rm7cl"
Oct 31 12:36:09.813118 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:09.813040    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vmzbr"
Oct 31 12:36:09.856376 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:09.856339    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-rm7cl"
Oct 31 12:36:10.170226 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:10.170196    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-rm7cl"]
Oct 31 12:36:10.173535 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:36:10.173477    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8292eaa8_0a02_46be_8654_3c4136ff604f.slice/crio-6802f9b0faabdddca9f8357d57f1079dee5871930ae68d53ebe70c477b42dc15 WatchSource:0}: Error finding container 6802f9b0faabdddca9f8357d57f1079dee5871930ae68d53ebe70c477b42dc15: Status 404 returned error can't find the container with id 6802f9b0faabdddca9f8357d57f1079dee5871930ae68d53ebe70c477b42dc15
Oct 31 12:36:10.236844 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:10.236802    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-rm7cl" event={"ID":"8292eaa8-0a02-46be-8654-3c4136ff604f","Type":"ContainerStarted","Data":"6802f9b0faabdddca9f8357d57f1079dee5871930ae68d53ebe70c477b42dc15"}
Oct 31 12:36:11.244246 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:11.244205    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-rm7cl" event={"ID":"8292eaa8-0a02-46be-8654-3c4136ff604f","Type":"ContainerStarted","Data":"915049c8930cc2b94f40a2deda3c5fd336e360b4c358f140e17c7db42e1b4f05"}
Oct 31 12:36:11.264629 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:11.264576    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-rm7cl" podStartSLOduration=1.984395148 podStartE2EDuration="2.264561887s" totalImagesPullingTime="280.166739ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:36:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:36:11.264011942 +0000 UTC m=+534.367079654" watchObservedRunningTime="2026-10-31 12:36:11.264561887 +0000 UTC m=+534.367629598"
Oct 31 12:36:11.351289 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:11.351250    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x2qzv"
Oct 31 12:36:11.364411 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:11.364387    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x2qzv"
Oct 31 12:36:11.856774 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:11.856736    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-rm7cl"
Oct 31 12:36:12.261686 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:12.261608    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x2qzv"
Oct 31 12:36:17.813273 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:17.813238    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vmzbr"
Oct 31 12:36:17.825893 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:17.825867    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vmzbr"
Oct 31 12:36:18.298151 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:18.298120    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vmzbr"
Oct 31 12:36:19.856914 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:19.856869    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-rm7cl"
Oct 31 12:36:19.869444 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:19.869418    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-rm7cl"
Oct 31 12:36:20.310208 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:20.310181    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-rm7cl"
Oct 31 12:36:22.878628 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:22.878585    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5ccbfdc857-zp7zh"]
Oct 31 12:36:22.882609 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:22.882591    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5ccbfdc857-zp7zh"
Oct 31 12:36:22.891719 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:22.891692    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5ccbfdc857-zp7zh"]
Oct 31 12:36:22.940030 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:22.939991    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sccfm\" (UniqueName: \"kubernetes.io/projected/9a585b0c-1708-4bb9-8896-3e10e29824d3-kube-api-access-sccfm\") pod \"authorino-5ccbfdc857-zp7zh\" (UID: \"9a585b0c-1708-4bb9-8896-3e10e29824d3\") " pod="kuadrant-system/authorino-5ccbfdc857-zp7zh"
Oct 31 12:36:22.940174 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:22.940107    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9a585b0c-1708-4bb9-8896-3e10e29824d3-tls-cert\") pod \"authorino-5ccbfdc857-zp7zh\" (UID: \"9a585b0c-1708-4bb9-8896-3e10e29824d3\") " pod="kuadrant-system/authorino-5ccbfdc857-zp7zh"
Oct 31 12:36:23.041256 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:23.041217    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9a585b0c-1708-4bb9-8896-3e10e29824d3-tls-cert\") pod \"authorino-5ccbfdc857-zp7zh\" (UID: \"9a585b0c-1708-4bb9-8896-3e10e29824d3\") " pod="kuadrant-system/authorino-5ccbfdc857-zp7zh"
Oct 31 12:36:23.041416 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:23.041290    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sccfm\" (UniqueName: \"kubernetes.io/projected/9a585b0c-1708-4bb9-8896-3e10e29824d3-kube-api-access-sccfm\") pod \"authorino-5ccbfdc857-zp7zh\" (UID: \"9a585b0c-1708-4bb9-8896-3e10e29824d3\") " pod="kuadrant-system/authorino-5ccbfdc857-zp7zh"
Oct 31 12:36:23.043668 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:23.043646    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9a585b0c-1708-4bb9-8896-3e10e29824d3-tls-cert\") pod \"authorino-5ccbfdc857-zp7zh\" (UID: \"9a585b0c-1708-4bb9-8896-3e10e29824d3\") " pod="kuadrant-system/authorino-5ccbfdc857-zp7zh"
Oct 31 12:36:23.053839 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:23.053817    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sccfm\" (UniqueName: \"kubernetes.io/projected/9a585b0c-1708-4bb9-8896-3e10e29824d3-kube-api-access-sccfm\") pod \"authorino-5ccbfdc857-zp7zh\" (UID: \"9a585b0c-1708-4bb9-8896-3e10e29824d3\") " pod="kuadrant-system/authorino-5ccbfdc857-zp7zh"
Oct 31 12:36:23.196678 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:23.196599    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5ccbfdc857-zp7zh"
Oct 31 12:36:23.473637 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:23.473605    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5ccbfdc857-zp7zh"]
Oct 31 12:36:23.475408 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:36:23.475375    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9a585b0c_1708_4bb9_8896_3e10e29824d3.slice/crio-0d668606cf64d33811a8cccab12c5f1e6f547499517067fff510ff97c780fc28 WatchSource:0}: Error finding container 0d668606cf64d33811a8cccab12c5f1e6f547499517067fff510ff97c780fc28: Status 404 returned error can't find the container with id 0d668606cf64d33811a8cccab12c5f1e6f547499517067fff510ff97c780fc28
Oct 31 12:36:24.322027 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:24.321984    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5ccbfdc857-zp7zh" event={"ID":"9a585b0c-1708-4bb9-8896-3e10e29824d3","Type":"ContainerStarted","Data":"0d668606cf64d33811a8cccab12c5f1e6f547499517067fff510ff97c780fc28"}
Oct 31 12:36:25.333152 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:25.333109    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5ccbfdc857-zp7zh" event={"ID":"9a585b0c-1708-4bb9-8896-3e10e29824d3","Type":"ContainerStarted","Data":"225f6cbeb82cb26ce26172990f41de74292c92181bbb3e13ded1f4ec58530c90"}
Oct 31 12:36:25.355194 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:25.352564    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-5ccbfdc857-zp7zh" podStartSLOduration=2.434032928 podStartE2EDuration="3.352542687s" totalImagesPullingTime="918.509759ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:36:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:36:25.351743963 +0000 UTC m=+548.454811676" watchObservedRunningTime="2026-10-31 12:36:25.352542687 +0000 UTC m=+548.455610401"
Oct 31 12:36:25.393512 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:25.388459    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5b68ffb5fd-6s2ss"]
Oct 31 12:36:25.406511 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:25.403771    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-5b68ffb5fd-6s2ss" podUID="bfc6317b-c496-4193-96fc-730f641a6b53" containerName="authorino" containerID="cri-o://6203b03c38abd072f086eb730ebb7e3a963464efd323a65d5095b0ae875f5197" gracePeriod=30
Oct 31 12:36:25.711914 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:25.711883    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5b68ffb5fd-6s2ss"
Oct 31 12:36:25.774244 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:25.774200    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bfc6317b-c496-4193-96fc-730f641a6b53-tls-cert\" (UniqueName: \"kubernetes.io/secret/bfc6317b-c496-4193-96fc-730f641a6b53-tls-cert\") pod \"bfc6317b-c496-4193-96fc-730f641a6b53\" (UID: \"bfc6317b-c496-4193-96fc-730f641a6b53\") "
Oct 31 12:36:25.774430 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:25.774256    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfc6317b-c496-4193-96fc-730f641a6b53-kube-api-access-mh2tz\" (UniqueName: \"kubernetes.io/projected/bfc6317b-c496-4193-96fc-730f641a6b53-kube-api-access-mh2tz\") pod \"bfc6317b-c496-4193-96fc-730f641a6b53\" (UID: \"bfc6317b-c496-4193-96fc-730f641a6b53\") "
Oct 31 12:36:25.776534 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:25.776447    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfc6317b-c496-4193-96fc-730f641a6b53-kube-api-access-mh2tz" pod "bfc6317b-c496-4193-96fc-730f641a6b53" (UID: "bfc6317b-c496-4193-96fc-730f641a6b53"). InnerVolumeSpecName "kube-api-access-mh2tz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:36:25.786393 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:25.786358    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bfc6317b-c496-4193-96fc-730f641a6b53-tls-cert" pod "bfc6317b-c496-4193-96fc-730f641a6b53" (UID: "bfc6317b-c496-4193-96fc-730f641a6b53"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:36:25.875214 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:25.875128    5060 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/bfc6317b-c496-4193-96fc-730f641a6b53-tls-cert\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:36:25.875214 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:25.875171    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mh2tz\" (UniqueName: \"kubernetes.io/projected/bfc6317b-c496-4193-96fc-730f641a6b53-kube-api-access-mh2tz\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:36:25.972515 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:25.972464    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-697b76b867-cfz9w"]
Oct 31 12:36:25.972925 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:25.972904    5060 cpu_manager.go:414] "containerMap: removing container" podUID="bfc6317b-c496-4193-96fc-730f641a6b53" containerName="authorino"
Oct 31 12:36:25.972925 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:25.972923    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfc6317b-c496-4193-96fc-730f641a6b53" containerName="authorino"
Oct 31 12:36:25.973096 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:25.973029    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfc6317b-c496-4193-96fc-730f641a6b53" containerName="authorino"
Oct 31 12:36:25.976218 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:25.976195    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-697b76b867-cfz9w"
Oct 31 12:36:25.983030 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:25.982999    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\""
Oct 31 12:36:25.983370 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:25.983314    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-872nt\""
Oct 31 12:36:25.983437 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:25.983411    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\""
Oct 31 12:36:25.986116 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:25.986092    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Oct 31 12:36:25.986116 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:25.986092    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Oct 31 12:36:25.990508 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:25.990469    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-697b76b867-cfz9w"]
Oct 31 12:36:26.076656 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:26.076607    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18755224-59d2-427d-bfab-ee573c6b54aa-metrics-certs\") pod \"ai-gateway-controller-697b76b867-cfz9w\" (UID: \"18755224-59d2-427d-bfab-ee573c6b54aa\") " pod="opendatahub/ai-gateway-controller-697b76b867-cfz9w"
Oct 31 12:36:26.076656 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:26.076653    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8kvc\" (UniqueName: \"kubernetes.io/projected/18755224-59d2-427d-bfab-ee573c6b54aa-kube-api-access-s8kvc\") pod \"ai-gateway-controller-697b76b867-cfz9w\" (UID: \"18755224-59d2-427d-bfab-ee573c6b54aa\") " pod="opendatahub/ai-gateway-controller-697b76b867-cfz9w"
Oct 31 12:36:26.178025 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:26.177940    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18755224-59d2-427d-bfab-ee573c6b54aa-metrics-certs\") pod \"ai-gateway-controller-697b76b867-cfz9w\" (UID: \"18755224-59d2-427d-bfab-ee573c6b54aa\") " pod="opendatahub/ai-gateway-controller-697b76b867-cfz9w"
Oct 31 12:36:26.178025 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:26.177980    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8kvc\" (UniqueName: \"kubernetes.io/projected/18755224-59d2-427d-bfab-ee573c6b54aa-kube-api-access-s8kvc\") pod \"ai-gateway-controller-697b76b867-cfz9w\" (UID: \"18755224-59d2-427d-bfab-ee573c6b54aa\") " pod="opendatahub/ai-gateway-controller-697b76b867-cfz9w"
Oct 31 12:36:26.180658 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:26.180632    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18755224-59d2-427d-bfab-ee573c6b54aa-metrics-certs\") pod \"ai-gateway-controller-697b76b867-cfz9w\" (UID: \"18755224-59d2-427d-bfab-ee573c6b54aa\") " pod="opendatahub/ai-gateway-controller-697b76b867-cfz9w"
Oct 31 12:36:26.192508 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:26.192463    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8kvc\" (UniqueName: \"kubernetes.io/projected/18755224-59d2-427d-bfab-ee573c6b54aa-kube-api-access-s8kvc\") pod \"ai-gateway-controller-697b76b867-cfz9w\" (UID: \"18755224-59d2-427d-bfab-ee573c6b54aa\") " pod="opendatahub/ai-gateway-controller-697b76b867-cfz9w"
Oct 31 12:36:26.293343 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:26.293309    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-697b76b867-cfz9w"
Oct 31 12:36:26.344850 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:26.343258    5060 generic.go:356] "Generic (PLEG): container finished" podID="bfc6317b-c496-4193-96fc-730f641a6b53" containerID="6203b03c38abd072f086eb730ebb7e3a963464efd323a65d5095b0ae875f5197" exitCode=0
Oct 31 12:36:26.344850 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:26.343358    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5b68ffb5fd-6s2ss"
Oct 31 12:36:26.344850 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:26.343393    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5b68ffb5fd-6s2ss" event={"ID":"bfc6317b-c496-4193-96fc-730f641a6b53","Type":"ContainerDied","Data":"6203b03c38abd072f086eb730ebb7e3a963464efd323a65d5095b0ae875f5197"}
Oct 31 12:36:26.344850 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:26.343426    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5b68ffb5fd-6s2ss" event={"ID":"bfc6317b-c496-4193-96fc-730f641a6b53","Type":"ContainerDied","Data":"f7f75ca6ad8e67d5a8143c261ad566852f31f92984875783823a68b392649d95"}
Oct 31 12:36:26.344850 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:26.343447    5060 scope.go:122] "RemoveContainer" containerID="6203b03c38abd072f086eb730ebb7e3a963464efd323a65d5095b0ae875f5197"
Oct 31 12:36:26.356191 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:26.356165    5060 scope.go:122] "RemoveContainer" containerID="6203b03c38abd072f086eb730ebb7e3a963464efd323a65d5095b0ae875f5197"
Oct 31 12:36:26.356574 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:36:26.356528    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6203b03c38abd072f086eb730ebb7e3a963464efd323a65d5095b0ae875f5197\": container with ID starting with 6203b03c38abd072f086eb730ebb7e3a963464efd323a65d5095b0ae875f5197 not found: ID does not exist" containerID="6203b03c38abd072f086eb730ebb7e3a963464efd323a65d5095b0ae875f5197"
Oct 31 12:36:26.356709 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:26.356569    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6203b03c38abd072f086eb730ebb7e3a963464efd323a65d5095b0ae875f5197"} err="failed to get container status \"6203b03c38abd072f086eb730ebb7e3a963464efd323a65d5095b0ae875f5197\": rpc error: code = NotFound desc = could not find container \"6203b03c38abd072f086eb730ebb7e3a963464efd323a65d5095b0ae875f5197\": container with ID starting with 6203b03c38abd072f086eb730ebb7e3a963464efd323a65d5095b0ae875f5197 not found: ID does not exist"
Oct 31 12:36:26.373660 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:26.373625    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5b68ffb5fd-6s2ss"]
Oct 31 12:36:26.387542 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:26.385273    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5b68ffb5fd-6s2ss"]
Oct 31 12:36:26.586851 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:26.586560    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-697b76b867-cfz9w"]
Oct 31 12:36:26.589730 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:36:26.589693    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18755224_59d2_427d_bfab_ee573c6b54aa.slice/crio-03fb0f04ad17a61b653775e611141a3d540b2bdc89045e0598d73b0497d13506 WatchSource:0}: Error finding container 03fb0f04ad17a61b653775e611141a3d540b2bdc89045e0598d73b0497d13506: Status 404 returned error can't find the container with id 03fb0f04ad17a61b653775e611141a3d540b2bdc89045e0598d73b0497d13506
Oct 31 12:36:27.048928 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:27.048891    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bfc6317b-c496-4193-96fc-730f641a6b53" path="/var/lib/kubelet/pods/bfc6317b-c496-4193-96fc-730f641a6b53/volumes"
Oct 31 12:36:27.354969 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:27.354870    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-697b76b867-cfz9w" event={"ID":"18755224-59d2-427d-bfab-ee573c6b54aa","Type":"ContainerStarted","Data":"03fb0f04ad17a61b653775e611141a3d540b2bdc89045e0598d73b0497d13506"}
Oct 31 12:36:29.202647 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:29.202611    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-7psf5"]
Oct 31 12:36:29.216302 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:29.216276    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-7psf5"
Oct 31 12:36:29.224630 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:29.222043    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-f7jrv\""
Oct 31 12:36:29.224630 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:29.223376    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-7psf5"]
Oct 31 12:36:29.241670 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:29.241622    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Oct 31 12:36:29.317419 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:29.317384    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37aec2da-59cb-41ed-9e6b-a443f6ff84c7-workload-socket\") pod \"maas-default-gateway-openshift-default-645cf464d7-7psf5\" (UID: \"37aec2da-59cb-41ed-9e6b-a443f6ff84c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-7psf5"
Oct 31 12:36:29.317627 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:29.317437    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37aec2da-59cb-41ed-9e6b-a443f6ff84c7-workload-certs\") pod \"maas-default-gateway-openshift-default-645cf464d7-7psf5\" (UID: \"37aec2da-59cb-41ed-9e6b-a443f6ff84c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-7psf5"
Oct 31 12:36:29.317627 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:29.317528    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37aec2da-59cb-41ed-9e6b-a443f6ff84c7-istio-token\") pod \"maas-default-gateway-openshift-default-645cf464d7-7psf5\" (UID: \"37aec2da-59cb-41ed-9e6b-a443f6ff84c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-7psf5"
Oct 31 12:36:29.317627 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:29.317603    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37aec2da-59cb-41ed-9e6b-a443f6ff84c7-istio-data\") pod \"maas-default-gateway-openshift-default-645cf464d7-7psf5\" (UID: \"37aec2da-59cb-41ed-9e6b-a443f6ff84c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-7psf5"
Oct 31 12:36:29.317627 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:29.317622    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37aec2da-59cb-41ed-9e6b-a443f6ff84c7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-645cf464d7-7psf5\" (UID: \"37aec2da-59cb-41ed-9e6b-a443f6ff84c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-7psf5"
Oct 31 12:36:29.317843 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:29.317640    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37aec2da-59cb-41ed-9e6b-a443f6ff84c7-credential-socket\") pod \"maas-default-gateway-openshift-default-645cf464d7-7psf5\" (UID: \"37aec2da-59cb-41ed-9e6b-a443f6ff84c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-7psf5"
Oct 31 12:36:29.317843 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:29.317685    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwlpg\" (UniqueName: \"kubernetes.io/projected/37aec2da-59cb-41ed-9e6b-a443f6ff84c7-kube-api-access-gwlpg\") pod \"maas-default-gateway-openshift-default-645cf464d7-7psf5\" (UID: \"37aec2da-59cb-41ed-9e6b-a443f6ff84c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-7psf5"
Oct 31 12:36:29.317843 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:29.317728    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37aec2da-59cb-41ed-9e6b-a443f6ff84c7-istio-envoy\") pod \"maas-default-gateway-openshift-default-645cf464d7-7psf5\" (UID: \"37aec2da-59cb-41ed-9e6b-a443f6ff84c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-7psf5"
Oct 31 12:36:29.317843 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:29.317787    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37aec2da-59cb-41ed-9e6b-a443f6ff84c7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-645cf464d7-7psf5\" (UID: \"37aec2da-59cb-41ed-9e6b-a443f6ff84c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-7psf5"
Oct 31 12:36:29.419360 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:29.419319    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37aec2da-59cb-41ed-9e6b-a443f6ff84c7-istio-token\") pod \"maas-default-gateway-openshift-default-645cf464d7-7psf5\" (UID: \"37aec2da-59cb-41ed-9e6b-a443f6ff84c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-7psf5"
Oct 31 12:36:29.419575 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:29.419387    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37aec2da-59cb-41ed-9e6b-a443f6ff84c7-istio-data\") pod \"maas-default-gateway-openshift-default-645cf464d7-7psf5\" (UID: \"37aec2da-59cb-41ed-9e6b-a443f6ff84c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-7psf5"
Oct 31 12:36:29.419575 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:29.419414    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37aec2da-59cb-41ed-9e6b-a443f6ff84c7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-645cf464d7-7psf5\" (UID: \"37aec2da-59cb-41ed-9e6b-a443f6ff84c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-7psf5"
Oct 31 12:36:29.419575 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:29.419442    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37aec2da-59cb-41ed-9e6b-a443f6ff84c7-credential-socket\") pod \"maas-default-gateway-openshift-default-645cf464d7-7psf5\" (UID: \"37aec2da-59cb-41ed-9e6b-a443f6ff84c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-7psf5"
Oct 31 12:36:29.419575 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:29.419485    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwlpg\" (UniqueName: \"kubernetes.io/projected/37aec2da-59cb-41ed-9e6b-a443f6ff84c7-kube-api-access-gwlpg\") pod \"maas-default-gateway-openshift-default-645cf464d7-7psf5\" (UID: \"37aec2da-59cb-41ed-9e6b-a443f6ff84c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-7psf5"
Oct 31 12:36:29.419575 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:29.419554    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37aec2da-59cb-41ed-9e6b-a443f6ff84c7-istio-envoy\") pod \"maas-default-gateway-openshift-default-645cf464d7-7psf5\" (UID: \"37aec2da-59cb-41ed-9e6b-a443f6ff84c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-7psf5"
Oct 31 12:36:29.419849 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:29.419586    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37aec2da-59cb-41ed-9e6b-a443f6ff84c7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-645cf464d7-7psf5\" (UID: \"37aec2da-59cb-41ed-9e6b-a443f6ff84c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-7psf5"
Oct 31 12:36:29.419849 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:29.419686    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37aec2da-59cb-41ed-9e6b-a443f6ff84c7-workload-socket\") pod \"maas-default-gateway-openshift-default-645cf464d7-7psf5\" (UID: \"37aec2da-59cb-41ed-9e6b-a443f6ff84c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-7psf5"
Oct 31 12:36:29.419849 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:29.419752    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37aec2da-59cb-41ed-9e6b-a443f6ff84c7-workload-certs\") pod \"maas-default-gateway-openshift-default-645cf464d7-7psf5\" (UID: \"37aec2da-59cb-41ed-9e6b-a443f6ff84c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-7psf5"
Oct 31 12:36:29.420293 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:29.420268    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37aec2da-59cb-41ed-9e6b-a443f6ff84c7-workload-certs\") pod \"maas-default-gateway-openshift-default-645cf464d7-7psf5\" (UID: \"37aec2da-59cb-41ed-9e6b-a443f6ff84c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-7psf5"
Oct 31 12:36:29.420850 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:29.420831    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37aec2da-59cb-41ed-9e6b-a443f6ff84c7-istio-data\") pod \"maas-default-gateway-openshift-default-645cf464d7-7psf5\" (UID: \"37aec2da-59cb-41ed-9e6b-a443f6ff84c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-7psf5"
Oct 31 12:36:29.421298 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:29.421281    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37aec2da-59cb-41ed-9e6b-a443f6ff84c7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-645cf464d7-7psf5\" (UID: \"37aec2da-59cb-41ed-9e6b-a443f6ff84c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-7psf5"
Oct 31 12:36:29.421475 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:29.421460    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37aec2da-59cb-41ed-9e6b-a443f6ff84c7-credential-socket\") pod \"maas-default-gateway-openshift-default-645cf464d7-7psf5\" (UID: \"37aec2da-59cb-41ed-9e6b-a443f6ff84c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-7psf5"
Oct 31 12:36:29.422215 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:29.421825    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37aec2da-59cb-41ed-9e6b-a443f6ff84c7-workload-socket\") pod \"maas-default-gateway-openshift-default-645cf464d7-7psf5\" (UID: \"37aec2da-59cb-41ed-9e6b-a443f6ff84c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-7psf5"
Oct 31 12:36:29.424949 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:29.424534    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37aec2da-59cb-41ed-9e6b-a443f6ff84c7-istio-envoy\") pod \"maas-default-gateway-openshift-default-645cf464d7-7psf5\" (UID: \"37aec2da-59cb-41ed-9e6b-a443f6ff84c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-7psf5"
Oct 31 12:36:29.428070 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:29.428046    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37aec2da-59cb-41ed-9e6b-a443f6ff84c7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-645cf464d7-7psf5\" (UID: \"37aec2da-59cb-41ed-9e6b-a443f6ff84c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-7psf5"
Oct 31 12:36:29.438776 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:29.438724    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37aec2da-59cb-41ed-9e6b-a443f6ff84c7-istio-token\") pod \"maas-default-gateway-openshift-default-645cf464d7-7psf5\" (UID: \"37aec2da-59cb-41ed-9e6b-a443f6ff84c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-7psf5"
Oct 31 12:36:29.443191 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:29.443136    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwlpg\" (UniqueName: \"kubernetes.io/projected/37aec2da-59cb-41ed-9e6b-a443f6ff84c7-kube-api-access-gwlpg\") pod \"maas-default-gateway-openshift-default-645cf464d7-7psf5\" (UID: \"37aec2da-59cb-41ed-9e6b-a443f6ff84c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-7psf5"
Oct 31 12:36:29.537790 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:29.537749    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-7psf5"
Oct 31 12:36:29.845463 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:29.845341    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-7psf5"]
Oct 31 12:36:29.848590 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:36:29.848561    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37aec2da_59cb_41ed_9e6b_a443f6ff84c7.slice/crio-3e1a3368775aeb4a9ed387ca947696ae3d512c081d3a47f58727a901a78be296 WatchSource:0}: Error finding container 3e1a3368775aeb4a9ed387ca947696ae3d512c081d3a47f58727a901a78be296: Status 404 returned error can't find the container with id 3e1a3368775aeb4a9ed387ca947696ae3d512c081d3a47f58727a901a78be296
Oct 31 12:36:29.923329 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:36:29.923283    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7phb" containerName="extractor"
Oct 31 12:36:30.390282 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:30.390241    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-7psf5" event={"ID":"37aec2da-59cb-41ed-9e6b-a443f6ff84c7","Type":"ContainerStarted","Data":"3e1a3368775aeb4a9ed387ca947696ae3d512c081d3a47f58727a901a78be296"}
Oct 31 12:36:30.392942 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:30.392908    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-697b76b867-cfz9w" event={"ID":"18755224-59d2-427d-bfab-ee573c6b54aa","Type":"ContainerStarted","Data":"e8b8dbe67572e2aa8d7134d86790deeea1e3f279eb79577467817037c81e54c9"}
Oct 31 12:36:30.393297 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:30.393261    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-697b76b867-cfz9w"
Oct 31 12:36:30.418368 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:30.418319    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-697b76b867-cfz9w" podStartSLOduration=2.073690446 podStartE2EDuration="5.418304875s" totalImagesPullingTime="3.344614429s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:36:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:36:30.416879463 +0000 UTC m=+553.519947174" watchObservedRunningTime="2026-10-31 12:36:30.418304875 +0000 UTC m=+553.521372587"
Oct 31 12:36:31.745444 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:31.745413    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-vtzns"]
Oct 31 12:36:31.749129 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:31.749104    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-vtzns"
Oct 31 12:36:31.758527 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:31.757876    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Oct 31 12:36:31.762235 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:31.762204    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-v452b"]
Oct 31 12:36:31.777295 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:31.777237    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-vtzns"]
Oct 31 12:36:31.777519 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:31.777466    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-v452b"
Oct 31 12:36:31.781883 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:31.781855    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-l7jvw\""
Oct 31 12:36:31.795220 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:31.795178    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-v452b"]
Oct 31 12:36:31.845272 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:31.845233    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3ee1ad7-1684-4a48-b7fb-b35b3a45b86d-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-vtzns\" (UID: \"d3ee1ad7-1684-4a48-b7fb-b35b3a45b86d\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-vtzns"
Oct 31 12:36:31.845455 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:31.845282    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ww6k9\" (UniqueName: \"kubernetes.io/projected/6e5b132f-d4fc-48de-beab-1c638e61d6a4-kube-api-access-ww6k9\") pod \"payload-processing-55fbbbf9dc-v452b\" (UID: \"6e5b132f-d4fc-48de-beab-1c638e61d6a4\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-v452b"
Oct 31 12:36:31.845455 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:31.845313    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8g7f\" (UniqueName: \"kubernetes.io/projected/d3ee1ad7-1684-4a48-b7fb-b35b3a45b86d-kube-api-access-f8g7f\") pod \"payload-pre-processing-7b98bb678c-vtzns\" (UID: \"d3ee1ad7-1684-4a48-b7fb-b35b3a45b86d\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-vtzns"
Oct 31 12:36:31.845614 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:31.845525    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e5b132f-d4fc-48de-beab-1c638e61d6a4-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-v452b\" (UID: \"6e5b132f-d4fc-48de-beab-1c638e61d6a4\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-v452b"
Oct 31 12:36:31.946803 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:31.946769    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e5b132f-d4fc-48de-beab-1c638e61d6a4-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-v452b\" (UID: \"6e5b132f-d4fc-48de-beab-1c638e61d6a4\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-v452b"
Oct 31 12:36:31.946975 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:31.946836    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3ee1ad7-1684-4a48-b7fb-b35b3a45b86d-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-vtzns\" (UID: \"d3ee1ad7-1684-4a48-b7fb-b35b3a45b86d\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-vtzns"
Oct 31 12:36:31.946975 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:31.946863    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ww6k9\" (UniqueName: \"kubernetes.io/projected/6e5b132f-d4fc-48de-beab-1c638e61d6a4-kube-api-access-ww6k9\") pod \"payload-processing-55fbbbf9dc-v452b\" (UID: \"6e5b132f-d4fc-48de-beab-1c638e61d6a4\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-v452b"
Oct 31 12:36:31.946975 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:31.946888    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8g7f\" (UniqueName: \"kubernetes.io/projected/d3ee1ad7-1684-4a48-b7fb-b35b3a45b86d-kube-api-access-f8g7f\") pod \"payload-pre-processing-7b98bb678c-vtzns\" (UID: \"d3ee1ad7-1684-4a48-b7fb-b35b3a45b86d\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-vtzns"
Oct 31 12:36:31.947613 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:31.947590    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e5b132f-d4fc-48de-beab-1c638e61d6a4-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-v452b\" (UID: \"6e5b132f-d4fc-48de-beab-1c638e61d6a4\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-v452b"
Oct 31 12:36:31.947663 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:31.947636    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3ee1ad7-1684-4a48-b7fb-b35b3a45b86d-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-vtzns\" (UID: \"d3ee1ad7-1684-4a48-b7fb-b35b3a45b86d\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-vtzns"
Oct 31 12:36:31.959605 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:31.959570    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ww6k9\" (UniqueName: \"kubernetes.io/projected/6e5b132f-d4fc-48de-beab-1c638e61d6a4-kube-api-access-ww6k9\") pod \"payload-processing-55fbbbf9dc-v452b\" (UID: \"6e5b132f-d4fc-48de-beab-1c638e61d6a4\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-v452b"
Oct 31 12:36:31.959742 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:31.959703    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8g7f\" (UniqueName: \"kubernetes.io/projected/d3ee1ad7-1684-4a48-b7fb-b35b3a45b86d-kube-api-access-f8g7f\") pod \"payload-pre-processing-7b98bb678c-vtzns\" (UID: \"d3ee1ad7-1684-4a48-b7fb-b35b3a45b86d\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-vtzns"
Oct 31 12:36:32.078739 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:32.078700    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-vtzns"
Oct 31 12:36:32.099878 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:32.099850    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-v452b"
Oct 31 12:36:32.379073 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:32.379044    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-vtzns"]
Oct 31 12:36:32.380892 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:36:32.380861    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3ee1ad7_1684_4a48_b7fb_b35b3a45b86d.slice/crio-3cac9e5cf197164a5ba687733245a9f9765ae12ee8a89e503ffe756bd7e3a3e1 WatchSource:0}: Error finding container 3cac9e5cf197164a5ba687733245a9f9765ae12ee8a89e503ffe756bd7e3a3e1: Status 404 returned error can't find the container with id 3cac9e5cf197164a5ba687733245a9f9765ae12ee8a89e503ffe756bd7e3a3e1
Oct 31 12:36:32.407366 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:32.407336    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-v452b"]
Oct 31 12:36:32.408337 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:32.408309    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-vtzns" event={"ID":"d3ee1ad7-1684-4a48-b7fb-b35b3a45b86d","Type":"ContainerStarted","Data":"3cac9e5cf197164a5ba687733245a9f9765ae12ee8a89e503ffe756bd7e3a3e1"}
Oct 31 12:36:32.408546 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:36:32.408525    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e5b132f_d4fc_48de_beab_1c638e61d6a4.slice/crio-9aa90705c7d0649245e811a8275b6cc61bdfd3e1d6af6fb43f8f6d407f172839 WatchSource:0}: Error finding container 9aa90705c7d0649245e811a8275b6cc61bdfd3e1d6af6fb43f8f6d407f172839: Status 404 returned error can't find the container with id 9aa90705c7d0649245e811a8275b6cc61bdfd3e1d6af6fb43f8f6d407f172839
Oct 31 12:36:33.418367 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:33.418280    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-v452b" event={"ID":"6e5b132f-d4fc-48de-beab-1c638e61d6a4","Type":"ContainerStarted","Data":"9aa90705c7d0649245e811a8275b6cc61bdfd3e1d6af6fb43f8f6d407f172839"}
Oct 31 12:36:33.939743 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:33.939705    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:36:33.939908 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:33.939789    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:36:33.939908 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:33.939821    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:36:34.428453 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:34.428405    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-7psf5" event={"ID":"37aec2da-59cb-41ed-9e6b-a443f6ff84c7","Type":"ContainerStarted","Data":"3bb80413cbd17c421ee5c661be4f34da50dff2649f2327241893fcf7787296ca"}
Oct 31 12:36:34.453898 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:34.452796    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-7psf5" podStartSLOduration=1.365136971 podStartE2EDuration="5.452778238s" totalImagesPullingTime="4.087641267s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:36:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:36:34.451871161 +0000 UTC m=+557.554938882" watchObservedRunningTime="2026-10-31 12:36:34.452778238 +0000 UTC m=+557.555845951"
Oct 31 12:36:34.538443 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:34.538406    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-7psf5"
Oct 31 12:36:35.540471 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:35.540425    5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-645cf464d7-7psf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Oct 31 12:36:35.540965 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:35.540529    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-7psf5" podUID="37aec2da-59cb-41ed-9e6b-a443f6ff84c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Oct 31 12:36:36.444599 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:36.444558    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-v452b" event={"ID":"6e5b132f-d4fc-48de-beab-1c638e61d6a4","Type":"ContainerStarted","Data":"33b1529ef38efb4b6d9c2bc860f616e6fb0e64fe0e6e42e0981410280a374240"}
Oct 31 12:36:36.444815 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:36.444797    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-v452b"
Oct 31 12:36:36.446238 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:36.446209    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-vtzns" event={"ID":"d3ee1ad7-1684-4a48-b7fb-b35b3a45b86d","Type":"ContainerStarted","Data":"9e27128a4678bc8bea5dc6ebbd70fecac95da39aa609176242fa3f0d0b3a2071"}
Oct 31 12:36:36.446441 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:36.446422    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-vtzns"
Oct 31 12:36:36.470195 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:36.470133    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-55fbbbf9dc-v452b" podStartSLOduration=1.890812136 podStartE2EDuration="5.470113849s" totalImagesPullingTime="3.579301713s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:36:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:36:36.468869324 +0000 UTC m=+559.571937037" watchObservedRunningTime="2026-10-31 12:36:36.470113849 +0000 UTC m=+559.573181563"
Oct 31 12:36:36.488804 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:36.488758    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-vtzns" podStartSLOduration=1.933726107 podStartE2EDuration="5.488745204s" totalImagesPullingTime="3.555019097s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:36:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:36:36.486845948 +0000 UTC m=+559.589913663" watchObservedRunningTime="2026-10-31 12:36:36.488745204 +0000 UTC m=+559.591812916"
Oct 31 12:36:36.538114 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:36.538075    5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-645cf464d7-7psf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Oct 31 12:36:36.538301 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:36.538143    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-7psf5" podUID="37aec2da-59cb-41ed-9e6b-a443f6ff84c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Oct 31 12:36:37.538483 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:37.538446    5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-645cf464d7-7psf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Oct 31 12:36:37.540808 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:37.538537    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-7psf5" podUID="37aec2da-59cb-41ed-9e6b-a443f6ff84c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Oct 31 12:36:38.538373 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:38.538338    5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-645cf464d7-7psf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Oct 31 12:36:38.538596 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:38.538399    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-7psf5" podUID="37aec2da-59cb-41ed-9e6b-a443f6ff84c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Oct 31 12:36:39.538124 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:39.538084    5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-645cf464d7-7psf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Oct 31 12:36:39.538310 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:39.538148    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-7psf5" podUID="37aec2da-59cb-41ed-9e6b-a443f6ff84c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Oct 31 12:36:40.538522 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:40.538471    5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-645cf464d7-7psf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Oct 31 12:36:40.539019 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:40.538535    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-7psf5" podUID="37aec2da-59cb-41ed-9e6b-a443f6ff84c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Oct 31 12:36:41.402462 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:41.402431    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-697b76b867-cfz9w"
Oct 31 12:36:41.662469 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:41.662375    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-56dd777d6d-cnzrb"]
Oct 31 12:36:41.666426 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:41.666406    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-56dd777d6d-cnzrb"
Oct 31 12:36:41.678187 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:41.678162    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-56dd777d6d-cnzrb"]
Oct 31 12:36:41.746697 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:41.746659    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4535c997-310c-4fb4-bd16-d268151053ab-metrics-certs\") pod \"ai-gateway-controller-56dd777d6d-cnzrb\" (UID: \"4535c997-310c-4fb4-bd16-d268151053ab\") " pod="opendatahub/ai-gateway-controller-56dd777d6d-cnzrb"
Oct 31 12:36:41.746856 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:41.746706    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8tkg\" (UniqueName: \"kubernetes.io/projected/4535c997-310c-4fb4-bd16-d268151053ab-kube-api-access-j8tkg\") pod \"ai-gateway-controller-56dd777d6d-cnzrb\" (UID: \"4535c997-310c-4fb4-bd16-d268151053ab\") " pod="opendatahub/ai-gateway-controller-56dd777d6d-cnzrb"
Oct 31 12:36:41.847782 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:41.847748    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4535c997-310c-4fb4-bd16-d268151053ab-metrics-certs\") pod \"ai-gateway-controller-56dd777d6d-cnzrb\" (UID: \"4535c997-310c-4fb4-bd16-d268151053ab\") " pod="opendatahub/ai-gateway-controller-56dd777d6d-cnzrb"
Oct 31 12:36:41.847945 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:41.847786    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8tkg\" (UniqueName: \"kubernetes.io/projected/4535c997-310c-4fb4-bd16-d268151053ab-kube-api-access-j8tkg\") pod \"ai-gateway-controller-56dd777d6d-cnzrb\" (UID: \"4535c997-310c-4fb4-bd16-d268151053ab\") " pod="opendatahub/ai-gateway-controller-56dd777d6d-cnzrb"
Oct 31 12:36:41.850280 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:41.850252    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4535c997-310c-4fb4-bd16-d268151053ab-metrics-certs\") pod \"ai-gateway-controller-56dd777d6d-cnzrb\" (UID: \"4535c997-310c-4fb4-bd16-d268151053ab\") " pod="opendatahub/ai-gateway-controller-56dd777d6d-cnzrb"
Oct 31 12:36:41.873053 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:41.872885    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8tkg\" (UniqueName: \"kubernetes.io/projected/4535c997-310c-4fb4-bd16-d268151053ab-kube-api-access-j8tkg\") pod \"ai-gateway-controller-56dd777d6d-cnzrb\" (UID: \"4535c997-310c-4fb4-bd16-d268151053ab\") " pod="opendatahub/ai-gateway-controller-56dd777d6d-cnzrb"
Oct 31 12:36:41.984165 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:41.984066    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-56dd777d6d-cnzrb"
Oct 31 12:36:42.266734 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:42.266706    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-56dd777d6d-cnzrb"]
Oct 31 12:36:42.332412 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:42.332377    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-7psf5"
Oct 31 12:36:42.333163 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:42.333138    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-7psf5"
Oct 31 12:36:42.333980 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:42.333958    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-7psf5"
Oct 31 12:36:42.505840 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:42.505798    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-56dd777d6d-cnzrb" event={"ID":"4535c997-310c-4fb4-bd16-d268151053ab","Type":"ContainerStarted","Data":"069a9092d09617b0161e03f12d84d643a2f82f8bd803137e4ed2b956976b8896"}
Oct 31 12:36:43.513914 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:43.513870    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-56dd777d6d-cnzrb" event={"ID":"4535c997-310c-4fb4-bd16-d268151053ab","Type":"ContainerStarted","Data":"8d5d0e41721a907d29b7cf91aab53aad60245e1e6713da1fdce05d1239e69cdf"}
Oct 31 12:36:43.514336 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:43.514037    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-56dd777d6d-cnzrb"
Oct 31 12:36:43.534684 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:43.534640    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-56dd777d6d-cnzrb" podStartSLOduration=2.108373733 podStartE2EDuration="2.534627442s" totalImagesPullingTime="426.253709ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:36:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:36:43.533064838 +0000 UTC m=+566.636132550" watchObservedRunningTime="2026-10-31 12:36:43.534627442 +0000 UTC m=+566.637695154"
Oct 31 12:36:44.217459 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:44.217420    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-v2sv5"]
Oct 31 12:36:44.221449 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:44.221331    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-v2sv5"
Oct 31 12:36:44.235059 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:44.235032    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-v2sv5"]
Oct 31 12:36:44.271144 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:44.271112    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e33b790-15e5-4754-929f-c0a7330b9361-istio-envoy\") pod \"maas-default-gateway-openshift-default-645cf464d7-v2sv5\" (UID: \"1e33b790-15e5-4754-929f-c0a7330b9361\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-v2sv5"
Oct 31 12:36:44.271144 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:44.271149    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e33b790-15e5-4754-929f-c0a7330b9361-credential-socket\") pod \"maas-default-gateway-openshift-default-645cf464d7-v2sv5\" (UID: \"1e33b790-15e5-4754-929f-c0a7330b9361\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-v2sv5"
Oct 31 12:36:44.271344 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:44.271167    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e33b790-15e5-4754-929f-c0a7330b9361-workload-certs\") pod \"maas-default-gateway-openshift-default-645cf464d7-v2sv5\" (UID: \"1e33b790-15e5-4754-929f-c0a7330b9361\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-v2sv5"
Oct 31 12:36:44.271344 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:44.271190    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e33b790-15e5-4754-929f-c0a7330b9361-istio-data\") pod \"maas-default-gateway-openshift-default-645cf464d7-v2sv5\" (UID: \"1e33b790-15e5-4754-929f-c0a7330b9361\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-v2sv5"
Oct 31 12:36:44.271344 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:44.271236    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e33b790-15e5-4754-929f-c0a7330b9361-istio-token\") pod \"maas-default-gateway-openshift-default-645cf464d7-v2sv5\" (UID: \"1e33b790-15e5-4754-929f-c0a7330b9361\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-v2sv5"
Oct 31 12:36:44.271344 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:44.271327    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e33b790-15e5-4754-929f-c0a7330b9361-istio-podinfo\") pod \"maas-default-gateway-openshift-default-645cf464d7-v2sv5\" (UID: \"1e33b790-15e5-4754-929f-c0a7330b9361\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-v2sv5"
Oct 31 12:36:44.271546 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:44.271384    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9lpm\" (UniqueName: \"kubernetes.io/projected/1e33b790-15e5-4754-929f-c0a7330b9361-kube-api-access-f9lpm\") pod \"maas-default-gateway-openshift-default-645cf464d7-v2sv5\" (UID: \"1e33b790-15e5-4754-929f-c0a7330b9361\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-v2sv5"
Oct 31 12:36:44.271546 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:44.271511    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e33b790-15e5-4754-929f-c0a7330b9361-workload-socket\") pod \"maas-default-gateway-openshift-default-645cf464d7-v2sv5\" (UID: \"1e33b790-15e5-4754-929f-c0a7330b9361\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-v2sv5"
Oct 31 12:36:44.271644 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:44.271574    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e33b790-15e5-4754-929f-c0a7330b9361-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-645cf464d7-v2sv5\" (UID: \"1e33b790-15e5-4754-929f-c0a7330b9361\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-v2sv5"
Oct 31 12:36:44.372881 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:44.372849    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e33b790-15e5-4754-929f-c0a7330b9361-istio-data\") pod \"maas-default-gateway-openshift-default-645cf464d7-v2sv5\" (UID: \"1e33b790-15e5-4754-929f-c0a7330b9361\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-v2sv5"
Oct 31 12:36:44.372881 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:44.372883    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e33b790-15e5-4754-929f-c0a7330b9361-istio-token\") pod \"maas-default-gateway-openshift-default-645cf464d7-v2sv5\" (UID: \"1e33b790-15e5-4754-929f-c0a7330b9361\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-v2sv5"
Oct 31 12:36:44.373082 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:44.372905    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e33b790-15e5-4754-929f-c0a7330b9361-istio-podinfo\") pod \"maas-default-gateway-openshift-default-645cf464d7-v2sv5\" (UID: \"1e33b790-15e5-4754-929f-c0a7330b9361\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-v2sv5"
Oct 31 12:36:44.373082 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:44.372949    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9lpm\" (UniqueName: \"kubernetes.io/projected/1e33b790-15e5-4754-929f-c0a7330b9361-kube-api-access-f9lpm\") pod \"maas-default-gateway-openshift-default-645cf464d7-v2sv5\" (UID: \"1e33b790-15e5-4754-929f-c0a7330b9361\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-v2sv5"
Oct 31 12:36:44.373082 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:44.372997    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e33b790-15e5-4754-929f-c0a7330b9361-workload-socket\") pod \"maas-default-gateway-openshift-default-645cf464d7-v2sv5\" (UID: \"1e33b790-15e5-4754-929f-c0a7330b9361\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-v2sv5"
Oct 31 12:36:44.373082 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:44.373044    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e33b790-15e5-4754-929f-c0a7330b9361-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-645cf464d7-v2sv5\" (UID: \"1e33b790-15e5-4754-929f-c0a7330b9361\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-v2sv5"
Oct 31 12:36:44.373288 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:44.373089    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e33b790-15e5-4754-929f-c0a7330b9361-istio-envoy\") pod \"maas-default-gateway-openshift-default-645cf464d7-v2sv5\" (UID: \"1e33b790-15e5-4754-929f-c0a7330b9361\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-v2sv5"
Oct 31 12:36:44.373288 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:44.373115    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e33b790-15e5-4754-929f-c0a7330b9361-credential-socket\") pod \"maas-default-gateway-openshift-default-645cf464d7-v2sv5\" (UID: \"1e33b790-15e5-4754-929f-c0a7330b9361\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-v2sv5"
Oct 31 12:36:44.373288 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:44.373222    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e33b790-15e5-4754-929f-c0a7330b9361-workload-certs\") pod \"maas-default-gateway-openshift-default-645cf464d7-v2sv5\" (UID: \"1e33b790-15e5-4754-929f-c0a7330b9361\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-v2sv5"
Oct 31 12:36:44.373453 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:44.373303    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e33b790-15e5-4754-929f-c0a7330b9361-istio-data\") pod \"maas-default-gateway-openshift-default-645cf464d7-v2sv5\" (UID: \"1e33b790-15e5-4754-929f-c0a7330b9361\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-v2sv5"
Oct 31 12:36:44.373544 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:44.373461    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e33b790-15e5-4754-929f-c0a7330b9361-credential-socket\") pod \"maas-default-gateway-openshift-default-645cf464d7-v2sv5\" (UID: \"1e33b790-15e5-4754-929f-c0a7330b9361\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-v2sv5"
Oct 31 12:36:44.373601 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:44.373541    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e33b790-15e5-4754-929f-c0a7330b9361-workload-socket\") pod \"maas-default-gateway-openshift-default-645cf464d7-v2sv5\" (UID: \"1e33b790-15e5-4754-929f-c0a7330b9361\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-v2sv5"
Oct 31 12:36:44.373651 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:44.373631    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e33b790-15e5-4754-929f-c0a7330b9361-workload-certs\") pod \"maas-default-gateway-openshift-default-645cf464d7-v2sv5\" (UID: \"1e33b790-15e5-4754-929f-c0a7330b9361\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-v2sv5"
Oct 31 12:36:44.373980 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:44.373954    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e33b790-15e5-4754-929f-c0a7330b9361-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-645cf464d7-v2sv5\" (UID: \"1e33b790-15e5-4754-929f-c0a7330b9361\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-v2sv5"
Oct 31 12:36:44.375365 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:44.375344    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e33b790-15e5-4754-929f-c0a7330b9361-istio-envoy\") pod \"maas-default-gateway-openshift-default-645cf464d7-v2sv5\" (UID: \"1e33b790-15e5-4754-929f-c0a7330b9361\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-v2sv5"
Oct 31 12:36:44.375607 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:44.375592    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e33b790-15e5-4754-929f-c0a7330b9361-istio-podinfo\") pod \"maas-default-gateway-openshift-default-645cf464d7-v2sv5\" (UID: \"1e33b790-15e5-4754-929f-c0a7330b9361\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-v2sv5"
Oct 31 12:36:44.385539 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:44.385518    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e33b790-15e5-4754-929f-c0a7330b9361-istio-token\") pod \"maas-default-gateway-openshift-default-645cf464d7-v2sv5\" (UID: \"1e33b790-15e5-4754-929f-c0a7330b9361\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-v2sv5"
Oct 31 12:36:44.385651 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:44.385594    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9lpm\" (UniqueName: \"kubernetes.io/projected/1e33b790-15e5-4754-929f-c0a7330b9361-kube-api-access-f9lpm\") pod \"maas-default-gateway-openshift-default-645cf464d7-v2sv5\" (UID: \"1e33b790-15e5-4754-929f-c0a7330b9361\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-v2sv5"
Oct 31 12:36:44.538712 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:44.538687    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-v2sv5"
Oct 31 12:36:44.824584 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:44.824334    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-v2sv5"]
Oct 31 12:36:44.826940 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:36:44.826913    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e33b790_15e5_4754_929f_c0a7330b9361.slice/crio-fa7d629062693e7f49fbb860fb02ae52298084218816068bdc9c8973af8f3fa6 WatchSource:0}: Error finding container fa7d629062693e7f49fbb860fb02ae52298084218816068bdc9c8973af8f3fa6: Status 404 returned error can't find the container with id fa7d629062693e7f49fbb860fb02ae52298084218816068bdc9c8973af8f3fa6
Oct 31 12:36:44.830154 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:44.830120    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:36:44.830231 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:44.830187    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:36:44.830231 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:44.830214    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:36:45.528547 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:45.528480    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-v2sv5" event={"ID":"1e33b790-15e5-4754-929f-c0a7330b9361","Type":"ContainerStarted","Data":"61d1f66705dbeb17fd9cbd694f56a41316654ebc7e876f422e77fca134c16d3f"}
Oct 31 12:36:45.528738 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:45.528572    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-v2sv5" event={"ID":"1e33b790-15e5-4754-929f-c0a7330b9361","Type":"ContainerStarted","Data":"fa7d629062693e7f49fbb860fb02ae52298084218816068bdc9c8973af8f3fa6"}
Oct 31 12:36:45.539267 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:45.539241    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-v2sv5"
Oct 31 12:36:45.540782 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:45.540757    5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-645cf464d7-v2sv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body=
Oct 31 12:36:45.540907 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:45.540807    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-v2sv5" podUID="1e33b790-15e5-4754-929f-c0a7330b9361" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused"
Oct 31 12:36:45.553893 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:45.553850    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-v2sv5" podStartSLOduration=1.5538385799999999 podStartE2EDuration="1.55383858s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:36:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:36:45.551979963 +0000 UTC m=+568.655047670" watchObservedRunningTime="2026-10-31 12:36:45.55383858 +0000 UTC m=+568.656906300"
Oct 31 12:36:46.539357 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:46.539325    5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-645cf464d7-v2sv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body=
Oct 31 12:36:46.539772 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:46.539375    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-v2sv5" podUID="1e33b790-15e5-4754-929f-c0a7330b9361" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused"
Oct 31 12:36:47.454554 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:47.454522    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-vtzns"
Oct 31 12:36:47.461519 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:47.454851    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-v452b"
Oct 31 12:36:47.539591 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:47.539555    5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-645cf464d7-v2sv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body=
Oct 31 12:36:47.539973 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:47.539629    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-v2sv5" podUID="1e33b790-15e5-4754-929f-c0a7330b9361" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused"
Oct 31 12:36:48.539003 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:48.538964    5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-645cf464d7-v2sv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body=
Oct 31 12:36:48.539153 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:48.539033    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-v2sv5" podUID="1e33b790-15e5-4754-929f-c0a7330b9361" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused"
Oct 31 12:36:49.539667 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:49.539632    5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-645cf464d7-v2sv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body=
Oct 31 12:36:49.540065 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:49.539701    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-v2sv5" podUID="1e33b790-15e5-4754-929f-c0a7330b9361" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused"
Oct 31 12:36:50.539606 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:50.539571    5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-645cf464d7-v2sv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body=
Oct 31 12:36:50.539781 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:50.539631    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-v2sv5" podUID="1e33b790-15e5-4754-929f-c0a7330b9361" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused"
Oct 31 12:36:51.539468 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:51.539429    5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-645cf464d7-v2sv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body=
Oct 31 12:36:51.539650 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:51.539484    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-v2sv5" podUID="1e33b790-15e5-4754-929f-c0a7330b9361" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused"
Oct 31 12:36:53.012781 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:53.012747    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-v2sv5"
Oct 31 12:36:53.013255 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:53.013186    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-v2sv5"
Oct 31 12:36:53.013964 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:53.013944    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-v2sv5"
Oct 31 12:36:54.523938 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:54.523899    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-56dd777d6d-cnzrb"
Oct 31 12:36:54.572300 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:54.572268    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-697b76b867-cfz9w"]
Oct 31 12:36:54.572478 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:54.572445    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-697b76b867-cfz9w" podUID="18755224-59d2-427d-bfab-ee573c6b54aa" containerName="manager" containerID="cri-o://e8b8dbe67572e2aa8d7134d86790deeea1e3f279eb79577467817037c81e54c9" gracePeriod=30
Oct 31 12:36:54.819975 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:54.819946    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-697b76b867-cfz9w"
Oct 31 12:36:54.872829 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:54.872783    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18755224-59d2-427d-bfab-ee573c6b54aa-kube-api-access-s8kvc\" (UniqueName: \"kubernetes.io/projected/18755224-59d2-427d-bfab-ee573c6b54aa-kube-api-access-s8kvc\") pod \"18755224-59d2-427d-bfab-ee573c6b54aa\" (UID: \"18755224-59d2-427d-bfab-ee573c6b54aa\") "
Oct 31 12:36:54.872974 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:54.872874    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18755224-59d2-427d-bfab-ee573c6b54aa-metrics-certs\" (UniqueName: \"kubernetes.io/secret/18755224-59d2-427d-bfab-ee573c6b54aa-metrics-certs\") pod \"18755224-59d2-427d-bfab-ee573c6b54aa\" (UID: \"18755224-59d2-427d-bfab-ee573c6b54aa\") "
Oct 31 12:36:54.875821 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:54.875792    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18755224-59d2-427d-bfab-ee573c6b54aa-metrics-certs" pod "18755224-59d2-427d-bfab-ee573c6b54aa" (UID: "18755224-59d2-427d-bfab-ee573c6b54aa"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:36:54.876002 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:54.875907    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18755224-59d2-427d-bfab-ee573c6b54aa-kube-api-access-s8kvc" pod "18755224-59d2-427d-bfab-ee573c6b54aa" (UID: "18755224-59d2-427d-bfab-ee573c6b54aa"). InnerVolumeSpecName "kube-api-access-s8kvc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:36:54.973629 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:54.973593    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8kvc\" (UniqueName: \"kubernetes.io/projected/18755224-59d2-427d-bfab-ee573c6b54aa-kube-api-access-s8kvc\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:36:54.973629 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:54.973625    5060 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18755224-59d2-427d-bfab-ee573c6b54aa-metrics-certs\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:36:55.596515 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:55.596449    5060 generic.go:356] "Generic (PLEG): container finished" podID="18755224-59d2-427d-bfab-ee573c6b54aa" containerID="e8b8dbe67572e2aa8d7134d86790deeea1e3f279eb79577467817037c81e54c9" exitCode=0
Oct 31 12:36:55.596935 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:55.596523    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-697b76b867-cfz9w"
Oct 31 12:36:55.596935 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:55.596539    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-697b76b867-cfz9w" event={"ID":"18755224-59d2-427d-bfab-ee573c6b54aa","Type":"ContainerDied","Data":"e8b8dbe67572e2aa8d7134d86790deeea1e3f279eb79577467817037c81e54c9"}
Oct 31 12:36:55.596935 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:55.596598    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-697b76b867-cfz9w" event={"ID":"18755224-59d2-427d-bfab-ee573c6b54aa","Type":"ContainerDied","Data":"03fb0f04ad17a61b653775e611141a3d540b2bdc89045e0598d73b0497d13506"}
Oct 31 12:36:55.596935 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:55.596620    5060 scope.go:122] "RemoveContainer" containerID="e8b8dbe67572e2aa8d7134d86790deeea1e3f279eb79577467817037c81e54c9"
Oct 31 12:36:55.606034 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:55.606017    5060 scope.go:122] "RemoveContainer" containerID="e8b8dbe67572e2aa8d7134d86790deeea1e3f279eb79577467817037c81e54c9"
Oct 31 12:36:55.606289 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:36:55.606267    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8b8dbe67572e2aa8d7134d86790deeea1e3f279eb79577467817037c81e54c9\": container with ID starting with e8b8dbe67572e2aa8d7134d86790deeea1e3f279eb79577467817037c81e54c9 not found: ID does not exist" containerID="e8b8dbe67572e2aa8d7134d86790deeea1e3f279eb79577467817037c81e54c9"
Oct 31 12:36:55.606363 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:55.606297    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8b8dbe67572e2aa8d7134d86790deeea1e3f279eb79577467817037c81e54c9"} err="failed to get container status \"e8b8dbe67572e2aa8d7134d86790deeea1e3f279eb79577467817037c81e54c9\": rpc error: code = NotFound desc = could not find container \"e8b8dbe67572e2aa8d7134d86790deeea1e3f279eb79577467817037c81e54c9\": container with ID starting with e8b8dbe67572e2aa8d7134d86790deeea1e3f279eb79577467817037c81e54c9 not found: ID does not exist"
Oct 31 12:36:55.614571 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:55.614549    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-697b76b867-cfz9w"]
Oct 31 12:36:55.617042 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:55.617022    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-697b76b867-cfz9w"]
Oct 31 12:36:57.047450 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:36:57.047419    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18755224-59d2-427d-bfab-ee573c6b54aa" path="/var/lib/kubelet/pods/18755224-59d2-427d-bfab-ee573c6b54aa/volumes"
Oct 31 12:36:59.920728 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:36:59.920653    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7phb" containerName="extractor"
Oct 31 12:37:17.245232 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:17.245192    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pcvvm_f032909d-bc39-4c9e-99b7-f3d800842119/console-operator/2.log"
Oct 31 12:37:17.262424 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:17.262398    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pcvvm_f032909d-bc39-4c9e-99b7-f3d800842119/console-operator/2.log"
Oct 31 12:37:21.433966 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:21.433930    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wnh97"]
Oct 31 12:37:21.434602 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:21.434581    5060 cpu_manager.go:414] "containerMap: removing container" podUID="18755224-59d2-427d-bfab-ee573c6b54aa" containerName="manager"
Oct 31 12:37:21.434602 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:21.434602    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18755224-59d2-427d-bfab-ee573c6b54aa" containerName="manager"
Oct 31 12:37:21.434791 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:21.434727    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="18755224-59d2-427d-bfab-ee573c6b54aa" containerName="manager"
Oct 31 12:37:21.446549 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:21.446519    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wnh97"
Oct 31 12:37:21.455991 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:21.455961    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-fb0343-openshift-default-dockercfg-pktb6\""
Oct 31 12:37:21.459973 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:21.459945    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wnh97"]
Oct 31 12:37:21.615530 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:21.615482    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/330523c8-06c9-42a2-ac3f-9dd4ad62c392-workload-socket\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wnh97\" (UID: \"330523c8-06c9-42a2-ac3f-9dd4ad62c392\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wnh97"
Oct 31 12:37:21.615695 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:21.615591    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/330523c8-06c9-42a2-ac3f-9dd4ad62c392-istio-podinfo\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wnh97\" (UID: \"330523c8-06c9-42a2-ac3f-9dd4ad62c392\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wnh97"
Oct 31 12:37:21.615695 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:21.615671    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/330523c8-06c9-42a2-ac3f-9dd4ad62c392-istio-data\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wnh97\" (UID: \"330523c8-06c9-42a2-ac3f-9dd4ad62c392\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wnh97"
Oct 31 12:37:21.615793 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:21.615720    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sm4q5\" (UniqueName: \"kubernetes.io/projected/330523c8-06c9-42a2-ac3f-9dd4ad62c392-kube-api-access-sm4q5\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wnh97\" (UID: \"330523c8-06c9-42a2-ac3f-9dd4ad62c392\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wnh97"
Oct 31 12:37:21.615793 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:21.615772    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/330523c8-06c9-42a2-ac3f-9dd4ad62c392-workload-certs\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wnh97\" (UID: \"330523c8-06c9-42a2-ac3f-9dd4ad62c392\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wnh97"
Oct 31 12:37:21.615895 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:21.615795    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/330523c8-06c9-42a2-ac3f-9dd4ad62c392-istiod-ca-cert\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wnh97\" (UID: \"330523c8-06c9-42a2-ac3f-9dd4ad62c392\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wnh97"
Oct 31 12:37:21.615895 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:21.615815    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/330523c8-06c9-42a2-ac3f-9dd4ad62c392-istio-envoy\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wnh97\" (UID: \"330523c8-06c9-42a2-ac3f-9dd4ad62c392\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wnh97"
Oct 31 12:37:21.616002 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:21.615960    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/330523c8-06c9-42a2-ac3f-9dd4ad62c392-istio-token\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wnh97\" (UID: \"330523c8-06c9-42a2-ac3f-9dd4ad62c392\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wnh97"
Oct 31 12:37:21.616063 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:21.616042    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/330523c8-06c9-42a2-ac3f-9dd4ad62c392-credential-socket\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wnh97\" (UID: \"330523c8-06c9-42a2-ac3f-9dd4ad62c392\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wnh97"
Oct 31 12:37:21.717667 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:21.717560    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/330523c8-06c9-42a2-ac3f-9dd4ad62c392-istio-token\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wnh97\" (UID: \"330523c8-06c9-42a2-ac3f-9dd4ad62c392\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wnh97"
Oct 31 12:37:21.717667 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:21.717633    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/330523c8-06c9-42a2-ac3f-9dd4ad62c392-credential-socket\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wnh97\" (UID: \"330523c8-06c9-42a2-ac3f-9dd4ad62c392\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wnh97"
Oct 31 12:37:21.717887 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:21.717676    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/330523c8-06c9-42a2-ac3f-9dd4ad62c392-workload-socket\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wnh97\" (UID: \"330523c8-06c9-42a2-ac3f-9dd4ad62c392\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wnh97"
Oct 31 12:37:21.717887 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:21.717706    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/330523c8-06c9-42a2-ac3f-9dd4ad62c392-istio-podinfo\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wnh97\" (UID: \"330523c8-06c9-42a2-ac3f-9dd4ad62c392\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wnh97"
Oct 31 12:37:21.717887 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:21.717765    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/330523c8-06c9-42a2-ac3f-9dd4ad62c392-istio-data\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wnh97\" (UID: \"330523c8-06c9-42a2-ac3f-9dd4ad62c392\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wnh97"
Oct 31 12:37:21.717887 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:21.717842    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sm4q5\" (UniqueName: \"kubernetes.io/projected/330523c8-06c9-42a2-ac3f-9dd4ad62c392-kube-api-access-sm4q5\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wnh97\" (UID: \"330523c8-06c9-42a2-ac3f-9dd4ad62c392\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wnh97"
Oct 31 12:37:21.717887 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:21.717882    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/330523c8-06c9-42a2-ac3f-9dd4ad62c392-workload-certs\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wnh97\" (UID: \"330523c8-06c9-42a2-ac3f-9dd4ad62c392\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wnh97"
Oct 31 12:37:21.718083 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:21.717920    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/330523c8-06c9-42a2-ac3f-9dd4ad62c392-istiod-ca-cert\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wnh97\" (UID: \"330523c8-06c9-42a2-ac3f-9dd4ad62c392\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wnh97"
Oct 31 12:37:21.718083 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:21.717939    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/330523c8-06c9-42a2-ac3f-9dd4ad62c392-istio-envoy\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wnh97\" (UID: \"330523c8-06c9-42a2-ac3f-9dd4ad62c392\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wnh97"
Oct 31 12:37:21.718722 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:21.718695    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/330523c8-06c9-42a2-ac3f-9dd4ad62c392-workload-certs\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wnh97\" (UID: \"330523c8-06c9-42a2-ac3f-9dd4ad62c392\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wnh97"
Oct 31 12:37:21.718917 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:21.718889    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/330523c8-06c9-42a2-ac3f-9dd4ad62c392-istio-data\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wnh97\" (UID: \"330523c8-06c9-42a2-ac3f-9dd4ad62c392\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wnh97"
Oct 31 12:37:21.719916 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:21.719583    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/330523c8-06c9-42a2-ac3f-9dd4ad62c392-istiod-ca-cert\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wnh97\" (UID: \"330523c8-06c9-42a2-ac3f-9dd4ad62c392\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wnh97"
Oct 31 12:37:21.720039 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:21.719607    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/330523c8-06c9-42a2-ac3f-9dd4ad62c392-workload-socket\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wnh97\" (UID: \"330523c8-06c9-42a2-ac3f-9dd4ad62c392\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wnh97"
Oct 31 12:37:21.720039 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:21.719853    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/330523c8-06c9-42a2-ac3f-9dd4ad62c392-credential-socket\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wnh97\" (UID: \"330523c8-06c9-42a2-ac3f-9dd4ad62c392\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wnh97"
Oct 31 12:37:21.720924 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:21.720897    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/330523c8-06c9-42a2-ac3f-9dd4ad62c392-istio-envoy\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wnh97\" (UID: \"330523c8-06c9-42a2-ac3f-9dd4ad62c392\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wnh97"
Oct 31 12:37:21.721088 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:21.721064    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/330523c8-06c9-42a2-ac3f-9dd4ad62c392-istio-podinfo\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wnh97\" (UID: \"330523c8-06c9-42a2-ac3f-9dd4ad62c392\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wnh97"
Oct 31 12:37:21.750859 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:21.750825    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/330523c8-06c9-42a2-ac3f-9dd4ad62c392-istio-token\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wnh97\" (UID: \"330523c8-06c9-42a2-ac3f-9dd4ad62c392\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wnh97"
Oct 31 12:37:21.763808 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:21.763779    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sm4q5\" (UniqueName: \"kubernetes.io/projected/330523c8-06c9-42a2-ac3f-9dd4ad62c392-kube-api-access-sm4q5\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wnh97\" (UID: \"330523c8-06c9-42a2-ac3f-9dd4ad62c392\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wnh97"
Oct 31 12:37:21.831452 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:21.831418    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-jf8bm"]
Oct 31 12:37:21.836542 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:21.836521    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-jf8bm"
Oct 31 12:37:21.847556 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:21.847521    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-b218d3-openshift-default-dockercfg-kz5nw\""
Oct 31 12:37:21.853638 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:21.853613    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-jf8bm"]
Oct 31 12:37:21.920744 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:21.920695    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-986cp\" (UniqueName: \"kubernetes.io/projected/02360407-0dfa-45b7-b276-6487dcfb02f0-kube-api-access-986cp\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-jf8bm\" (UID: \"02360407-0dfa-45b7-b276-6487dcfb02f0\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-jf8bm"
Oct 31 12:37:21.920931 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:21.920748    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02360407-0dfa-45b7-b276-6487dcfb02f0-workload-certs\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-jf8bm\" (UID: \"02360407-0dfa-45b7-b276-6487dcfb02f0\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-jf8bm"
Oct 31 12:37:21.920931 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:21.920867    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02360407-0dfa-45b7-b276-6487dcfb02f0-istio-envoy\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-jf8bm\" (UID: \"02360407-0dfa-45b7-b276-6487dcfb02f0\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-jf8bm"
Oct 31 12:37:21.920931 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:21.920904    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02360407-0dfa-45b7-b276-6487dcfb02f0-istiod-ca-cert\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-jf8bm\" (UID: \"02360407-0dfa-45b7-b276-6487dcfb02f0\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-jf8bm"
Oct 31 12:37:21.921160 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:21.920963    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02360407-0dfa-45b7-b276-6487dcfb02f0-istio-token\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-jf8bm\" (UID: \"02360407-0dfa-45b7-b276-6487dcfb02f0\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-jf8bm"
Oct 31 12:37:21.921160 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:21.921059    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02360407-0dfa-45b7-b276-6487dcfb02f0-istio-data\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-jf8bm\" (UID: \"02360407-0dfa-45b7-b276-6487dcfb02f0\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-jf8bm"
Oct 31 12:37:21.921160 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:21.921086    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02360407-0dfa-45b7-b276-6487dcfb02f0-workload-socket\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-jf8bm\" (UID: \"02360407-0dfa-45b7-b276-6487dcfb02f0\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-jf8bm"
Oct 31 12:37:21.921160 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:21.921114    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02360407-0dfa-45b7-b276-6487dcfb02f0-credential-socket\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-jf8bm\" (UID: \"02360407-0dfa-45b7-b276-6487dcfb02f0\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-jf8bm"
Oct 31 12:37:21.921363 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:21.921164    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02360407-0dfa-45b7-b276-6487dcfb02f0-istio-podinfo\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-jf8bm\" (UID: \"02360407-0dfa-45b7-b276-6487dcfb02f0\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-jf8bm"
Oct 31 12:37:22.021867 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:22.021780    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-986cp\" (UniqueName: \"kubernetes.io/projected/02360407-0dfa-45b7-b276-6487dcfb02f0-kube-api-access-986cp\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-jf8bm\" (UID: \"02360407-0dfa-45b7-b276-6487dcfb02f0\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-jf8bm"
Oct 31 12:37:22.021867 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:22.021823    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02360407-0dfa-45b7-b276-6487dcfb02f0-workload-certs\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-jf8bm\" (UID: \"02360407-0dfa-45b7-b276-6487dcfb02f0\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-jf8bm"
Oct 31 12:37:22.022095 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:22.021869    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02360407-0dfa-45b7-b276-6487dcfb02f0-istio-envoy\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-jf8bm\" (UID: \"02360407-0dfa-45b7-b276-6487dcfb02f0\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-jf8bm"
Oct 31 12:37:22.022095 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:22.021889    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02360407-0dfa-45b7-b276-6487dcfb02f0-istiod-ca-cert\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-jf8bm\" (UID: \"02360407-0dfa-45b7-b276-6487dcfb02f0\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-jf8bm"
Oct 31 12:37:22.022095 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:22.021946    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02360407-0dfa-45b7-b276-6487dcfb02f0-istio-token\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-jf8bm\" (UID: \"02360407-0dfa-45b7-b276-6487dcfb02f0\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-jf8bm"
Oct 31 12:37:22.022095 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:22.022036    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02360407-0dfa-45b7-b276-6487dcfb02f0-istio-data\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-jf8bm\" (UID: \"02360407-0dfa-45b7-b276-6487dcfb02f0\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-jf8bm"
Oct 31 12:37:22.022095 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:22.022059    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02360407-0dfa-45b7-b276-6487dcfb02f0-workload-socket\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-jf8bm\" (UID: \"02360407-0dfa-45b7-b276-6487dcfb02f0\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-jf8bm"
Oct 31 12:37:22.022095 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:22.022089    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02360407-0dfa-45b7-b276-6487dcfb02f0-credential-socket\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-jf8bm\" (UID: \"02360407-0dfa-45b7-b276-6487dcfb02f0\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-jf8bm"
Oct 31 12:37:22.022392 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:22.022138    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02360407-0dfa-45b7-b276-6487dcfb02f0-istio-podinfo\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-jf8bm\" (UID: \"02360407-0dfa-45b7-b276-6487dcfb02f0\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-jf8bm"
Oct 31 12:37:22.022392 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:22.022219    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02360407-0dfa-45b7-b276-6487dcfb02f0-workload-certs\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-jf8bm\" (UID: \"02360407-0dfa-45b7-b276-6487dcfb02f0\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-jf8bm"
Oct 31 12:37:22.022539 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:22.022518    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02360407-0dfa-45b7-b276-6487dcfb02f0-credential-socket\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-jf8bm\" (UID: \"02360407-0dfa-45b7-b276-6487dcfb02f0\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-jf8bm"
Oct 31 12:37:22.022684 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:22.022665    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02360407-0dfa-45b7-b276-6487dcfb02f0-istiod-ca-cert\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-jf8bm\" (UID: \"02360407-0dfa-45b7-b276-6487dcfb02f0\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-jf8bm"
Oct 31 12:37:22.022792 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:22.022778    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02360407-0dfa-45b7-b276-6487dcfb02f0-istio-data\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-jf8bm\" (UID: \"02360407-0dfa-45b7-b276-6487dcfb02f0\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-jf8bm"
Oct 31 12:37:22.022961 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:22.022945    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02360407-0dfa-45b7-b276-6487dcfb02f0-workload-socket\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-jf8bm\" (UID: \"02360407-0dfa-45b7-b276-6487dcfb02f0\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-jf8bm"
Oct 31 12:37:22.024409 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:22.024391    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02360407-0dfa-45b7-b276-6487dcfb02f0-istio-envoy\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-jf8bm\" (UID: \"02360407-0dfa-45b7-b276-6487dcfb02f0\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-jf8bm"
Oct 31 12:37:22.024871 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:22.024853    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02360407-0dfa-45b7-b276-6487dcfb02f0-istio-podinfo\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-jf8bm\" (UID: \"02360407-0dfa-45b7-b276-6487dcfb02f0\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-jf8bm"
Oct 31 12:37:22.039260 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:22.039236    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02360407-0dfa-45b7-b276-6487dcfb02f0-istio-token\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-jf8bm\" (UID: \"02360407-0dfa-45b7-b276-6487dcfb02f0\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-jf8bm"
Oct 31 12:37:22.040712 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:22.040525    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-986cp\" (UniqueName: \"kubernetes.io/projected/02360407-0dfa-45b7-b276-6487dcfb02f0-kube-api-access-986cp\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-jf8bm\" (UID: \"02360407-0dfa-45b7-b276-6487dcfb02f0\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-jf8bm"
Oct 31 12:37:22.055018 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:22.054989    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-58cq4"]
Oct 31 12:37:22.060021 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:22.060001    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-58cq4"
Oct 31 12:37:22.061674 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:22.061650    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wnh97"
Oct 31 12:37:22.071279 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:22.070943    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-85ce08-openshift-default-dockercfg-5nn7z\""
Oct 31 12:37:22.077786 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:22.075997    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-58cq4"]
Oct 31 12:37:22.151807 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:22.151439    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-jf8bm"
Oct 31 12:37:22.225769 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:22.223995    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c86d1b8b-6bc6-45ff-9412-179f069cf080-credential-socket\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-58cq4\" (UID: \"c86d1b8b-6bc6-45ff-9412-179f069cf080\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-58cq4"
Oct 31 12:37:22.225769 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:22.224070    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c86d1b8b-6bc6-45ff-9412-179f069cf080-workload-certs\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-58cq4\" (UID: \"c86d1b8b-6bc6-45ff-9412-179f069cf080\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-58cq4"
Oct 31 12:37:22.225769 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:22.224110    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c86d1b8b-6bc6-45ff-9412-179f069cf080-istio-token\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-58cq4\" (UID: \"c86d1b8b-6bc6-45ff-9412-179f069cf080\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-58cq4"
Oct 31 12:37:22.225769 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:22.224140    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c86d1b8b-6bc6-45ff-9412-179f069cf080-workload-socket\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-58cq4\" (UID: \"c86d1b8b-6bc6-45ff-9412-179f069cf080\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-58cq4"
Oct 31 12:37:22.225769 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:22.224191    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c86d1b8b-6bc6-45ff-9412-179f069cf080-istiod-ca-cert\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-58cq4\" (UID: \"c86d1b8b-6bc6-45ff-9412-179f069cf080\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-58cq4"
Oct 31 12:37:22.226402 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:22.226118    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c86d1b8b-6bc6-45ff-9412-179f069cf080-istio-podinfo\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-58cq4\" (UID: \"c86d1b8b-6bc6-45ff-9412-179f069cf080\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-58cq4"
Oct 31 12:37:22.226402 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:22.226178    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dnth\" (UniqueName: \"kubernetes.io/projected/c86d1b8b-6bc6-45ff-9412-179f069cf080-kube-api-access-4dnth\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-58cq4\" (UID: \"c86d1b8b-6bc6-45ff-9412-179f069cf080\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-58cq4"
Oct 31 12:37:22.226402 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:22.226214    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c86d1b8b-6bc6-45ff-9412-179f069cf080-istio-envoy\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-58cq4\" (UID: \"c86d1b8b-6bc6-45ff-9412-179f069cf080\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-58cq4"
Oct 31 12:37:22.226402 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:22.226262    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c86d1b8b-6bc6-45ff-9412-179f069cf080-istio-data\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-58cq4\" (UID: \"c86d1b8b-6bc6-45ff-9412-179f069cf080\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-58cq4"
Oct 31 12:37:22.327756 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:22.327714    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c86d1b8b-6bc6-45ff-9412-179f069cf080-istio-podinfo\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-58cq4\" (UID: \"c86d1b8b-6bc6-45ff-9412-179f069cf080\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-58cq4"
Oct 31 12:37:22.327934 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:22.327783    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dnth\" (UniqueName: \"kubernetes.io/projected/c86d1b8b-6bc6-45ff-9412-179f069cf080-kube-api-access-4dnth\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-58cq4\" (UID: \"c86d1b8b-6bc6-45ff-9412-179f069cf080\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-58cq4"
Oct 31 12:37:22.327934 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:22.327820    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c86d1b8b-6bc6-45ff-9412-179f069cf080-istio-envoy\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-58cq4\" (UID: \"c86d1b8b-6bc6-45ff-9412-179f069cf080\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-58cq4"
Oct 31 12:37:22.327934 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:22.327869    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c86d1b8b-6bc6-45ff-9412-179f069cf080-istio-data\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-58cq4\" (UID: \"c86d1b8b-6bc6-45ff-9412-179f069cf080\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-58cq4"
Oct 31 12:37:22.328045 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:22.327951    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c86d1b8b-6bc6-45ff-9412-179f069cf080-credential-socket\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-58cq4\" (UID: \"c86d1b8b-6bc6-45ff-9412-179f069cf080\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-58cq4"
Oct 31 12:37:22.328045 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:22.327999    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c86d1b8b-6bc6-45ff-9412-179f069cf080-workload-certs\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-58cq4\" (UID: \"c86d1b8b-6bc6-45ff-9412-179f069cf080\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-58cq4"
Oct 31 12:37:22.328122 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:22.328048    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c86d1b8b-6bc6-45ff-9412-179f069cf080-istio-token\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-58cq4\" (UID: \"c86d1b8b-6bc6-45ff-9412-179f069cf080\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-58cq4"
Oct 31 12:37:22.328122 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:22.328083    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c86d1b8b-6bc6-45ff-9412-179f069cf080-workload-socket\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-58cq4\" (UID: \"c86d1b8b-6bc6-45ff-9412-179f069cf080\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-58cq4"
Oct 31 12:37:22.328218 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:22.328125    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c86d1b8b-6bc6-45ff-9412-179f069cf080-istiod-ca-cert\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-58cq4\" (UID: \"c86d1b8b-6bc6-45ff-9412-179f069cf080\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-58cq4"
Oct 31 12:37:22.328828 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:22.328679    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c86d1b8b-6bc6-45ff-9412-179f069cf080-credential-socket\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-58cq4\" (UID: \"c86d1b8b-6bc6-45ff-9412-179f069cf080\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-58cq4"
Oct 31 12:37:22.328828 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:22.328713    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c86d1b8b-6bc6-45ff-9412-179f069cf080-istio-data\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-58cq4\" (UID: \"c86d1b8b-6bc6-45ff-9412-179f069cf080\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-58cq4"
Oct 31 12:37:22.328828 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:22.328748    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c86d1b8b-6bc6-45ff-9412-179f069cf080-workload-certs\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-58cq4\" (UID: \"c86d1b8b-6bc6-45ff-9412-179f069cf080\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-58cq4"
Oct 31 12:37:22.328828 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:22.328780    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c86d1b8b-6bc6-45ff-9412-179f069cf080-workload-socket\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-58cq4\" (UID: \"c86d1b8b-6bc6-45ff-9412-179f069cf080\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-58cq4"
Oct 31 12:37:22.329132 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:22.328935    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c86d1b8b-6bc6-45ff-9412-179f069cf080-istiod-ca-cert\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-58cq4\" (UID: \"c86d1b8b-6bc6-45ff-9412-179f069cf080\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-58cq4"
Oct 31 12:37:22.330610 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:22.330556    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c86d1b8b-6bc6-45ff-9412-179f069cf080-istio-podinfo\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-58cq4\" (UID: \"c86d1b8b-6bc6-45ff-9412-179f069cf080\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-58cq4"
Oct 31 12:37:22.330610 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:22.330580    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c86d1b8b-6bc6-45ff-9412-179f069cf080-istio-envoy\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-58cq4\" (UID: \"c86d1b8b-6bc6-45ff-9412-179f069cf080\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-58cq4"
Oct 31 12:37:22.344028 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:22.344003    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dnth\" (UniqueName: \"kubernetes.io/projected/c86d1b8b-6bc6-45ff-9412-179f069cf080-kube-api-access-4dnth\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-58cq4\" (UID: \"c86d1b8b-6bc6-45ff-9412-179f069cf080\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-58cq4"
Oct 31 12:37:22.348847 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:22.348824    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c86d1b8b-6bc6-45ff-9412-179f069cf080-istio-token\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-58cq4\" (UID: \"c86d1b8b-6bc6-45ff-9412-179f069cf080\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-58cq4"
Oct 31 12:37:22.364876 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:22.364854    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wnh97"]
Oct 31 12:37:22.367830 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:37:22.367799    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod330523c8_06c9_42a2_ac3f_9dd4ad62c392.slice/crio-6b069694c4651774f5a12305bb7b29c2a2d5e20b54b370ae4d8fe13bc2ca6ad4 WatchSource:0}: Error finding container 6b069694c4651774f5a12305bb7b29c2a2d5e20b54b370ae4d8fe13bc2ca6ad4: Status 404 returned error can't find the container with id 6b069694c4651774f5a12305bb7b29c2a2d5e20b54b370ae4d8fe13bc2ca6ad4
Oct 31 12:37:22.370757 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:22.370724    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:37:22.370835 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:22.370789    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:37:22.370835 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:22.370823    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:37:22.389821 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:22.389798    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-58cq4"
Oct 31 12:37:22.467748 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:22.462454    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-jf8bm"]
Oct 31 12:37:22.477821 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:22.477769    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:37:22.478106 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:22.478033    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:37:22.478275 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:22.478212    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:37:22.693665 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:22.693635    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-58cq4"]
Oct 31 12:37:22.695705 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:37:22.695674    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc86d1b8b_6bc6_45ff_9412_179f069cf080.slice/crio-c1b4131e812b90c5f160ed0d22334ecdf4a1e19946f47559cbc11c099f90472c WatchSource:0}: Error finding container c1b4131e812b90c5f160ed0d22334ecdf4a1e19946f47559cbc11c099f90472c: Status 404 returned error can't find the container with id c1b4131e812b90c5f160ed0d22334ecdf4a1e19946f47559cbc11c099f90472c
Oct 31 12:37:22.698943 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:22.698902    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:37:22.699046 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:22.699014    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:37:22.699110 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:22.699078    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:37:22.763335 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:22.762524    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-jf8bm" event={"ID":"02360407-0dfa-45b7-b276-6487dcfb02f0","Type":"ContainerStarted","Data":"bf1be783bcbba3e5a2ef5eb2d7ebf98c19e54cbde409381b7e1695dd214e1156"}
Oct 31 12:37:22.763335 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:22.762578    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-jf8bm" event={"ID":"02360407-0dfa-45b7-b276-6487dcfb02f0","Type":"ContainerStarted","Data":"2a012caa6d497a800ee0a3dee28d50bf49ef23cc7a94e6a9c4e43a3b864e055d"}
Oct 31 12:37:22.765272 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:22.765235    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wnh97" event={"ID":"330523c8-06c9-42a2-ac3f-9dd4ad62c392","Type":"ContainerStarted","Data":"c81f960cbd8694792bbef5bd454b1dcd679f0054ccc4a589d9e6a72eb1ed5557"}
Oct 31 12:37:22.765409 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:22.765295    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wnh97" event={"ID":"330523c8-06c9-42a2-ac3f-9dd4ad62c392","Type":"ContainerStarted","Data":"6b069694c4651774f5a12305bb7b29c2a2d5e20b54b370ae4d8fe13bc2ca6ad4"}
Oct 31 12:37:22.767942 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:22.767881    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-58cq4" event={"ID":"c86d1b8b-6bc6-45ff-9412-179f069cf080","Type":"ContainerStarted","Data":"c1b4131e812b90c5f160ed0d22334ecdf4a1e19946f47559cbc11c099f90472c"}
Oct 31 12:37:22.790375 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:22.790326    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-jf8bm" podStartSLOduration=1.790311564 podStartE2EDuration="1.790311564s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:37:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:37:22.788353811 +0000 UTC m=+605.891421549" watchObservedRunningTime="2026-10-31 12:37:22.790311564 +0000 UTC m=+605.893379273"
Oct 31 12:37:22.817742 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:22.817688    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wnh97" podStartSLOduration=1.817674272 podStartE2EDuration="1.817674272s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:37:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:37:22.816012499 +0000 UTC m=+605.919080209" watchObservedRunningTime="2026-10-31 12:37:22.817674272 +0000 UTC m=+605.920741985"
Oct 31 12:37:23.064307 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:23.062455    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wnh97"
Oct 31 12:37:23.073320 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:23.073290    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wnh97"
Oct 31 12:37:23.152555 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:23.152518    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-jf8bm"
Oct 31 12:37:23.157801 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:23.157774    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-jf8bm"
Oct 31 12:37:23.775954 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:23.775919    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-58cq4" event={"ID":"c86d1b8b-6bc6-45ff-9412-179f069cf080","Type":"ContainerStarted","Data":"c5d4d07861968414635e12412ade308cb520be5914e7ae0cad43698ac25e5dda"}
Oct 31 12:37:23.776441 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:23.776408    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-jf8bm"
Oct 31 12:37:23.776544 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:23.776451    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wnh97"
Oct 31 12:37:23.777155 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:23.777130    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-jf8bm"
Oct 31 12:37:23.777415 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:23.777398    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wnh97"
Oct 31 12:37:23.801951 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:23.801896    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-58cq4" podStartSLOduration=1.801882965 podStartE2EDuration="1.801882965s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:37:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:37:23.799628703 +0000 UTC m=+606.902696616" watchObservedRunningTime="2026-10-31 12:37:23.801882965 +0000 UTC m=+606.904950677"
Oct 31 12:37:24.390259 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:24.390217    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-58cq4"
Oct 31 12:37:24.396006 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:24.395978    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-58cq4"
Oct 31 12:37:24.783091 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:24.783049    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-58cq4"
Oct 31 12:37:24.783784 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:24.783767    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-58cq4"
Oct 31 12:37:26.419886 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:26.419854    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-fb0343-5d8d4b7fb7-dtkzd"]
Oct 31 12:37:26.423900 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:26.423883    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-fb0343-5d8d4b7fb7-dtkzd"
Oct 31 12:37:26.429591 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:26.429567    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-fb0343-85b486979f-blth7"]
Oct 31 12:37:26.432795 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:26.432775    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-fb0343\""
Oct 31 12:37:26.433302 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:26.433283    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-fb0343-85b486979f-blth7"
Oct 31 12:37:26.450187 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:26.450162    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-fb0343-5d8d4b7fb7-dtkzd"]
Oct 31 12:37:26.454874 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:26.454848    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-fb0343-85b486979f-blth7"]
Oct 31 12:37:26.455001 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:26.454971    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-fb0343-dockercfg-6f865\""
Oct 31 12:37:26.478444 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:26.478411    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f8db300-b095-4b06-9fa5-1804c2a64df8-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-fb0343-85b486979f-blth7\" (UID: \"5f8db300-b095-4b06-9fa5-1804c2a64df8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-fb0343-85b486979f-blth7"
Oct 31 12:37:26.478628 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:26.478456    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a041d29-7b54-4d07-be06-2c96f8fc097c-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-fb0343-5d8d4b7fb7-dtkzd\" (UID: \"6a041d29-7b54-4d07-be06-2c96f8fc097c\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-fb0343-5d8d4b7fb7-dtkzd"
Oct 31 12:37:26.478628 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:26.478592    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4g4k\" (UniqueName: \"kubernetes.io/projected/5f8db300-b095-4b06-9fa5-1804c2a64df8-kube-api-access-s4g4k\") pod \"payload-pre-processing-e2e-worker-w0-fb0343-85b486979f-blth7\" (UID: \"5f8db300-b095-4b06-9fa5-1804c2a64df8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-fb0343-85b486979f-blth7"
Oct 31 12:37:26.478742 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:26.478639    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drbkt\" (UniqueName: \"kubernetes.io/projected/6a041d29-7b54-4d07-be06-2c96f8fc097c-kube-api-access-drbkt\") pod \"payload-processing-e2e-worker-w0-fb0343-5d8d4b7fb7-dtkzd\" (UID: \"6a041d29-7b54-4d07-be06-2c96f8fc097c\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-fb0343-5d8d4b7fb7-dtkzd"
Oct 31 12:37:26.579339 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:26.579295    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f8db300-b095-4b06-9fa5-1804c2a64df8-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-fb0343-85b486979f-blth7\" (UID: \"5f8db300-b095-4b06-9fa5-1804c2a64df8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-fb0343-85b486979f-blth7"
Oct 31 12:37:26.579339 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:26.579338    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a041d29-7b54-4d07-be06-2c96f8fc097c-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-fb0343-5d8d4b7fb7-dtkzd\" (UID: \"6a041d29-7b54-4d07-be06-2c96f8fc097c\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-fb0343-5d8d4b7fb7-dtkzd"
Oct 31 12:37:26.579586 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:26.579374    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4g4k\" (UniqueName: \"kubernetes.io/projected/5f8db300-b095-4b06-9fa5-1804c2a64df8-kube-api-access-s4g4k\") pod \"payload-pre-processing-e2e-worker-w0-fb0343-85b486979f-blth7\" (UID: \"5f8db300-b095-4b06-9fa5-1804c2a64df8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-fb0343-85b486979f-blth7"
Oct 31 12:37:26.579586 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:26.579413    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drbkt\" (UniqueName: \"kubernetes.io/projected/6a041d29-7b54-4d07-be06-2c96f8fc097c-kube-api-access-drbkt\") pod \"payload-processing-e2e-worker-w0-fb0343-5d8d4b7fb7-dtkzd\" (UID: \"6a041d29-7b54-4d07-be06-2c96f8fc097c\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-fb0343-5d8d4b7fb7-dtkzd"
Oct 31 12:37:26.580145 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:26.580124    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a041d29-7b54-4d07-be06-2c96f8fc097c-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-fb0343-5d8d4b7fb7-dtkzd\" (UID: \"6a041d29-7b54-4d07-be06-2c96f8fc097c\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-fb0343-5d8d4b7fb7-dtkzd"
Oct 31 12:37:26.580199 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:26.580143    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f8db300-b095-4b06-9fa5-1804c2a64df8-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-fb0343-85b486979f-blth7\" (UID: \"5f8db300-b095-4b06-9fa5-1804c2a64df8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-fb0343-85b486979f-blth7"
Oct 31 12:37:26.595994 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:26.595973    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4g4k\" (UniqueName: \"kubernetes.io/projected/5f8db300-b095-4b06-9fa5-1804c2a64df8-kube-api-access-s4g4k\") pod \"payload-pre-processing-e2e-worker-w0-fb0343-85b486979f-blth7\" (UID: \"5f8db300-b095-4b06-9fa5-1804c2a64df8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-fb0343-85b486979f-blth7"
Oct 31 12:37:26.595994 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:26.595988    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drbkt\" (UniqueName: \"kubernetes.io/projected/6a041d29-7b54-4d07-be06-2c96f8fc097c-kube-api-access-drbkt\") pod \"payload-processing-e2e-worker-w0-fb0343-5d8d4b7fb7-dtkzd\" (UID: \"6a041d29-7b54-4d07-be06-2c96f8fc097c\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-fb0343-5d8d4b7fb7-dtkzd"
Oct 31 12:37:26.739125 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:26.739035    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-fb0343-5d8d4b7fb7-dtkzd"
Oct 31 12:37:26.747306 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:26.747276    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-fb0343-85b486979f-blth7"
Oct 31 12:37:27.024123 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:27.024097    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-fb0343-5d8d4b7fb7-dtkzd"]
Oct 31 12:37:27.026085 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:37:27.026056    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a041d29_7b54_4d07_be06_2c96f8fc097c.slice/crio-2e64981718a5452ee7cf6828574cff1103b1a8ad00b2f36732201b956f8541e1 WatchSource:0}: Error finding container 2e64981718a5452ee7cf6828574cff1103b1a8ad00b2f36732201b956f8541e1: Status 404 returned error can't find the container with id 2e64981718a5452ee7cf6828574cff1103b1a8ad00b2f36732201b956f8541e1
Oct 31 12:37:27.040386 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:27.040362    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-fb0343-85b486979f-blth7"]
Oct 31 12:37:27.042061 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:37:27.041871    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f8db300_b095_4b06_9fa5_1804c2a64df8.slice/crio-73eb3427f1dbbb16373ba8b90a77f301f6b2659bbafc377770768c044bb357c7 WatchSource:0}: Error finding container 73eb3427f1dbbb16373ba8b90a77f301f6b2659bbafc377770768c044bb357c7: Status 404 returned error can't find the container with id 73eb3427f1dbbb16373ba8b90a77f301f6b2659bbafc377770768c044bb357c7
Oct 31 12:37:27.261735 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:27.261697    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-67d410-5c9cd96795-tvtnh"]
Oct 31 12:37:27.265636 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:27.265613    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-67d410-5c9cd96795-tvtnh"
Oct 31 12:37:27.271899 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:27.271873    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w6-67d410-dockercfg-vbghv\""
Oct 31 12:37:27.276159 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:27.276093    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w6-67d410\""
Oct 31 12:37:27.278998 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:27.278976    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-67d410-5c9cd96795-tvtnh"]
Oct 31 12:37:27.285580 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:27.285555    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4swg\" (UniqueName: \"kubernetes.io/projected/6772255c-c420-435e-8075-f52344d30bdb-kube-api-access-t4swg\") pod \"payload-pre-processing-e2e-worker-w6-67d410-5c9cd96795-tvtnh\" (UID: \"6772255c-c420-435e-8075-f52344d30bdb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-67d410-5c9cd96795-tvtnh"
Oct 31 12:37:27.285697 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:27.285676    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6772255c-c420-435e-8075-f52344d30bdb-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-67d410-5c9cd96795-tvtnh\" (UID: \"6772255c-c420-435e-8075-f52344d30bdb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-67d410-5c9cd96795-tvtnh"
Oct 31 12:37:27.304998 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:27.304964    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-67d410-6c74c4dc47-k6m8r"]
Oct 31 12:37:27.309901 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:27.309869    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-67d410-6c74c4dc47-k6m8r"
Oct 31 12:37:27.336538 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:27.336507    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-67d410-6c74c4dc47-k6m8r"]
Oct 31 12:37:27.386567 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:27.386532    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6772255c-c420-435e-8075-f52344d30bdb-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-67d410-5c9cd96795-tvtnh\" (UID: \"6772255c-c420-435e-8075-f52344d30bdb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-67d410-5c9cd96795-tvtnh"
Oct 31 12:37:27.386720 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:27.386576    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f85a5783-97ad-4058-bcdc-0981cea23afd-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-67d410-6c74c4dc47-k6m8r\" (UID: \"f85a5783-97ad-4058-bcdc-0981cea23afd\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-67d410-6c74c4dc47-k6m8r"
Oct 31 12:37:27.386720 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:27.386615    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5l9kj\" (UniqueName: \"kubernetes.io/projected/f85a5783-97ad-4058-bcdc-0981cea23afd-kube-api-access-5l9kj\") pod \"payload-processing-e2e-worker-w6-67d410-6c74c4dc47-k6m8r\" (UID: \"f85a5783-97ad-4058-bcdc-0981cea23afd\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-67d410-6c74c4dc47-k6m8r"
Oct 31 12:37:27.386799 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:27.386757    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4swg\" (UniqueName: \"kubernetes.io/projected/6772255c-c420-435e-8075-f52344d30bdb-kube-api-access-t4swg\") pod \"payload-pre-processing-e2e-worker-w6-67d410-5c9cd96795-tvtnh\" (UID: \"6772255c-c420-435e-8075-f52344d30bdb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-67d410-5c9cd96795-tvtnh"
Oct 31 12:37:27.387217 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:27.387198    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6772255c-c420-435e-8075-f52344d30bdb-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-67d410-5c9cd96795-tvtnh\" (UID: \"6772255c-c420-435e-8075-f52344d30bdb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-67d410-5c9cd96795-tvtnh"
Oct 31 12:37:27.402474 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:27.402446    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4swg\" (UniqueName: \"kubernetes.io/projected/6772255c-c420-435e-8075-f52344d30bdb-kube-api-access-t4swg\") pod \"payload-pre-processing-e2e-worker-w6-67d410-5c9cd96795-tvtnh\" (UID: \"6772255c-c420-435e-8075-f52344d30bdb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-67d410-5c9cd96795-tvtnh"
Oct 31 12:37:27.487590 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:27.487554    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f85a5783-97ad-4058-bcdc-0981cea23afd-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-67d410-6c74c4dc47-k6m8r\" (UID: \"f85a5783-97ad-4058-bcdc-0981cea23afd\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-67d410-6c74c4dc47-k6m8r"
Oct 31 12:37:27.487590 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:27.487602    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5l9kj\" (UniqueName: \"kubernetes.io/projected/f85a5783-97ad-4058-bcdc-0981cea23afd-kube-api-access-5l9kj\") pod \"payload-processing-e2e-worker-w6-67d410-6c74c4dc47-k6m8r\" (UID: \"f85a5783-97ad-4058-bcdc-0981cea23afd\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-67d410-6c74c4dc47-k6m8r"
Oct 31 12:37:27.488262 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:27.488240    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f85a5783-97ad-4058-bcdc-0981cea23afd-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-67d410-6c74c4dc47-k6m8r\" (UID: \"f85a5783-97ad-4058-bcdc-0981cea23afd\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-67d410-6c74c4dc47-k6m8r"
Oct 31 12:37:27.518560 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:27.518533    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5l9kj\" (UniqueName: \"kubernetes.io/projected/f85a5783-97ad-4058-bcdc-0981cea23afd-kube-api-access-5l9kj\") pod \"payload-processing-e2e-worker-w6-67d410-6c74c4dc47-k6m8r\" (UID: \"f85a5783-97ad-4058-bcdc-0981cea23afd\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-67d410-6c74c4dc47-k6m8r"
Oct 31 12:37:27.580172 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:27.580095    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-67d410-5c9cd96795-tvtnh"
Oct 31 12:37:27.625555 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:27.625507    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-67d410-6c74c4dc47-k6m8r"
Oct 31 12:37:27.838021 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:27.837857    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-fb0343-85b486979f-blth7" event={"ID":"5f8db300-b095-4b06-9fa5-1804c2a64df8","Type":"ContainerStarted","Data":"7e07cca37a3d92c5eee144bc04bab214a169ce484dfa655084775f7a977435bc"}
Oct 31 12:37:27.838021 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:27.837942    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-fb0343-85b486979f-blth7" event={"ID":"5f8db300-b095-4b06-9fa5-1804c2a64df8","Type":"ContainerStarted","Data":"73eb3427f1dbbb16373ba8b90a77f301f6b2659bbafc377770768c044bb357c7"}
Oct 31 12:37:27.838254 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:27.838080    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-fb0343-85b486979f-blth7"
Oct 31 12:37:27.839547 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:27.839522    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-fb0343-5d8d4b7fb7-dtkzd" event={"ID":"6a041d29-7b54-4d07-be06-2c96f8fc097c","Type":"ContainerStarted","Data":"481c99373b0e0f398f783a1ef91a8e7c1944fa271ed2428a6fb1c0632dde60c2"}
Oct 31 12:37:27.839695 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:27.839569    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-fb0343-5d8d4b7fb7-dtkzd" event={"ID":"6a041d29-7b54-4d07-be06-2c96f8fc097c","Type":"ContainerStarted","Data":"2e64981718a5452ee7cf6828574cff1103b1a8ad00b2f36732201b956f8541e1"}
Oct 31 12:37:27.839695 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:27.839598    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-fb0343-5d8d4b7fb7-dtkzd"
Oct 31 12:37:27.934944 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:27.934913    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-67d410-5c9cd96795-tvtnh"]
Oct 31 12:37:27.936031 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:37:27.936001    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6772255c_c420_435e_8075_f52344d30bdb.slice/crio-a2e547320249af02a43494d9d9904dbe1587e0bc756591b2e0bdc5b5263b5072 WatchSource:0}: Error finding container a2e547320249af02a43494d9d9904dbe1587e0bc756591b2e0bdc5b5263b5072: Status 404 returned error can't find the container with id a2e547320249af02a43494d9d9904dbe1587e0bc756591b2e0bdc5b5263b5072
Oct 31 12:37:27.964705 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:27.964653    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-fb0343-85b486979f-blth7" podStartSLOduration=1.964637217 podStartE2EDuration="1.964637217s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:37:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:37:27.95994619 +0000 UTC m=+611.063013902" watchObservedRunningTime="2026-10-31 12:37:27.964637217 +0000 UTC m=+611.067705327"
Oct 31 12:37:27.996456 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:27.996430    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-67d410-6c74c4dc47-k6m8r"]
Oct 31 12:37:27.996571 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:37:27.996478    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf85a5783_97ad_4058_bcdc_0981cea23afd.slice/crio-1519ac1298dcc41c7d47fde810ef9ef87c09a8d685ebf6cff36a40a611b0f056 WatchSource:0}: Error finding container 1519ac1298dcc41c7d47fde810ef9ef87c09a8d685ebf6cff36a40a611b0f056: Status 404 returned error can't find the container with id 1519ac1298dcc41c7d47fde810ef9ef87c09a8d685ebf6cff36a40a611b0f056
Oct 31 12:37:28.035269 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:28.035225    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-fb0343-5d8d4b7fb7-dtkzd" podStartSLOduration=2.035210338 podStartE2EDuration="2.035210338s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:37:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:37:28.034752285 +0000 UTC m=+611.137820000" watchObservedRunningTime="2026-10-31 12:37:28.035210338 +0000 UTC m=+611.138278050"
Oct 31 12:37:28.525031 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:28.524995    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-c67de2-7d5dc5649c-62vgx"]
Oct 31 12:37:28.528831 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:28.528812    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-c67de2-7d5dc5649c-62vgx"
Oct 31 12:37:28.541270 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:28.541246    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-c67de2-7d5dc5649c-62vgx"]
Oct 31 12:37:28.555773 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:28.555729    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-c67de2-5656c49759-8llqd"]
Oct 31 12:37:28.559606 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:28.559584    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c67de2-5656c49759-8llqd"
Oct 31 12:37:28.574515 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:28.572234    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-c67de2-5656c49759-8llqd"]
Oct 31 12:37:28.590313 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:28.590284    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-c67de2-dockercfg-nz7b7\""
Oct 31 12:37:28.600244 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:28.600212    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkwhk\" (UniqueName: \"kubernetes.io/projected/a6fa2170-8276-4138-9614-551ddc387fb8-kube-api-access-hkwhk\") pod \"payload-processing-e2e-worker-w4-c67de2-7d5dc5649c-62vgx\" (UID: \"a6fa2170-8276-4138-9614-551ddc387fb8\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-c67de2-7d5dc5649c-62vgx"
Oct 31 12:37:28.600338 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:28.600312    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6fa2170-8276-4138-9614-551ddc387fb8-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-c67de2-7d5dc5649c-62vgx\" (UID: \"a6fa2170-8276-4138-9614-551ddc387fb8\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-c67de2-7d5dc5649c-62vgx"
Oct 31 12:37:28.600397 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:28.600382    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d496cda1-2b40-48f1-8d49-a5ecd7f28a2b-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-c67de2-5656c49759-8llqd\" (UID: \"d496cda1-2b40-48f1-8d49-a5ecd7f28a2b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c67de2-5656c49759-8llqd"
Oct 31 12:37:28.600437 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:28.600426    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmng4\" (UniqueName: \"kubernetes.io/projected/d496cda1-2b40-48f1-8d49-a5ecd7f28a2b-kube-api-access-fmng4\") pod \"payload-pre-processing-e2e-worker-w4-c67de2-5656c49759-8llqd\" (UID: \"d496cda1-2b40-48f1-8d49-a5ecd7f28a2b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c67de2-5656c49759-8llqd"
Oct 31 12:37:28.603465 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:28.603441    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-c67de2\""
Oct 31 12:37:28.701037 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:28.700999    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkwhk\" (UniqueName: \"kubernetes.io/projected/a6fa2170-8276-4138-9614-551ddc387fb8-kube-api-access-hkwhk\") pod \"payload-processing-e2e-worker-w4-c67de2-7d5dc5649c-62vgx\" (UID: \"a6fa2170-8276-4138-9614-551ddc387fb8\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-c67de2-7d5dc5649c-62vgx"
Oct 31 12:37:28.701231 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:28.701098    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6fa2170-8276-4138-9614-551ddc387fb8-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-c67de2-7d5dc5649c-62vgx\" (UID: \"a6fa2170-8276-4138-9614-551ddc387fb8\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-c67de2-7d5dc5649c-62vgx"
Oct 31 12:37:28.701231 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:28.701169    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d496cda1-2b40-48f1-8d49-a5ecd7f28a2b-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-c67de2-5656c49759-8llqd\" (UID: \"d496cda1-2b40-48f1-8d49-a5ecd7f28a2b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c67de2-5656c49759-8llqd"
Oct 31 12:37:28.701231 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:28.701208    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmng4\" (UniqueName: \"kubernetes.io/projected/d496cda1-2b40-48f1-8d49-a5ecd7f28a2b-kube-api-access-fmng4\") pod \"payload-pre-processing-e2e-worker-w4-c67de2-5656c49759-8llqd\" (UID: \"d496cda1-2b40-48f1-8d49-a5ecd7f28a2b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c67de2-5656c49759-8llqd"
Oct 31 12:37:28.702058 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:28.702033    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6fa2170-8276-4138-9614-551ddc387fb8-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-c67de2-7d5dc5649c-62vgx\" (UID: \"a6fa2170-8276-4138-9614-551ddc387fb8\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-c67de2-7d5dc5649c-62vgx"
Oct 31 12:37:28.702201 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:28.702060    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d496cda1-2b40-48f1-8d49-a5ecd7f28a2b-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-c67de2-5656c49759-8llqd\" (UID: \"d496cda1-2b40-48f1-8d49-a5ecd7f28a2b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c67de2-5656c49759-8llqd"
Oct 31 12:37:28.716438 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:28.716407    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmng4\" (UniqueName: \"kubernetes.io/projected/d496cda1-2b40-48f1-8d49-a5ecd7f28a2b-kube-api-access-fmng4\") pod \"payload-pre-processing-e2e-worker-w4-c67de2-5656c49759-8llqd\" (UID: \"d496cda1-2b40-48f1-8d49-a5ecd7f28a2b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c67de2-5656c49759-8llqd"
Oct 31 12:37:28.716916 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:28.716895    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkwhk\" (UniqueName: \"kubernetes.io/projected/a6fa2170-8276-4138-9614-551ddc387fb8-kube-api-access-hkwhk\") pod \"payload-processing-e2e-worker-w4-c67de2-7d5dc5649c-62vgx\" (UID: \"a6fa2170-8276-4138-9614-551ddc387fb8\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-c67de2-7d5dc5649c-62vgx"
Oct 31 12:37:28.843249 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:28.843158    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-c67de2-7d5dc5649c-62vgx"
Oct 31 12:37:28.847702 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:28.847670    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-67d410-6c74c4dc47-k6m8r" event={"ID":"f85a5783-97ad-4058-bcdc-0981cea23afd","Type":"ContainerStarted","Data":"4e2808ace3cd5f2e086bd0fae56e55008f333403e24023588a1e7d5238b9c297"}
Oct 31 12:37:28.847859 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:28.847735    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-67d410-6c74c4dc47-k6m8r" event={"ID":"f85a5783-97ad-4058-bcdc-0981cea23afd","Type":"ContainerStarted","Data":"1519ac1298dcc41c7d47fde810ef9ef87c09a8d685ebf6cff36a40a611b0f056"}
Oct 31 12:37:28.847926 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:28.847877    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-67d410-6c74c4dc47-k6m8r"
Oct 31 12:37:28.853730 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:28.853682    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-67d410-5c9cd96795-tvtnh" event={"ID":"6772255c-c420-435e-8075-f52344d30bdb","Type":"ContainerStarted","Data":"40634ad90f586d271990f309270efe61e24804a62b4c02e119264e9dca9287e2"}
Oct 31 12:37:28.853854 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:28.853735    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-67d410-5c9cd96795-tvtnh" event={"ID":"6772255c-c420-435e-8075-f52344d30bdb","Type":"ContainerStarted","Data":"a2e547320249af02a43494d9d9904dbe1587e0bc756591b2e0bdc5b5263b5072"}
Oct 31 12:37:28.854126 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:28.854057    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-67d410-5c9cd96795-tvtnh"
Oct 31 12:37:28.875701 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:28.875668    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c67de2-5656c49759-8llqd"
Oct 31 12:37:28.900634 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:28.900572    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w6-67d410-6c74c4dc47-k6m8r" podStartSLOduration=1.9005505889999998 podStartE2EDuration="1.900550589s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:37:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:37:28.879745399 +0000 UTC m=+611.982813113" watchObservedRunningTime="2026-10-31 12:37:28.900550589 +0000 UTC m=+612.003618305"
Oct 31 12:37:28.911315 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:28.911269    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-67d410-5c9cd96795-tvtnh" podStartSLOduration=1.911254407 podStartE2EDuration="1.911254407s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:37:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:37:28.9078408 +0000 UTC m=+612.010908516" watchObservedRunningTime="2026-10-31 12:37:28.911254407 +0000 UTC m=+612.014322123"
Oct 31 12:37:29.066749 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:29.066712    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-b218d3-766c68ffbf-76pwg"]
Oct 31 12:37:29.072308 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:29.072282    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-b218d3-766c68ffbf-76pwg"
Oct 31 12:37:29.090134 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:29.090104    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-b218d3-766c68ffbf-76pwg"]
Oct 31 12:37:29.105857 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:29.105769    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ec2c204-58e3-4d0f-b06b-5112806add83-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-b218d3-766c68ffbf-76pwg\" (UID: \"2ec2c204-58e3-4d0f-b06b-5112806add83\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-b218d3-766c68ffbf-76pwg"
Oct 31 12:37:29.105857 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:29.105824    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvvj5\" (UniqueName: \"kubernetes.io/projected/2ec2c204-58e3-4d0f-b06b-5112806add83-kube-api-access-dvvj5\") pod \"payload-processing-e2e-worker-w3-b218d3-766c68ffbf-76pwg\" (UID: \"2ec2c204-58e3-4d0f-b06b-5112806add83\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-b218d3-766c68ffbf-76pwg"
Oct 31 12:37:29.137320 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:29.137294    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-b218d3\""
Oct 31 12:37:29.139641 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:29.139624    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-b218d3-dockercfg-zfvtp\""
Oct 31 12:37:29.153242 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:29.153218    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-c67de2-7d5dc5649c-62vgx"]
Oct 31 12:37:29.154998 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:37:29.154972    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6fa2170_8276_4138_9614_551ddc387fb8.slice/crio-89986bb03af6ee0561ab06c858069e984f01d692ec198a9eb25494a4abdfaa77 WatchSource:0}: Error finding container 89986bb03af6ee0561ab06c858069e984f01d692ec198a9eb25494a4abdfaa77: Status 404 returned error can't find the container with id 89986bb03af6ee0561ab06c858069e984f01d692ec198a9eb25494a4abdfaa77
Oct 31 12:37:29.202988 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:29.202965    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-c67de2-5656c49759-8llqd"]
Oct 31 12:37:29.207413 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:29.206877    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ec2c204-58e3-4d0f-b06b-5112806add83-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-b218d3-766c68ffbf-76pwg\" (UID: \"2ec2c204-58e3-4d0f-b06b-5112806add83\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-b218d3-766c68ffbf-76pwg"
Oct 31 12:37:29.207413 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:29.206931    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvvj5\" (UniqueName: \"kubernetes.io/projected/2ec2c204-58e3-4d0f-b06b-5112806add83-kube-api-access-dvvj5\") pod \"payload-processing-e2e-worker-w3-b218d3-766c68ffbf-76pwg\" (UID: \"2ec2c204-58e3-4d0f-b06b-5112806add83\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-b218d3-766c68ffbf-76pwg"
Oct 31 12:37:29.207413 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:37:29.207131    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd496cda1_2b40_48f1_8d49_a5ecd7f28a2b.slice/crio-d4e4ae99f052ec65738dee2ca2efcdada2a49db5aaca6aec9f4ba1fd65d78c04 WatchSource:0}: Error finding container d4e4ae99f052ec65738dee2ca2efcdada2a49db5aaca6aec9f4ba1fd65d78c04: Status 404 returned error can't find the container with id d4e4ae99f052ec65738dee2ca2efcdada2a49db5aaca6aec9f4ba1fd65d78c04
Oct 31 12:37:29.207840 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:29.207777    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ec2c204-58e3-4d0f-b06b-5112806add83-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-b218d3-766c68ffbf-76pwg\" (UID: \"2ec2c204-58e3-4d0f-b06b-5112806add83\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-b218d3-766c68ffbf-76pwg"
Oct 31 12:37:29.222022 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:29.221998    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvvj5\" (UniqueName: \"kubernetes.io/projected/2ec2c204-58e3-4d0f-b06b-5112806add83-kube-api-access-dvvj5\") pod \"payload-processing-e2e-worker-w3-b218d3-766c68ffbf-76pwg\" (UID: \"2ec2c204-58e3-4d0f-b06b-5112806add83\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-b218d3-766c68ffbf-76pwg"
Oct 31 12:37:29.395511 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:29.395432    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-b218d3-766c68ffbf-76pwg"
Oct 31 12:37:29.679262 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:29.679229    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-b218d3-766c68ffbf-76pwg"]
Oct 31 12:37:29.685082 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:37:29.685049    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ec2c204_58e3_4d0f_b06b_5112806add83.slice/crio-1f2dae6f86b7390a2d71e8061033f3cb7898368a436f615cc175f317c437463d WatchSource:0}: Error finding container 1f2dae6f86b7390a2d71e8061033f3cb7898368a436f615cc175f317c437463d: Status 404 returned error can't find the container with id 1f2dae6f86b7390a2d71e8061033f3cb7898368a436f615cc175f317c437463d
Oct 31 12:37:29.864277 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:29.864237    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c67de2-5656c49759-8llqd" event={"ID":"d496cda1-2b40-48f1-8d49-a5ecd7f28a2b","Type":"ContainerStarted","Data":"1e7016a5833af9a9d8fd6407b4e62e48484ca6d04e5d21270de1aa7ff8736173"}
Oct 31 12:37:29.864699 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:29.864303    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c67de2-5656c49759-8llqd" event={"ID":"d496cda1-2b40-48f1-8d49-a5ecd7f28a2b","Type":"ContainerStarted","Data":"d4e4ae99f052ec65738dee2ca2efcdada2a49db5aaca6aec9f4ba1fd65d78c04"}
Oct 31 12:37:29.864699 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:29.864462    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c67de2-5656c49759-8llqd"
Oct 31 12:37:29.866658 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:29.866632    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-c67de2-7d5dc5649c-62vgx" event={"ID":"a6fa2170-8276-4138-9614-551ddc387fb8","Type":"ContainerStarted","Data":"e96b8eab5f9c9c0a8f6acb59152966c8dc7b7f7eda603b139dc8644ab53ab73e"}
Oct 31 12:37:29.866787 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:29.866679    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-c67de2-7d5dc5649c-62vgx" event={"ID":"a6fa2170-8276-4138-9614-551ddc387fb8","Type":"ContainerStarted","Data":"89986bb03af6ee0561ab06c858069e984f01d692ec198a9eb25494a4abdfaa77"}
Oct 31 12:37:29.866787 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:29.866767    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-c67de2-7d5dc5649c-62vgx"
Oct 31 12:37:29.868553 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:29.868530    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-b218d3-766c68ffbf-76pwg" event={"ID":"2ec2c204-58e3-4d0f-b06b-5112806add83","Type":"ContainerStarted","Data":"3990f663560e6baf90c55d56bae73742e4746e910ea63c074d0ad6670e4db361"}
Oct 31 12:37:29.868666 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:29.868570    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-b218d3-766c68ffbf-76pwg" event={"ID":"2ec2c204-58e3-4d0f-b06b-5112806add83","Type":"ContainerStarted","Data":"1f2dae6f86b7390a2d71e8061033f3cb7898368a436f615cc175f317c437463d"}
Oct 31 12:37:29.868666 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:29.868601    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-b218d3-766c68ffbf-76pwg"
Oct 31 12:37:29.885228 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:29.885182    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c67de2-5656c49759-8llqd" podStartSLOduration=1.885170808 podStartE2EDuration="1.885170808s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:37:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:37:29.881607141 +0000 UTC m=+612.984674854" watchObservedRunningTime="2026-10-31 12:37:29.885170808 +0000 UTC m=+612.988238519"
Oct 31 12:37:29.910562 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:29.910503    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-b218d3-766c68ffbf-76pwg" podStartSLOduration=0.910471963 podStartE2EDuration="910.471963ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:37:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:37:29.9102415 +0000 UTC m=+613.013309217" watchObservedRunningTime="2026-10-31 12:37:29.910471963 +0000 UTC m=+613.013539675"
Oct 31 12:37:29.932030 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:37:29.931949    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7phb" containerName="extractor"
Oct 31 12:37:29.936322 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:29.936276    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-c67de2-7d5dc5649c-62vgx" podStartSLOduration=1.936257339 podStartE2EDuration="1.936257339s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:37:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:37:29.934509817 +0000 UTC m=+613.037577526" watchObservedRunningTime="2026-10-31 12:37:29.936257339 +0000 UTC m=+613.039325052"
Oct 31 12:37:30.113450 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:30.113320    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-b25d3b-66f84f5d8f-6w4n2"]
Oct 31 12:37:30.117504 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:30.117474    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-b25d3b-66f84f5d8f-6w4n2"
Oct 31 12:37:30.130005 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:30.129983    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-b25d3b-66f84f5d8f-6w4n2"]
Oct 31 12:37:30.206930 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:30.206784    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w5-b25d3b\""
Oct 31 12:37:30.206930 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:30.206829    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w5-b25d3b-dockercfg-g2knq\""
Oct 31 12:37:30.216984 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:30.216948    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85cd1407-54f9-493a-b8fd-9e638a6615a5-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-b25d3b-66f84f5d8f-6w4n2\" (UID: \"85cd1407-54f9-493a-b8fd-9e638a6615a5\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-b25d3b-66f84f5d8f-6w4n2"
Oct 31 12:37:30.217167 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:30.217073    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9bzr\" (UniqueName: \"kubernetes.io/projected/85cd1407-54f9-493a-b8fd-9e638a6615a5-kube-api-access-c9bzr\") pod \"payload-processing-e2e-worker-w5-b25d3b-66f84f5d8f-6w4n2\" (UID: \"85cd1407-54f9-493a-b8fd-9e638a6615a5\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-b25d3b-66f84f5d8f-6w4n2"
Oct 31 12:37:30.318295 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:30.318259    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85cd1407-54f9-493a-b8fd-9e638a6615a5-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-b25d3b-66f84f5d8f-6w4n2\" (UID: \"85cd1407-54f9-493a-b8fd-9e638a6615a5\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-b25d3b-66f84f5d8f-6w4n2"
Oct 31 12:37:30.318508 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:30.318331    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9bzr\" (UniqueName: \"kubernetes.io/projected/85cd1407-54f9-493a-b8fd-9e638a6615a5-kube-api-access-c9bzr\") pod \"payload-processing-e2e-worker-w5-b25d3b-66f84f5d8f-6w4n2\" (UID: \"85cd1407-54f9-493a-b8fd-9e638a6615a5\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-b25d3b-66f84f5d8f-6w4n2"
Oct 31 12:37:30.318970 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:30.318946    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85cd1407-54f9-493a-b8fd-9e638a6615a5-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-b25d3b-66f84f5d8f-6w4n2\" (UID: \"85cd1407-54f9-493a-b8fd-9e638a6615a5\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-b25d3b-66f84f5d8f-6w4n2"
Oct 31 12:37:30.330747 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:30.330720    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9bzr\" (UniqueName: \"kubernetes.io/projected/85cd1407-54f9-493a-b8fd-9e638a6615a5-kube-api-access-c9bzr\") pod \"payload-processing-e2e-worker-w5-b25d3b-66f84f5d8f-6w4n2\" (UID: \"85cd1407-54f9-493a-b8fd-9e638a6615a5\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-b25d3b-66f84f5d8f-6w4n2"
Oct 31 12:37:30.433640 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:30.433604    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-b25d3b-66f84f5d8f-6w4n2"
Oct 31 12:37:30.715549 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:30.715522    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-b25d3b-66f84f5d8f-6w4n2"]
Oct 31 12:37:30.717465 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:37:30.717434    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85cd1407_54f9_493a_b8fd_9e638a6615a5.slice/crio-7fb845f41e3876323691e0e1bb5ed767c63209ca39d22d360f9ecedf915c3c49 WatchSource:0}: Error finding container 7fb845f41e3876323691e0e1bb5ed767c63209ca39d22d360f9ecedf915c3c49: Status 404 returned error can't find the container with id 7fb845f41e3876323691e0e1bb5ed767c63209ca39d22d360f9ecedf915c3c49
Oct 31 12:37:30.754219 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:30.754184    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-e6c79d-7f6dbd5c9-psn2f"]
Oct 31 12:37:30.761616 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:30.761594    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-e6c79d-7f6dbd5c9-psn2f"
Oct 31 12:37:30.781502 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:30.781439    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-e6c79d\""
Oct 31 12:37:30.781652 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:30.781549    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-e6c79d-dockercfg-t25fz\""
Oct 31 12:37:30.785854 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:30.785482    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-e6c79d-7f6dbd5c9-psn2f"]
Oct 31 12:37:30.822826 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:30.822800    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f6d17e0-1d51-41b8-a52c-a386a8fdcc17-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-e6c79d-7f6dbd5c9-psn2f\" (UID: \"3f6d17e0-1d51-41b8-a52c-a386a8fdcc17\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-e6c79d-7f6dbd5c9-psn2f"
Oct 31 12:37:30.822949 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:30.822884    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4zkd\" (UniqueName: \"kubernetes.io/projected/3f6d17e0-1d51-41b8-a52c-a386a8fdcc17-kube-api-access-h4zkd\") pod \"payload-processing-e2e-worker-w2-e6c79d-7f6dbd5c9-psn2f\" (UID: \"3f6d17e0-1d51-41b8-a52c-a386a8fdcc17\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-e6c79d-7f6dbd5c9-psn2f"
Oct 31 12:37:30.882479 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:30.882375    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-b25d3b-66f84f5d8f-6w4n2" event={"ID":"85cd1407-54f9-493a-b8fd-9e638a6615a5","Type":"ContainerStarted","Data":"751646ab827e81ab96b1b3b08200b865727aa93d32279c669f803a16eb4d57ba"}
Oct 31 12:37:30.882743 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:30.882717    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-b25d3b-66f84f5d8f-6w4n2" event={"ID":"85cd1407-54f9-493a-b8fd-9e638a6615a5","Type":"ContainerStarted","Data":"7fb845f41e3876323691e0e1bb5ed767c63209ca39d22d360f9ecedf915c3c49"}
Oct 31 12:37:30.883859 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:30.883750    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-b25d3b-66f84f5d8f-6w4n2"
Oct 31 12:37:30.913990 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:30.913883    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w5-b25d3b-66f84f5d8f-6w4n2" podStartSLOduration=0.913858866 podStartE2EDuration="913.858866ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:37:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:37:30.909662681 +0000 UTC m=+614.012730394" watchObservedRunningTime="2026-10-31 12:37:30.913858866 +0000 UTC m=+614.016926575"
Oct 31 12:37:30.924469 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:30.924436    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4zkd\" (UniqueName: \"kubernetes.io/projected/3f6d17e0-1d51-41b8-a52c-a386a8fdcc17-kube-api-access-h4zkd\") pod \"payload-processing-e2e-worker-w2-e6c79d-7f6dbd5c9-psn2f\" (UID: \"3f6d17e0-1d51-41b8-a52c-a386a8fdcc17\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-e6c79d-7f6dbd5c9-psn2f"
Oct 31 12:37:30.924803 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:30.924774    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f6d17e0-1d51-41b8-a52c-a386a8fdcc17-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-e6c79d-7f6dbd5c9-psn2f\" (UID: \"3f6d17e0-1d51-41b8-a52c-a386a8fdcc17\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-e6c79d-7f6dbd5c9-psn2f"
Oct 31 12:37:30.925579 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:30.925549    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f6d17e0-1d51-41b8-a52c-a386a8fdcc17-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-e6c79d-7f6dbd5c9-psn2f\" (UID: \"3f6d17e0-1d51-41b8-a52c-a386a8fdcc17\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-e6c79d-7f6dbd5c9-psn2f"
Oct 31 12:37:30.939925 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:30.939895    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4zkd\" (UniqueName: \"kubernetes.io/projected/3f6d17e0-1d51-41b8-a52c-a386a8fdcc17-kube-api-access-h4zkd\") pod \"payload-processing-e2e-worker-w2-e6c79d-7f6dbd5c9-psn2f\" (UID: \"3f6d17e0-1d51-41b8-a52c-a386a8fdcc17\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-e6c79d-7f6dbd5c9-psn2f"
Oct 31 12:37:31.102442 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:31.102407    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-e6c79d-7f6dbd5c9-psn2f"
Oct 31 12:37:31.306514 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:31.292391    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-85ce08-5f656fc7f9-z5g9p"]
Oct 31 12:37:31.312668 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:31.312641    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-85ce08-5f656fc7f9-z5g9p"]
Oct 31 12:37:31.312952 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:31.312938    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-85ce08-5f656fc7f9-z5g9p"
Oct 31 12:37:31.341728 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:31.341691    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmkqf\" (UniqueName: \"kubernetes.io/projected/b03c9dc3-2388-458b-8e7f-09cf2811021f-kube-api-access-xmkqf\") pod \"payload-processing-e2e-worker-w1-85ce08-5f656fc7f9-z5g9p\" (UID: \"b03c9dc3-2388-458b-8e7f-09cf2811021f\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-85ce08-5f656fc7f9-z5g9p"
Oct 31 12:37:31.341884 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:31.341801    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b03c9dc3-2388-458b-8e7f-09cf2811021f-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-85ce08-5f656fc7f9-z5g9p\" (UID: \"b03c9dc3-2388-458b-8e7f-09cf2811021f\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-85ce08-5f656fc7f9-z5g9p"
Oct 31 12:37:31.382034 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:31.381950    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-85ce08\""
Oct 31 12:37:31.391541 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:31.391067    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-85ce08-dockercfg-m7ndj\""
Oct 31 12:37:31.411626 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:31.411593    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-e6c79d-7f6dbd5c9-psn2f"]
Oct 31 12:37:31.413362 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:37:31.413330    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f6d17e0_1d51_41b8_a52c_a386a8fdcc17.slice/crio-a909bf9bf2d1d13edc96f405b65ec759bcffa0bf05968dffe732368ca47c0420 WatchSource:0}: Error finding container a909bf9bf2d1d13edc96f405b65ec759bcffa0bf05968dffe732368ca47c0420: Status 404 returned error can't find the container with id a909bf9bf2d1d13edc96f405b65ec759bcffa0bf05968dffe732368ca47c0420
Oct 31 12:37:31.442554 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:31.442474    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b03c9dc3-2388-458b-8e7f-09cf2811021f-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-85ce08-5f656fc7f9-z5g9p\" (UID: \"b03c9dc3-2388-458b-8e7f-09cf2811021f\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-85ce08-5f656fc7f9-z5g9p"
Oct 31 12:37:31.442709 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:31.442674    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmkqf\" (UniqueName: \"kubernetes.io/projected/b03c9dc3-2388-458b-8e7f-09cf2811021f-kube-api-access-xmkqf\") pod \"payload-processing-e2e-worker-w1-85ce08-5f656fc7f9-z5g9p\" (UID: \"b03c9dc3-2388-458b-8e7f-09cf2811021f\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-85ce08-5f656fc7f9-z5g9p"
Oct 31 12:37:31.443302 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:31.443281    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b03c9dc3-2388-458b-8e7f-09cf2811021f-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-85ce08-5f656fc7f9-z5g9p\" (UID: \"b03c9dc3-2388-458b-8e7f-09cf2811021f\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-85ce08-5f656fc7f9-z5g9p"
Oct 31 12:37:31.460386 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:31.460356    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmkqf\" (UniqueName: \"kubernetes.io/projected/b03c9dc3-2388-458b-8e7f-09cf2811021f-kube-api-access-xmkqf\") pod \"payload-processing-e2e-worker-w1-85ce08-5f656fc7f9-z5g9p\" (UID: \"b03c9dc3-2388-458b-8e7f-09cf2811021f\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-85ce08-5f656fc7f9-z5g9p"
Oct 31 12:37:31.645157 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:31.645065    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-85ce08-5f656fc7f9-z5g9p"
Oct 31 12:37:31.889823 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:31.889785    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-e6c79d-7f6dbd5c9-psn2f" event={"ID":"3f6d17e0-1d51-41b8-a52c-a386a8fdcc17","Type":"ContainerStarted","Data":"3e3c21381670c8a6131d9ad5ce41c83a85369b90f0088941ea202ce7538723a7"}
Oct 31 12:37:31.890304 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:31.889846    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-e6c79d-7f6dbd5c9-psn2f" event={"ID":"3f6d17e0-1d51-41b8-a52c-a386a8fdcc17","Type":"ContainerStarted","Data":"a909bf9bf2d1d13edc96f405b65ec759bcffa0bf05968dffe732368ca47c0420"}
Oct 31 12:37:31.890304 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:31.889876    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-e6c79d-7f6dbd5c9-psn2f"
Oct 31 12:37:31.914993 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:31.914884    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-e6c79d-7f6dbd5c9-psn2f" podStartSLOduration=1.914864481 podStartE2EDuration="1.914864481s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:37:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:37:31.911932153 +0000 UTC m=+615.014999878" watchObservedRunningTime="2026-10-31 12:37:31.914864481 +0000 UTC m=+615.017932194"
Oct 31 12:37:31.932020 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:31.931995    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-85ce08-5f656fc7f9-z5g9p"]
Oct 31 12:37:31.933463 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:37:31.933437    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb03c9dc3_2388_458b_8e7f_09cf2811021f.slice/crio-4c5d05924f4d06fac07adcf6f71af0b2c6d4ece9becd17018b77cfe8205859c9 WatchSource:0}: Error finding container 4c5d05924f4d06fac07adcf6f71af0b2c6d4ece9becd17018b77cfe8205859c9: Status 404 returned error can't find the container with id 4c5d05924f4d06fac07adcf6f71af0b2c6d4ece9becd17018b77cfe8205859c9
Oct 31 12:37:32.899459 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:32.899412    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-85ce08-5f656fc7f9-z5g9p" event={"ID":"b03c9dc3-2388-458b-8e7f-09cf2811021f","Type":"ContainerStarted","Data":"b0a736c77932c00d5497c4435970c017fd800e61c497ebcfb4c5d9fc653ab315"}
Oct 31 12:37:32.899949 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:32.899881    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-85ce08-5f656fc7f9-z5g9p"
Oct 31 12:37:32.899949 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:32.899917    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-85ce08-5f656fc7f9-z5g9p" event={"ID":"b03c9dc3-2388-458b-8e7f-09cf2811021f","Type":"ContainerStarted","Data":"4c5d05924f4d06fac07adcf6f71af0b2c6d4ece9becd17018b77cfe8205859c9"}
Oct 31 12:37:32.918084 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:32.918034    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-85ce08-5f656fc7f9-z5g9p" podStartSLOduration=1.918018987 podStartE2EDuration="1.918018987s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:37:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:37:32.916952212 +0000 UTC m=+616.020019927" watchObservedRunningTime="2026-10-31 12:37:32.918018987 +0000 UTC m=+616.021086723"
Oct 31 12:37:38.876523 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:38.875651    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-fb0343-5d8d4b7fb7-dtkzd"
Oct 31 12:37:38.877019 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:38.876655    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-fb0343-85b486979f-blth7"
Oct 31 12:37:39.828946 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:39.828906    5060 patch_prober.go:29] interesting pod/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wnh97 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 12:37:39.829205 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:39.828993    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wnh97" podUID="330523c8-06c9-42a2-ac3f-9dd4ad62c392" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:37:39.888682 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:39.888646    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-67d410-5c9cd96795-tvtnh"
Oct 31 12:37:39.889343 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:39.889320    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-67d410-6c74c4dc47-k6m8r"
Oct 31 12:37:40.889209 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:40.889074    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-c67de2-7d5dc5649c-62vgx"
Oct 31 12:37:40.889768 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:40.889748    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c67de2-5656c49759-8llqd"
Oct 31 12:37:40.890786 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:40.890764    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-b218d3-766c68ffbf-76pwg"
Oct 31 12:37:42.905585 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:42.904542    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-e6c79d-7f6dbd5c9-psn2f"
Oct 31 12:37:42.920123 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:42.920086    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-b25d3b-66f84f5d8f-6w4n2"
Oct 31 12:37:44.050636 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:44.050605    5060 generic.go:356] "Generic (PLEG): container finished" podID="330523c8-06c9-42a2-ac3f-9dd4ad62c392" containerID="c81f960cbd8694792bbef5bd454b1dcd679f0054ccc4a589d9e6a72eb1ed5557" exitCode=0
Oct 31 12:37:44.051021 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:44.050690    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wnh97" event={"ID":"330523c8-06c9-42a2-ac3f-9dd4ad62c392","Type":"ContainerDied","Data":"c81f960cbd8694792bbef5bd454b1dcd679f0054ccc4a589d9e6a72eb1ed5557"}
Oct 31 12:37:44.051168 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:44.051150    5060 scope.go:122] "RemoveContainer" containerID="c81f960cbd8694792bbef5bd454b1dcd679f0054ccc4a589d9e6a72eb1ed5557"
Oct 31 12:37:44.052525 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:44.052464    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:37:44.052653 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:44.052581    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:37:44.052653 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:44.052629    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:37:44.926351 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:44.923323    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-85ce08-5f656fc7f9-z5g9p"
Oct 31 12:37:45.079530 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:45.075063    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wnh97" event={"ID":"330523c8-06c9-42a2-ac3f-9dd4ad62c392","Type":"ContainerStarted","Data":"d190e7520eebe52b0c749ad77fb372857ff3f35218eb3dc8fea952d63280f3d9"}
Oct 31 12:37:46.063221 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:46.063170    5060 patch_prober.go:29] interesting pod/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wnh97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body=
Oct 31 12:37:46.063794 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:46.063759    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wnh97" podUID="330523c8-06c9-42a2-ac3f-9dd4ad62c392" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused"
Oct 31 12:37:46.064109 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:46.062655    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wnh97"
Oct 31 12:37:47.064923 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:47.064883    5060 patch_prober.go:29] interesting pod/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wnh97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body=
Oct 31 12:37:47.065534 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:47.065481    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wnh97" podUID="330523c8-06c9-42a2-ac3f-9dd4ad62c392" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused"
Oct 31 12:37:48.063036 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:48.062995    5060 patch_prober.go:29] interesting pod/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wnh97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body=
Oct 31 12:37:48.063310 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:48.063283    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wnh97" podUID="330523c8-06c9-42a2-ac3f-9dd4ad62c392" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused"
Oct 31 12:37:49.085517 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:49.075944    5060 patch_prober.go:29] interesting pod/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wnh97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body=
Oct 31 12:37:49.085517 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:49.076009    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wnh97" podUID="330523c8-06c9-42a2-ac3f-9dd4ad62c392" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused"
Oct 31 12:37:49.130414 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:49.130383    5060 generic.go:356] "Generic (PLEG): container finished" podID="c86d1b8b-6bc6-45ff-9412-179f069cf080" containerID="c5d4d07861968414635e12412ade308cb520be5914e7ae0cad43698ac25e5dda" exitCode=0
Oct 31 12:37:49.130595 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:49.130432    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-58cq4" event={"ID":"c86d1b8b-6bc6-45ff-9412-179f069cf080","Type":"ContainerDied","Data":"c5d4d07861968414635e12412ade308cb520be5914e7ae0cad43698ac25e5dda"}
Oct 31 12:37:49.130876 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:49.130855    5060 scope.go:122] "RemoveContainer" containerID="c5d4d07861968414635e12412ade308cb520be5914e7ae0cad43698ac25e5dda"
Oct 31 12:37:49.132300 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:49.132268    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:37:49.132374 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:49.132343    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:37:49.132415 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:49.132378    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:37:50.070515 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:50.062912    5060 patch_prober.go:29] interesting pod/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wnh97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body=
Oct 31 12:37:50.070515 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:50.062978    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wnh97" podUID="330523c8-06c9-42a2-ac3f-9dd4ad62c392" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused"
Oct 31 12:37:50.148455 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:50.148418    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-58cq4" event={"ID":"c86d1b8b-6bc6-45ff-9412-179f069cf080","Type":"ContainerStarted","Data":"436dd479231de8934ac874f816c15f4b0d01f071f8d5b37931d737444f7b4523"}
Oct 31 12:37:50.390988 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:50.390843    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-58cq4"
Oct 31 12:37:50.399513 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:50.395551    5060 patch_prober.go:29] interesting pod/e2e-worker-w1-85ce08-openshift-default-77d6458845-58cq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body=
Oct 31 12:37:50.399513 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:50.395630    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-58cq4" podUID="c86d1b8b-6bc6-45ff-9412-179f069cf080" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused"
Oct 31 12:37:51.106858 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:51.106822    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wnh97"
Oct 31 12:37:51.107441 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:51.107385    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wnh97"
Oct 31 12:37:51.108595 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:51.108552    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wnh97"
Oct 31 12:37:51.393091 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:51.392575    5060 patch_prober.go:29] interesting pod/e2e-worker-w1-85ce08-openshift-default-77d6458845-58cq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body=
Oct 31 12:37:51.393091 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:51.392642    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-58cq4" podUID="c86d1b8b-6bc6-45ff-9412-179f069cf080" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused"
Oct 31 12:37:52.390956 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:52.390582    5060 patch_prober.go:29] interesting pod/e2e-worker-w1-85ce08-openshift-default-77d6458845-58cq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body=
Oct 31 12:37:52.390956 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:52.390652    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-58cq4" podUID="c86d1b8b-6bc6-45ff-9412-179f069cf080" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused"
Oct 31 12:37:53.391155 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:53.391115    5060 patch_prober.go:29] interesting pod/e2e-worker-w1-85ce08-openshift-default-77d6458845-58cq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body=
Oct 31 12:37:53.391583 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:53.391182    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-58cq4" podUID="c86d1b8b-6bc6-45ff-9412-179f069cf080" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused"
Oct 31 12:37:53.779245 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:53.778885    5060 patch_prober.go:29] interesting pod/e2e-worker-w3-b218d3-openshift-default-67596dd89f-jf8bm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body=
Oct 31 12:37:53.779245 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:53.778969    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-jf8bm" podUID="02360407-0dfa-45b7-b276-6487dcfb02f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused"
Oct 31 12:37:54.203324 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:54.203240    5060 generic.go:356] "Generic (PLEG): container finished" podID="02360407-0dfa-45b7-b276-6487dcfb02f0" containerID="bf1be783bcbba3e5a2ef5eb2d7ebf98c19e54cbde409381b7e1695dd214e1156" exitCode=0
Oct 31 12:37:54.203472 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:54.203313    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-jf8bm" event={"ID":"02360407-0dfa-45b7-b276-6487dcfb02f0","Type":"ContainerDied","Data":"bf1be783bcbba3e5a2ef5eb2d7ebf98c19e54cbde409381b7e1695dd214e1156"}
Oct 31 12:37:54.203851 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:54.203829    5060 scope.go:122] "RemoveContainer" containerID="bf1be783bcbba3e5a2ef5eb2d7ebf98c19e54cbde409381b7e1695dd214e1156"
Oct 31 12:37:54.205582 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:54.205548    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:37:54.205680 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:54.205638    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:37:54.205726 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:54.205683    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:37:54.392924 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:54.392877    5060 patch_prober.go:29] interesting pod/e2e-worker-w1-85ce08-openshift-default-77d6458845-58cq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body=
Oct 31 12:37:54.393386 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:54.392961    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-58cq4" podUID="c86d1b8b-6bc6-45ff-9412-179f069cf080" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused"
Oct 31 12:37:54.785514 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:54.784595    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-58cq4"
Oct 31 12:37:55.226502 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:55.225567    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-c67de2/simulator-w4-c67de2-kserve-868679cf9b-gqjbx"]
Oct 31 12:37:55.240539 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:55.235354    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-c67de2/simulator-w4-c67de2-kserve-868679cf9b-gqjbx"]
Oct 31 12:37:55.240539 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:55.235461    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c67de2/simulator-w4-c67de2-kserve-868679cf9b-gqjbx"
Oct 31 12:37:55.241815 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:55.241269    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-c67de2\"/\"default-dockercfg-n9qq6\""
Oct 31 12:37:55.241815 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:55.241541    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-c67de2\"/\"openshift-service-ca.crt\""
Oct 31 12:37:55.241815 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:55.241651    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-c67de2\"/\"simulator-w4-c67de2-kserve-self-signed-certs\""
Oct 31 12:37:55.242208 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:55.242142    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-c67de2\"/\"kube-root-ca.crt\""
Oct 31 12:37:55.246440 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:55.245269    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-jf8bm" event={"ID":"02360407-0dfa-45b7-b276-6487dcfb02f0","Type":"ContainerStarted","Data":"7c91c181e45356241d7139f6aeedfaae2dd5b0a8dd32e6953ea1f26c79face5f"}
Oct 31 12:37:55.413549 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:55.406882    5060 patch_prober.go:29] interesting pod/e2e-worker-w1-85ce08-openshift-default-77d6458845-58cq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body=
Oct 31 12:37:55.413549 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:55.406946    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-58cq4" podUID="c86d1b8b-6bc6-45ff-9412-179f069cf080" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused"
Oct 31 12:37:55.426706 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:55.426663    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e104d7da-1fb2-4c99-a480-b4ae84c4f273-tmp-dir\") pod \"simulator-w4-c67de2-kserve-868679cf9b-gqjbx\" (UID: \"e104d7da-1fb2-4c99-a480-b4ae84c4f273\") " pod="e2e-models-e2e-worker-w4-c67de2/simulator-w4-c67de2-kserve-868679cf9b-gqjbx"
Oct 31 12:37:55.426866 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:55.426766    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e104d7da-1fb2-4c99-a480-b4ae84c4f273-home\") pod \"simulator-w4-c67de2-kserve-868679cf9b-gqjbx\" (UID: \"e104d7da-1fb2-4c99-a480-b4ae84c4f273\") " pod="e2e-models-e2e-worker-w4-c67de2/simulator-w4-c67de2-kserve-868679cf9b-gqjbx"
Oct 31 12:37:55.426866 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:55.426801    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e104d7da-1fb2-4c99-a480-b4ae84c4f273-model-cache\") pod \"simulator-w4-c67de2-kserve-868679cf9b-gqjbx\" (UID: \"e104d7da-1fb2-4c99-a480-b4ae84c4f273\") " pod="e2e-models-e2e-worker-w4-c67de2/simulator-w4-c67de2-kserve-868679cf9b-gqjbx"
Oct 31 12:37:55.426944 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:55.426866    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gb4rx\" (UniqueName: \"kubernetes.io/projected/e104d7da-1fb2-4c99-a480-b4ae84c4f273-kube-api-access-gb4rx\") pod \"simulator-w4-c67de2-kserve-868679cf9b-gqjbx\" (UID: \"e104d7da-1fb2-4c99-a480-b4ae84c4f273\") " pod="e2e-models-e2e-worker-w4-c67de2/simulator-w4-c67de2-kserve-868679cf9b-gqjbx"
Oct 31 12:37:55.426996 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:55.426981    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e104d7da-1fb2-4c99-a480-b4ae84c4f273-dshm\") pod \"simulator-w4-c67de2-kserve-868679cf9b-gqjbx\" (UID: \"e104d7da-1fb2-4c99-a480-b4ae84c4f273\") " pod="e2e-models-e2e-worker-w4-c67de2/simulator-w4-c67de2-kserve-868679cf9b-gqjbx"
Oct 31 12:37:55.427102 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:55.427089    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e104d7da-1fb2-4c99-a480-b4ae84c4f273-tls-certs\") pod \"simulator-w4-c67de2-kserve-868679cf9b-gqjbx\" (UID: \"e104d7da-1fb2-4c99-a480-b4ae84c4f273\") " pod="e2e-models-e2e-worker-w4-c67de2/simulator-w4-c67de2-kserve-868679cf9b-gqjbx"
Oct 31 12:37:55.528201 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:55.528160    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e104d7da-1fb2-4c99-a480-b4ae84c4f273-home\") pod \"simulator-w4-c67de2-kserve-868679cf9b-gqjbx\" (UID: \"e104d7da-1fb2-4c99-a480-b4ae84c4f273\") " pod="e2e-models-e2e-worker-w4-c67de2/simulator-w4-c67de2-kserve-868679cf9b-gqjbx"
Oct 31 12:37:55.528391 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:55.528221    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e104d7da-1fb2-4c99-a480-b4ae84c4f273-model-cache\") pod \"simulator-w4-c67de2-kserve-868679cf9b-gqjbx\" (UID: \"e104d7da-1fb2-4c99-a480-b4ae84c4f273\") " pod="e2e-models-e2e-worker-w4-c67de2/simulator-w4-c67de2-kserve-868679cf9b-gqjbx"
Oct 31 12:37:55.528391 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:55.528275    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gb4rx\" (UniqueName: \"kubernetes.io/projected/e104d7da-1fb2-4c99-a480-b4ae84c4f273-kube-api-access-gb4rx\") pod \"simulator-w4-c67de2-kserve-868679cf9b-gqjbx\" (UID: \"e104d7da-1fb2-4c99-a480-b4ae84c4f273\") " pod="e2e-models-e2e-worker-w4-c67de2/simulator-w4-c67de2-kserve-868679cf9b-gqjbx"
Oct 31 12:37:55.528391 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:55.528354    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e104d7da-1fb2-4c99-a480-b4ae84c4f273-dshm\") pod \"simulator-w4-c67de2-kserve-868679cf9b-gqjbx\" (UID: \"e104d7da-1fb2-4c99-a480-b4ae84c4f273\") " pod="e2e-models-e2e-worker-w4-c67de2/simulator-w4-c67de2-kserve-868679cf9b-gqjbx"
Oct 31 12:37:55.528548 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:55.528434    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e104d7da-1fb2-4c99-a480-b4ae84c4f273-tls-certs\") pod \"simulator-w4-c67de2-kserve-868679cf9b-gqjbx\" (UID: \"e104d7da-1fb2-4c99-a480-b4ae84c4f273\") " pod="e2e-models-e2e-worker-w4-c67de2/simulator-w4-c67de2-kserve-868679cf9b-gqjbx"
Oct 31 12:37:55.528548 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:55.528469    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e104d7da-1fb2-4c99-a480-b4ae84c4f273-tmp-dir\") pod \"simulator-w4-c67de2-kserve-868679cf9b-gqjbx\" (UID: \"e104d7da-1fb2-4c99-a480-b4ae84c4f273\") " pod="e2e-models-e2e-worker-w4-c67de2/simulator-w4-c67de2-kserve-868679cf9b-gqjbx"
Oct 31 12:37:55.529005 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:55.528973    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e104d7da-1fb2-4c99-a480-b4ae84c4f273-tmp-dir\") pod \"simulator-w4-c67de2-kserve-868679cf9b-gqjbx\" (UID: \"e104d7da-1fb2-4c99-a480-b4ae84c4f273\") " pod="e2e-models-e2e-worker-w4-c67de2/simulator-w4-c67de2-kserve-868679cf9b-gqjbx"
Oct 31 12:37:55.529255 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:55.529237    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e104d7da-1fb2-4c99-a480-b4ae84c4f273-home\") pod \"simulator-w4-c67de2-kserve-868679cf9b-gqjbx\" (UID: \"e104d7da-1fb2-4c99-a480-b4ae84c4f273\") " pod="e2e-models-e2e-worker-w4-c67de2/simulator-w4-c67de2-kserve-868679cf9b-gqjbx"
Oct 31 12:37:55.529533 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:55.529477    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e104d7da-1fb2-4c99-a480-b4ae84c4f273-model-cache\") pod \"simulator-w4-c67de2-kserve-868679cf9b-gqjbx\" (UID: \"e104d7da-1fb2-4c99-a480-b4ae84c4f273\") " pod="e2e-models-e2e-worker-w4-c67de2/simulator-w4-c67de2-kserve-868679cf9b-gqjbx"
Oct 31 12:37:55.533332 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:55.533306    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e104d7da-1fb2-4c99-a480-b4ae84c4f273-dshm\") pod \"simulator-w4-c67de2-kserve-868679cf9b-gqjbx\" (UID: \"e104d7da-1fb2-4c99-a480-b4ae84c4f273\") " pod="e2e-models-e2e-worker-w4-c67de2/simulator-w4-c67de2-kserve-868679cf9b-gqjbx"
Oct 31 12:37:55.533993 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:55.533974    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e104d7da-1fb2-4c99-a480-b4ae84c4f273-tls-certs\") pod \"simulator-w4-c67de2-kserve-868679cf9b-gqjbx\" (UID: \"e104d7da-1fb2-4c99-a480-b4ae84c4f273\") " pod="e2e-models-e2e-worker-w4-c67de2/simulator-w4-c67de2-kserve-868679cf9b-gqjbx"
Oct 31 12:37:55.541829 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:55.541584    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gb4rx\" (UniqueName: \"kubernetes.io/projected/e104d7da-1fb2-4c99-a480-b4ae84c4f273-kube-api-access-gb4rx\") pod \"simulator-w4-c67de2-kserve-868679cf9b-gqjbx\" (UID: \"e104d7da-1fb2-4c99-a480-b4ae84c4f273\") " pod="e2e-models-e2e-worker-w4-c67de2/simulator-w4-c67de2-kserve-868679cf9b-gqjbx"
Oct 31 12:37:55.561554 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:55.554253    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c67de2/simulator-w4-c67de2-kserve-868679cf9b-gqjbx"
Oct 31 12:37:55.920912 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:55.920828    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-c67de2/simulator-w4-c67de2-kserve-868679cf9b-gqjbx"]
Oct 31 12:37:55.925704 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:37:55.925174    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode104d7da_1fb2_4c99_a480_b4ae84c4f273.slice/crio-4293f1dbc4ec0aecf0e2a3502484912dd2f9c2296523f950c3a5f744c903fc26 WatchSource:0}: Error finding container 4293f1dbc4ec0aecf0e2a3502484912dd2f9c2296523f950c3a5f744c903fc26: Status 404 returned error can't find the container with id 4293f1dbc4ec0aecf0e2a3502484912dd2f9c2296523f950c3a5f744c903fc26
Oct 31 12:37:56.152864 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:56.152824    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-jf8bm"
Oct 31 12:37:56.153096 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:56.153072    5060 patch_prober.go:29] interesting pod/e2e-worker-w3-b218d3-openshift-default-67596dd89f-jf8bm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body=
Oct 31 12:37:56.153146 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:56.153129    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-jf8bm" podUID="02360407-0dfa-45b7-b276-6487dcfb02f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused"
Oct 31 12:37:56.283511 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:56.278245    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c67de2/simulator-w4-c67de2-kserve-868679cf9b-gqjbx" event={"ID":"e104d7da-1fb2-4c99-a480-b4ae84c4f273","Type":"ContainerStarted","Data":"df554570f7de10a33f3fdb5dc1a575a7532829e536a1120d3f6d1e78374fed8d"}
Oct 31 12:37:56.283511 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:56.278296    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c67de2/simulator-w4-c67de2-kserve-868679cf9b-gqjbx" event={"ID":"e104d7da-1fb2-4c99-a480-b4ae84c4f273","Type":"ContainerStarted","Data":"4293f1dbc4ec0aecf0e2a3502484912dd2f9c2296523f950c3a5f744c903fc26"}
Oct 31 12:37:56.312515 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:56.302122    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-c67de2/simulator-w4-c67de2-kserve-868679cf9b-gqjbx" podStartSLOduration=1.302103988 podStartE2EDuration="1.302103988s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:37:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:37:56.296830963 +0000 UTC m=+639.399898688" watchObservedRunningTime="2026-10-31 12:37:56.302103988 +0000 UTC m=+639.405171703"
Oct 31 12:37:56.396317 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:56.396285    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-58cq4"
Oct 31 12:37:56.397443 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:56.397419    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-58cq4"
Oct 31 12:37:56.579675 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:56.579584    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c67de2/simulator-w4-c67de2-kserve-868679cf9b-gqjbx"
Oct 31 12:37:57.153003 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:57.152968    5060 patch_prober.go:29] interesting pod/e2e-worker-w3-b218d3-openshift-default-67596dd89f-jf8bm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body=
Oct 31 12:37:57.153204 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:57.153025    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-jf8bm" podUID="02360407-0dfa-45b7-b276-6487dcfb02f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused"
Oct 31 12:37:58.152077 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:58.152041    5060 patch_prober.go:29] interesting pod/e2e-worker-w3-b218d3-openshift-default-67596dd89f-jf8bm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body=
Oct 31 12:37:58.152471 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:58.152097    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-jf8bm" podUID="02360407-0dfa-45b7-b276-6487dcfb02f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused"
Oct 31 12:37:59.152680 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:59.152636    5060 patch_prober.go:29] interesting pod/e2e-worker-w3-b218d3-openshift-default-67596dd89f-jf8bm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body=
Oct 31 12:37:59.153080 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:37:59.152699    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-jf8bm" podUID="02360407-0dfa-45b7-b276-6487dcfb02f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused"
Oct 31 12:37:59.921315 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:37:59.921278    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7phb" containerName="extractor"
Oct 31 12:38:00.152810 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:38:00.152770    5060 patch_prober.go:29] interesting pod/e2e-worker-w3-b218d3-openshift-default-67596dd89f-jf8bm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body=
Oct 31 12:38:00.153237 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:38:00.152834    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-jf8bm" podUID="02360407-0dfa-45b7-b276-6487dcfb02f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused"
Oct 31 12:38:01.157057 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:38:01.157008    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-jf8bm"
Oct 31 12:38:01.157481 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:38:01.157408    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-jf8bm"
Oct 31 12:38:01.158539 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:38:01.158481    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-jf8bm"
Oct 31 12:38:05.554870 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:38:05.554827    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-c67de2/simulator-w4-c67de2-kserve-868679cf9b-gqjbx"
Oct 31 12:38:05.572813 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:38:05.572782    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-c67de2/simulator-w4-c67de2-kserve-868679cf9b-gqjbx"
Oct 31 12:38:06.375597 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:38:06.375562    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c67de2/simulator-w4-c67de2-kserve-868679cf9b-gqjbx"
Oct 31 12:38:29.921192 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:38:29.921102    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7phb" containerName="extractor"
Oct 31 12:38:37.606562 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:38:37.606520    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-e6c79d/premium-w2-e6c79d-kserve-6468bf46d4-h66f8"]
Oct 31 12:38:37.610708 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:38:37.610682    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e6c79d/premium-w2-e6c79d-kserve-6468bf46d4-h66f8"
Oct 31 12:38:37.616691 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:38:37.616659    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-e6c79d/premium-w2-e6c79d-kserve-6468bf46d4-h66f8"]
Oct 31 12:38:37.617303 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:38:37.617279    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-e6c79d\"/\"kube-root-ca.crt\""
Oct 31 12:38:37.621805 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:38:37.621778    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-e6c79d\"/\"openshift-service-ca.crt\""
Oct 31 12:38:37.625730 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:38:37.625684    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-e6c79d\"/\"default-dockercfg-mscfx\""
Oct 31 12:38:37.625863 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:38:37.625741    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-e6c79d\"/\"premium-w2-e6c79d-kserve-self-signed-certs\""
Oct 31 12:38:37.753515 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:38:37.753104    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd6a266f-8d12-4089-83ee-c75c716bf26c-model-cache\") pod \"premium-w2-e6c79d-kserve-6468bf46d4-h66f8\" (UID: \"cd6a266f-8d12-4089-83ee-c75c716bf26c\") " pod="e2e-models-e2e-worker-w2-e6c79d/premium-w2-e6c79d-kserve-6468bf46d4-h66f8"
Oct 31 12:38:37.753515 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:38:37.753160    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd6a266f-8d12-4089-83ee-c75c716bf26c-home\") pod \"premium-w2-e6c79d-kserve-6468bf46d4-h66f8\" (UID: \"cd6a266f-8d12-4089-83ee-c75c716bf26c\") " pod="e2e-models-e2e-worker-w2-e6c79d/premium-w2-e6c79d-kserve-6468bf46d4-h66f8"
Oct 31 12:38:37.753515 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:38:37.753244    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd6a266f-8d12-4089-83ee-c75c716bf26c-tls-certs\") pod \"premium-w2-e6c79d-kserve-6468bf46d4-h66f8\" (UID: \"cd6a266f-8d12-4089-83ee-c75c716bf26c\") " pod="e2e-models-e2e-worker-w2-e6c79d/premium-w2-e6c79d-kserve-6468bf46d4-h66f8"
Oct 31 12:38:37.753515 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:38:37.753297    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jnzw\" (UniqueName: \"kubernetes.io/projected/cd6a266f-8d12-4089-83ee-c75c716bf26c-kube-api-access-7jnzw\") pod \"premium-w2-e6c79d-kserve-6468bf46d4-h66f8\" (UID: \"cd6a266f-8d12-4089-83ee-c75c716bf26c\") " pod="e2e-models-e2e-worker-w2-e6c79d/premium-w2-e6c79d-kserve-6468bf46d4-h66f8"
Oct 31 12:38:37.753515 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:38:37.753367    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd6a266f-8d12-4089-83ee-c75c716bf26c-tmp-dir\") pod \"premium-w2-e6c79d-kserve-6468bf46d4-h66f8\" (UID: \"cd6a266f-8d12-4089-83ee-c75c716bf26c\") " pod="e2e-models-e2e-worker-w2-e6c79d/premium-w2-e6c79d-kserve-6468bf46d4-h66f8"
Oct 31 12:38:37.753515 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:38:37.753390    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd6a266f-8d12-4089-83ee-c75c716bf26c-dshm\") pod \"premium-w2-e6c79d-kserve-6468bf46d4-h66f8\" (UID: \"cd6a266f-8d12-4089-83ee-c75c716bf26c\") " pod="e2e-models-e2e-worker-w2-e6c79d/premium-w2-e6c79d-kserve-6468bf46d4-h66f8"
Oct 31 12:38:37.862155 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:38:37.862064    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd6a266f-8d12-4089-83ee-c75c716bf26c-tls-certs\") pod \"premium-w2-e6c79d-kserve-6468bf46d4-h66f8\" (UID: \"cd6a266f-8d12-4089-83ee-c75c716bf26c\") " pod="e2e-models-e2e-worker-w2-e6c79d/premium-w2-e6c79d-kserve-6468bf46d4-h66f8"
Oct 31 12:38:37.863858 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:38:37.863831    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jnzw\" (UniqueName: \"kubernetes.io/projected/cd6a266f-8d12-4089-83ee-c75c716bf26c-kube-api-access-7jnzw\") pod \"premium-w2-e6c79d-kserve-6468bf46d4-h66f8\" (UID: \"cd6a266f-8d12-4089-83ee-c75c716bf26c\") " pod="e2e-models-e2e-worker-w2-e6c79d/premium-w2-e6c79d-kserve-6468bf46d4-h66f8"
Oct 31 12:38:37.864104 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:38:37.864086    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd6a266f-8d12-4089-83ee-c75c716bf26c-tmp-dir\") pod \"premium-w2-e6c79d-kserve-6468bf46d4-h66f8\" (UID: \"cd6a266f-8d12-4089-83ee-c75c716bf26c\") " pod="e2e-models-e2e-worker-w2-e6c79d/premium-w2-e6c79d-kserve-6468bf46d4-h66f8"
Oct 31 12:38:37.864224 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:38:37.864211    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd6a266f-8d12-4089-83ee-c75c716bf26c-dshm\") pod \"premium-w2-e6c79d-kserve-6468bf46d4-h66f8\" (UID: \"cd6a266f-8d12-4089-83ee-c75c716bf26c\") " pod="e2e-models-e2e-worker-w2-e6c79d/premium-w2-e6c79d-kserve-6468bf46d4-h66f8"
Oct 31 12:38:37.864389 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:38:37.864376    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd6a266f-8d12-4089-83ee-c75c716bf26c-model-cache\") pod \"premium-w2-e6c79d-kserve-6468bf46d4-h66f8\" (UID: \"cd6a266f-8d12-4089-83ee-c75c716bf26c\") " pod="e2e-models-e2e-worker-w2-e6c79d/premium-w2-e6c79d-kserve-6468bf46d4-h66f8"
Oct 31 12:38:37.864525 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:38:37.864508    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd6a266f-8d12-4089-83ee-c75c716bf26c-home\") pod \"premium-w2-e6c79d-kserve-6468bf46d4-h66f8\" (UID: \"cd6a266f-8d12-4089-83ee-c75c716bf26c\") " pod="e2e-models-e2e-worker-w2-e6c79d/premium-w2-e6c79d-kserve-6468bf46d4-h66f8"
Oct 31 12:38:37.865060 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:38:37.865041    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd6a266f-8d12-4089-83ee-c75c716bf26c-home\") pod \"premium-w2-e6c79d-kserve-6468bf46d4-h66f8\" (UID: \"cd6a266f-8d12-4089-83ee-c75c716bf26c\") " pod="e2e-models-e2e-worker-w2-e6c79d/premium-w2-e6c79d-kserve-6468bf46d4-h66f8"
Oct 31 12:38:37.865713 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:38:37.865691    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd6a266f-8d12-4089-83ee-c75c716bf26c-tmp-dir\") pod \"premium-w2-e6c79d-kserve-6468bf46d4-h66f8\" (UID: \"cd6a266f-8d12-4089-83ee-c75c716bf26c\") " pod="e2e-models-e2e-worker-w2-e6c79d/premium-w2-e6c79d-kserve-6468bf46d4-h66f8"
Oct 31 12:38:37.866268 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:38:37.866247    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd6a266f-8d12-4089-83ee-c75c716bf26c-tls-certs\") pod \"premium-w2-e6c79d-kserve-6468bf46d4-h66f8\" (UID: \"cd6a266f-8d12-4089-83ee-c75c716bf26c\") " pod="e2e-models-e2e-worker-w2-e6c79d/premium-w2-e6c79d-kserve-6468bf46d4-h66f8"
Oct 31 12:38:37.866683 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:38:37.866662    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd6a266f-8d12-4089-83ee-c75c716bf26c-model-cache\") pod \"premium-w2-e6c79d-kserve-6468bf46d4-h66f8\" (UID: \"cd6a266f-8d12-4089-83ee-c75c716bf26c\") " pod="e2e-models-e2e-worker-w2-e6c79d/premium-w2-e6c79d-kserve-6468bf46d4-h66f8"
Oct 31 12:38:37.889820 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:38:37.889794    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd6a266f-8d12-4089-83ee-c75c716bf26c-dshm\") pod \"premium-w2-e6c79d-kserve-6468bf46d4-h66f8\" (UID: \"cd6a266f-8d12-4089-83ee-c75c716bf26c\") " pod="e2e-models-e2e-worker-w2-e6c79d/premium-w2-e6c79d-kserve-6468bf46d4-h66f8"
Oct 31 12:38:37.895325 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:38:37.895291    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jnzw\" (UniqueName: \"kubernetes.io/projected/cd6a266f-8d12-4089-83ee-c75c716bf26c-kube-api-access-7jnzw\") pod \"premium-w2-e6c79d-kserve-6468bf46d4-h66f8\" (UID: \"cd6a266f-8d12-4089-83ee-c75c716bf26c\") " pod="e2e-models-e2e-worker-w2-e6c79d/premium-w2-e6c79d-kserve-6468bf46d4-h66f8"
Oct 31 12:38:37.966514 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:38:37.965735    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e6c79d/premium-w2-e6c79d-kserve-6468bf46d4-h66f8"
Oct 31 12:38:38.273854 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:38:38.273829    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-e6c79d/premium-w2-e6c79d-kserve-6468bf46d4-h66f8"]
Oct 31 12:38:38.277576 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:38:38.277545    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcd6a266f_8d12_4089_83ee_c75c716bf26c.slice/crio-d044400765c484e767d3a28f77a8ae1428d4fd428f0ae87b9e7e702f03fda464 WatchSource:0}: Error finding container d044400765c484e767d3a28f77a8ae1428d4fd428f0ae87b9e7e702f03fda464: Status 404 returned error can't find the container with id d044400765c484e767d3a28f77a8ae1428d4fd428f0ae87b9e7e702f03fda464
Oct 31 12:38:38.633249 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:38:38.633163    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e6c79d/premium-w2-e6c79d-kserve-6468bf46d4-h66f8" event={"ID":"cd6a266f-8d12-4089-83ee-c75c716bf26c","Type":"ContainerStarted","Data":"4d45f66c55ebe797c041b70d4991a2c802448b1a226ac362d630ca349b5f5bf1"}
Oct 31 12:38:38.633249 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:38:38.633211    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e6c79d/premium-w2-e6c79d-kserve-6468bf46d4-h66f8" event={"ID":"cd6a266f-8d12-4089-83ee-c75c716bf26c","Type":"ContainerStarted","Data":"d044400765c484e767d3a28f77a8ae1428d4fd428f0ae87b9e7e702f03fda464"}
Oct 31 12:38:38.653962 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:38:38.653917    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-e6c79d/premium-w2-e6c79d-kserve-6468bf46d4-h66f8" podStartSLOduration=1.6539049110000001 podStartE2EDuration="1.653904911s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:38:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:38:38.651223458 +0000 UTC m=+681.754291170" watchObservedRunningTime="2026-10-31 12:38:38.653904911 +0000 UTC m=+681.756972623"
Oct 31 12:38:38.967034 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:38:38.966935    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e6c79d/premium-w2-e6c79d-kserve-6468bf46d4-h66f8"
Oct 31 12:38:47.967035 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:38:47.966995    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-e6c79d/premium-w2-e6c79d-kserve-6468bf46d4-h66f8"
Oct 31 12:38:47.990095 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:38:47.990065    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-e6c79d/premium-w2-e6c79d-kserve-6468bf46d4-h66f8"
Oct 31 12:38:48.721465 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:38:48.721429    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e6c79d/premium-w2-e6c79d-kserve-6468bf46d4-h66f8"
Oct 31 12:38:59.924413 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:38:59.924368    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7phb" containerName="extractor"
Oct 31 12:39:04.156898 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:04.156861    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04484e50-2g9bs"]
Oct 31 12:39:04.160872 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:04.160848    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04484e50-2g9bs"
Oct 31 12:39:04.167098 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:04.166985    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7wsmb\""
Oct 31 12:39:04.167098 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:04.167039    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 12:39:04.167596 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:04.167576    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 12:39:04.167699 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:04.167627    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 12:39:04.170704 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:04.170517    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04484e50-2g9bs"]
Oct 31 12:39:04.180165 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:04.180131    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ef9f970-7daf-4b8f-98d0-5b4315ef9504-maas-api-xks-ca\") pod \"maas-api-revoke-sub-04484e50-2g9bs\" (UID: \"4ef9f970-7daf-4b8f-98d0-5b4315ef9504\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04484e50-2g9bs"
Oct 31 12:39:04.180294 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:04.180189    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mdvt\" (UniqueName: \"kubernetes.io/projected/4ef9f970-7daf-4b8f-98d0-5b4315ef9504-kube-api-access-2mdvt\") pod \"maas-api-revoke-sub-04484e50-2g9bs\" (UID: \"4ef9f970-7daf-4b8f-98d0-5b4315ef9504\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04484e50-2g9bs"
Oct 31 12:39:04.180294 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:04.180216    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ef9f970-7daf-4b8f-98d0-5b4315ef9504-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-04484e50-2g9bs\" (UID: \"4ef9f970-7daf-4b8f-98d0-5b4315ef9504\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04484e50-2g9bs"
Oct 31 12:39:04.281926 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:04.281890    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ef9f970-7daf-4b8f-98d0-5b4315ef9504-maas-api-xks-ca\") pod \"maas-api-revoke-sub-04484e50-2g9bs\" (UID: \"4ef9f970-7daf-4b8f-98d0-5b4315ef9504\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04484e50-2g9bs"
Oct 31 12:39:04.282127 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:04.281933    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mdvt\" (UniqueName: \"kubernetes.io/projected/4ef9f970-7daf-4b8f-98d0-5b4315ef9504-kube-api-access-2mdvt\") pod \"maas-api-revoke-sub-04484e50-2g9bs\" (UID: \"4ef9f970-7daf-4b8f-98d0-5b4315ef9504\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04484e50-2g9bs"
Oct 31 12:39:04.282127 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:04.281953    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ef9f970-7daf-4b8f-98d0-5b4315ef9504-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-04484e50-2g9bs\" (UID: \"4ef9f970-7daf-4b8f-98d0-5b4315ef9504\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04484e50-2g9bs"
Oct 31 12:39:04.282687 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:04.282661    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ef9f970-7daf-4b8f-98d0-5b4315ef9504-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-04484e50-2g9bs\" (UID: \"4ef9f970-7daf-4b8f-98d0-5b4315ef9504\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04484e50-2g9bs"
Oct 31 12:39:04.284472 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:04.284443    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ef9f970-7daf-4b8f-98d0-5b4315ef9504-maas-api-xks-ca\") pod \"maas-api-revoke-sub-04484e50-2g9bs\" (UID: \"4ef9f970-7daf-4b8f-98d0-5b4315ef9504\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04484e50-2g9bs"
Oct 31 12:39:04.295302 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:04.295272    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mdvt\" (UniqueName: \"kubernetes.io/projected/4ef9f970-7daf-4b8f-98d0-5b4315ef9504-kube-api-access-2mdvt\") pod \"maas-api-revoke-sub-04484e50-2g9bs\" (UID: \"4ef9f970-7daf-4b8f-98d0-5b4315ef9504\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04484e50-2g9bs"
Oct 31 12:39:04.489372 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:04.489275    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04484e50-2g9bs"
Oct 31 12:39:04.771645 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:04.771609    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04484e50-2g9bs"]
Oct 31 12:39:04.772817 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:39:04.772783    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ef9f970_7daf_4b8f_98d0_5b4315ef9504.slice/crio-75b430e641c6058741cfdfbd9953880d4ef421bbf1b2d8b840a45453e4699af4 WatchSource:0}: Error finding container 75b430e641c6058741cfdfbd9953880d4ef421bbf1b2d8b840a45453e4699af4: Status 404 returned error can't find the container with id 75b430e641c6058741cfdfbd9953880d4ef421bbf1b2d8b840a45453e4699af4
Oct 31 12:39:04.775204 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:04.775184    5060 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 12:39:04.837201 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:04.837161    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04484e50-2g9bs" event={"ID":"4ef9f970-7daf-4b8f-98d0-5b4315ef9504","Type":"ContainerStarted","Data":"75b430e641c6058741cfdfbd9953880d4ef421bbf1b2d8b840a45453e4699af4"}
Oct 31 12:39:07.159231 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:07.159192    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-mb4nn"]
Oct 31 12:39:07.164999 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:07.164958    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-mb4nn"
Oct 31 12:39:07.173955 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:07.173925    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-c67de2-openshift-default-dockercfg-psh8p\""
Oct 31 12:39:07.183156 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:07.183121    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-mb4nn"]
Oct 31 12:39:07.210381 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:07.210336    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ade94c0-4cba-4ccd-87c0-cdd0cc2aa72d-istio-data\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-mb4nn\" (UID: \"0ade94c0-4cba-4ccd-87c0-cdd0cc2aa72d\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-mb4nn"
Oct 31 12:39:07.210587 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:07.210450    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ade94c0-4cba-4ccd-87c0-cdd0cc2aa72d-istio-envoy\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-mb4nn\" (UID: \"0ade94c0-4cba-4ccd-87c0-cdd0cc2aa72d\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-mb4nn"
Oct 31 12:39:07.210587 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:07.210530    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ade94c0-4cba-4ccd-87c0-cdd0cc2aa72d-istio-podinfo\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-mb4nn\" (UID: \"0ade94c0-4cba-4ccd-87c0-cdd0cc2aa72d\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-mb4nn"
Oct 31 12:39:07.210587 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:07.210584    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ade94c0-4cba-4ccd-87c0-cdd0cc2aa72d-istio-token\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-mb4nn\" (UID: \"0ade94c0-4cba-4ccd-87c0-cdd0cc2aa72d\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-mb4nn"
Oct 31 12:39:07.210740 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:07.210653    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ade94c0-4cba-4ccd-87c0-cdd0cc2aa72d-istiod-ca-cert\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-mb4nn\" (UID: \"0ade94c0-4cba-4ccd-87c0-cdd0cc2aa72d\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-mb4nn"
Oct 31 12:39:07.210740 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:07.210677    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfj75\" (UniqueName: \"kubernetes.io/projected/0ade94c0-4cba-4ccd-87c0-cdd0cc2aa72d-kube-api-access-gfj75\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-mb4nn\" (UID: \"0ade94c0-4cba-4ccd-87c0-cdd0cc2aa72d\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-mb4nn"
Oct 31 12:39:07.210740 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:07.210695    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ade94c0-4cba-4ccd-87c0-cdd0cc2aa72d-workload-certs\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-mb4nn\" (UID: \"0ade94c0-4cba-4ccd-87c0-cdd0cc2aa72d\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-mb4nn"
Oct 31 12:39:07.210740 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:07.210735    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ade94c0-4cba-4ccd-87c0-cdd0cc2aa72d-credential-socket\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-mb4nn\" (UID: \"0ade94c0-4cba-4ccd-87c0-cdd0cc2aa72d\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-mb4nn"
Oct 31 12:39:07.210886 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:07.210753    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ade94c0-4cba-4ccd-87c0-cdd0cc2aa72d-workload-socket\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-mb4nn\" (UID: \"0ade94c0-4cba-4ccd-87c0-cdd0cc2aa72d\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-mb4nn"
Oct 31 12:39:07.320540 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:07.311607    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ade94c0-4cba-4ccd-87c0-cdd0cc2aa72d-istio-envoy\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-mb4nn\" (UID: \"0ade94c0-4cba-4ccd-87c0-cdd0cc2aa72d\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-mb4nn"
Oct 31 12:39:07.320540 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:07.311659    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ade94c0-4cba-4ccd-87c0-cdd0cc2aa72d-istio-podinfo\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-mb4nn\" (UID: \"0ade94c0-4cba-4ccd-87c0-cdd0cc2aa72d\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-mb4nn"
Oct 31 12:39:07.320540 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:07.311712    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ade94c0-4cba-4ccd-87c0-cdd0cc2aa72d-istio-token\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-mb4nn\" (UID: \"0ade94c0-4cba-4ccd-87c0-cdd0cc2aa72d\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-mb4nn"
Oct 31 12:39:07.320540 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:07.311795    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ade94c0-4cba-4ccd-87c0-cdd0cc2aa72d-istiod-ca-cert\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-mb4nn\" (UID: \"0ade94c0-4cba-4ccd-87c0-cdd0cc2aa72d\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-mb4nn"
Oct 31 12:39:07.320540 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:07.311823    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfj75\" (UniqueName: \"kubernetes.io/projected/0ade94c0-4cba-4ccd-87c0-cdd0cc2aa72d-kube-api-access-gfj75\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-mb4nn\" (UID: \"0ade94c0-4cba-4ccd-87c0-cdd0cc2aa72d\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-mb4nn"
Oct 31 12:39:07.320540 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:07.311855    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ade94c0-4cba-4ccd-87c0-cdd0cc2aa72d-workload-certs\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-mb4nn\" (UID: \"0ade94c0-4cba-4ccd-87c0-cdd0cc2aa72d\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-mb4nn"
Oct 31 12:39:07.320540 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:07.311911    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ade94c0-4cba-4ccd-87c0-cdd0cc2aa72d-credential-socket\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-mb4nn\" (UID: \"0ade94c0-4cba-4ccd-87c0-cdd0cc2aa72d\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-mb4nn"
Oct 31 12:39:07.320540 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:07.311941    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ade94c0-4cba-4ccd-87c0-cdd0cc2aa72d-workload-socket\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-mb4nn\" (UID: \"0ade94c0-4cba-4ccd-87c0-cdd0cc2aa72d\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-mb4nn"
Oct 31 12:39:07.320540 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:07.311991    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ade94c0-4cba-4ccd-87c0-cdd0cc2aa72d-istio-data\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-mb4nn\" (UID: \"0ade94c0-4cba-4ccd-87c0-cdd0cc2aa72d\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-mb4nn"
Oct 31 12:39:07.320540 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:07.312340    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ade94c0-4cba-4ccd-87c0-cdd0cc2aa72d-istio-data\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-mb4nn\" (UID: \"0ade94c0-4cba-4ccd-87c0-cdd0cc2aa72d\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-mb4nn"
Oct 31 12:39:07.320540 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:07.312591    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ade94c0-4cba-4ccd-87c0-cdd0cc2aa72d-workload-certs\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-mb4nn\" (UID: \"0ade94c0-4cba-4ccd-87c0-cdd0cc2aa72d\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-mb4nn"
Oct 31 12:39:07.320540 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:07.312784    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ade94c0-4cba-4ccd-87c0-cdd0cc2aa72d-credential-socket\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-mb4nn\" (UID: \"0ade94c0-4cba-4ccd-87c0-cdd0cc2aa72d\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-mb4nn"
Oct 31 12:39:07.320540 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:07.312869    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ade94c0-4cba-4ccd-87c0-cdd0cc2aa72d-workload-socket\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-mb4nn\" (UID: \"0ade94c0-4cba-4ccd-87c0-cdd0cc2aa72d\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-mb4nn"
Oct 31 12:39:07.320540 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:07.313194    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ade94c0-4cba-4ccd-87c0-cdd0cc2aa72d-istiod-ca-cert\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-mb4nn\" (UID: \"0ade94c0-4cba-4ccd-87c0-cdd0cc2aa72d\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-mb4nn"
Oct 31 12:39:07.320540 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:07.314803    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ade94c0-4cba-4ccd-87c0-cdd0cc2aa72d-istio-podinfo\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-mb4nn\" (UID: \"0ade94c0-4cba-4ccd-87c0-cdd0cc2aa72d\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-mb4nn"
Oct 31 12:39:07.320540 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:07.315608    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ade94c0-4cba-4ccd-87c0-cdd0cc2aa72d-istio-envoy\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-mb4nn\" (UID: \"0ade94c0-4cba-4ccd-87c0-cdd0cc2aa72d\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-mb4nn"
Oct 31 12:39:07.325239 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:07.325207    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfj75\" (UniqueName: \"kubernetes.io/projected/0ade94c0-4cba-4ccd-87c0-cdd0cc2aa72d-kube-api-access-gfj75\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-mb4nn\" (UID: \"0ade94c0-4cba-4ccd-87c0-cdd0cc2aa72d\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-mb4nn"
Oct 31 12:39:07.326116 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:07.326091    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ade94c0-4cba-4ccd-87c0-cdd0cc2aa72d-istio-token\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-mb4nn\" (UID: \"0ade94c0-4cba-4ccd-87c0-cdd0cc2aa72d\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-mb4nn"
Oct 31 12:39:07.485884 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:07.485803    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-mb4nn"
Oct 31 12:39:07.780762 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:39:07.780724    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ade94c0_4cba_4ccd_87c0_cdd0cc2aa72d.slice/crio-43024daa959faca0f71635d631ac87ecd654825c8d1dbe8347f526c9464dfe4b WatchSource:0}: Error finding container 43024daa959faca0f71635d631ac87ecd654825c8d1dbe8347f526c9464dfe4b: Status 404 returned error can't find the container with id 43024daa959faca0f71635d631ac87ecd654825c8d1dbe8347f526c9464dfe4b
Oct 31 12:39:07.780762 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:07.780658    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-mb4nn"]
Oct 31 12:39:07.783431 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:07.783388    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:39:07.783562 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:07.783483    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:39:07.783628 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:07.783596    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:39:07.860069 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:07.860039    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-mb4nn" event={"ID":"0ade94c0-4cba-4ccd-87c0-cdd0cc2aa72d","Type":"ContainerStarted","Data":"43024daa959faca0f71635d631ac87ecd654825c8d1dbe8347f526c9464dfe4b"}
Oct 31 12:39:08.896744 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:08.896638    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-mb4nn" event={"ID":"0ade94c0-4cba-4ccd-87c0-cdd0cc2aa72d","Type":"ContainerStarted","Data":"b1792e1e8e3a369422a6939d613bdc1025975b26261bb7666cbc4972dcae2ebd"}
Oct 31 12:39:08.898155 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:08.898125    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04484e50-2g9bs" event={"ID":"4ef9f970-7daf-4b8f-98d0-5b4315ef9504","Type":"ContainerStarted","Data":"17ac645399249d4e7e7283ce79c8b50c48e6c3dca2982f14aaf643c8a90030e4"}
Oct 31 12:39:08.923175 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:08.923128    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-mb4nn" podStartSLOduration=1.92311389 podStartE2EDuration="1.92311389s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:39:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:39:08.920603443 +0000 UTC m=+712.023671154" watchObservedRunningTime="2026-10-31 12:39:08.92311389 +0000 UTC m=+712.026181603"
Oct 31 12:39:08.950987 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:08.950943    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04484e50-2g9bs" podStartSLOduration=1.417483997 podStartE2EDuration="4.950930294s" totalImagesPullingTime="3.533446297s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:39:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:39:08.94936953 +0000 UTC m=+712.052437221" watchObservedRunningTime="2026-10-31 12:39:08.950930294 +0000 UTC m=+712.053998034"
Oct 31 12:39:09.489872 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:09.489818    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-mb4nn"
Oct 31 12:39:09.491437 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:09.491410    5060 patch_prober.go:29] interesting pod/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-mb4nn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Oct 31 12:39:09.491685 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:09.491632    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-mb4nn" podUID="0ade94c0-4cba-4ccd-87c0-cdd0cc2aa72d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Oct 31 12:39:09.907180 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:09.907140    5060 generic.go:356] "Generic (PLEG): container finished" podID="4ef9f970-7daf-4b8f-98d0-5b4315ef9504" containerID="17ac645399249d4e7e7283ce79c8b50c48e6c3dca2982f14aaf643c8a90030e4" exitCode=0
Oct 31 12:39:09.907670 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:09.907223    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04484e50-2g9bs" event={"ID":"4ef9f970-7daf-4b8f-98d0-5b4315ef9504","Type":"ContainerDied","Data":"17ac645399249d4e7e7283ce79c8b50c48e6c3dca2982f14aaf643c8a90030e4"}
Oct 31 12:39:10.486985 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:10.486950    5060 patch_prober.go:29] interesting pod/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-mb4nn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Oct 31 12:39:10.487201 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:10.487003    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-mb4nn" podUID="0ade94c0-4cba-4ccd-87c0-cdd0cc2aa72d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Oct 31 12:39:11.127639 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:11.127606    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04484e50-2g9bs"
Oct 31 12:39:11.182032 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:11.181934    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4ef9f970-7daf-4b8f-98d0-5b4315ef9504-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ef9f970-7daf-4b8f-98d0-5b4315ef9504-maas-api-xks-ca\") pod \"4ef9f970-7daf-4b8f-98d0-5b4315ef9504\" (UID: \"4ef9f970-7daf-4b8f-98d0-5b4315ef9504\") "
Oct 31 12:39:11.182357 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:11.182328    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ef9f970-7daf-4b8f-98d0-5b4315ef9504-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ef9f970-7daf-4b8f-98d0-5b4315ef9504-maas-api-openshift-ca\") pod \"4ef9f970-7daf-4b8f-98d0-5b4315ef9504\" (UID: \"4ef9f970-7daf-4b8f-98d0-5b4315ef9504\") "
Oct 31 12:39:11.190618 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:11.190592    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ef9f970-7daf-4b8f-98d0-5b4315ef9504-kube-api-access-2mdvt\" (UniqueName: \"kubernetes.io/projected/4ef9f970-7daf-4b8f-98d0-5b4315ef9504-kube-api-access-2mdvt\") pod \"4ef9f970-7daf-4b8f-98d0-5b4315ef9504\" (UID: \"4ef9f970-7daf-4b8f-98d0-5b4315ef9504\") "
Oct 31 12:39:11.190928 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:11.184213    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ef9f970-7daf-4b8f-98d0-5b4315ef9504-maas-api-openshift-ca" pod "4ef9f970-7daf-4b8f-98d0-5b4315ef9504" (UID: "4ef9f970-7daf-4b8f-98d0-5b4315ef9504"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:39:11.190928 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:11.188884    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ef9f970-7daf-4b8f-98d0-5b4315ef9504-maas-api-xks-ca" pod "4ef9f970-7daf-4b8f-98d0-5b4315ef9504" (UID: "4ef9f970-7daf-4b8f-98d0-5b4315ef9504"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:39:11.191227 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:11.191204    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ef9f970-7daf-4b8f-98d0-5b4315ef9504-maas-api-xks-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:39:11.191313 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:11.191233    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ef9f970-7daf-4b8f-98d0-5b4315ef9504-maas-api-openshift-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:39:11.193405 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:11.193376    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ef9f970-7daf-4b8f-98d0-5b4315ef9504-kube-api-access-2mdvt" pod "4ef9f970-7daf-4b8f-98d0-5b4315ef9504" (UID: "4ef9f970-7daf-4b8f-98d0-5b4315ef9504"). InnerVolumeSpecName "kube-api-access-2mdvt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:39:11.292554 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:11.292514    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2mdvt\" (UniqueName: \"kubernetes.io/projected/4ef9f970-7daf-4b8f-98d0-5b4315ef9504-kube-api-access-2mdvt\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:39:11.486704 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:11.486612    5060 patch_prober.go:29] interesting pod/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-mb4nn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Oct 31 12:39:11.486704 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:11.486677    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-mb4nn" podUID="0ade94c0-4cba-4ccd-87c0-cdd0cc2aa72d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Oct 31 12:39:11.923573 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:11.923545    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04484e50-2g9bs"
Oct 31 12:39:11.923729 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:11.923583    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04484e50-2g9bs" event={"ID":"4ef9f970-7daf-4b8f-98d0-5b4315ef9504","Type":"ContainerDied","Data":"75b430e641c6058741cfdfbd9953880d4ef421bbf1b2d8b840a45453e4699af4"}
Oct 31 12:39:11.923729 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:11.923634    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="75b430e641c6058741cfdfbd9953880d4ef421bbf1b2d8b840a45453e4699af4"
Oct 31 12:39:12.487096 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:12.487059    5060 patch_prober.go:29] interesting pod/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-mb4nn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Oct 31 12:39:12.487554 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:12.487119    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-mb4nn" podUID="0ade94c0-4cba-4ccd-87c0-cdd0cc2aa72d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Oct 31 12:39:13.487107 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:13.487066    5060 patch_prober.go:29] interesting pod/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-mb4nn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Oct 31 12:39:13.487592 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:13.487150    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-mb4nn" podUID="0ade94c0-4cba-4ccd-87c0-cdd0cc2aa72d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Oct 31 12:39:14.517340 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:14.517214    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-mb4nn"
Oct 31 12:39:14.518394 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:14.518369    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-mb4nn"
Oct 31 12:39:14.519224 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:14.519203    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-mb4nn"
Oct 31 12:39:18.273423 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:18.273387    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cdc017bf-9swb5"]
Oct 31 12:39:18.273990 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:18.273973    5060 cpu_manager.go:414] "containerMap: removing container" podUID="4ef9f970-7daf-4b8f-98d0-5b4315ef9504" containerName="revoke-keys"
Oct 31 12:39:18.274042 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:18.273992    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ef9f970-7daf-4b8f-98d0-5b4315ef9504" containerName="revoke-keys"
Oct 31 12:39:18.274104 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:18.274093    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ef9f970-7daf-4b8f-98d0-5b4315ef9504" containerName="revoke-keys"
Oct 31 12:39:18.277906 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:18.277877    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdc017bf-9swb5"
Oct 31 12:39:18.282782 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:18.282758    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 12:39:18.282782 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:18.282767    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 12:39:18.283919 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:18.283879    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 12:39:18.283919 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:18.283903    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7wsmb\""
Oct 31 12:39:18.288375 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:18.288348    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cdc017bf-9swb5"]
Oct 31 12:39:18.382095 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:18.382063    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8a3573f-fa00-4b9d-9ea5-b961e6efd4f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cdc017bf-9swb5\" (UID: \"a8a3573f-fa00-4b9d-9ea5-b961e6efd4f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdc017bf-9swb5"
Oct 31 12:39:18.382252 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:18.382192    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgqnb\" (UniqueName: \"kubernetes.io/projected/a8a3573f-fa00-4b9d-9ea5-b961e6efd4f6-kube-api-access-rgqnb\") pod \"maas-api-revoke-sub-cdc017bf-9swb5\" (UID: \"a8a3573f-fa00-4b9d-9ea5-b961e6efd4f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdc017bf-9swb5"
Oct 31 12:39:18.382308 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:18.382281    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8a3573f-fa00-4b9d-9ea5-b961e6efd4f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cdc017bf-9swb5\" (UID: \"a8a3573f-fa00-4b9d-9ea5-b961e6efd4f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdc017bf-9swb5"
Oct 31 12:39:18.483596 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:18.483557    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgqnb\" (UniqueName: \"kubernetes.io/projected/a8a3573f-fa00-4b9d-9ea5-b961e6efd4f6-kube-api-access-rgqnb\") pod \"maas-api-revoke-sub-cdc017bf-9swb5\" (UID: \"a8a3573f-fa00-4b9d-9ea5-b961e6efd4f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdc017bf-9swb5"
Oct 31 12:39:18.483788 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:18.483654    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8a3573f-fa00-4b9d-9ea5-b961e6efd4f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cdc017bf-9swb5\" (UID: \"a8a3573f-fa00-4b9d-9ea5-b961e6efd4f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdc017bf-9swb5"
Oct 31 12:39:18.483788 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:18.483742    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8a3573f-fa00-4b9d-9ea5-b961e6efd4f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cdc017bf-9swb5\" (UID: \"a8a3573f-fa00-4b9d-9ea5-b961e6efd4f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdc017bf-9swb5"
Oct 31 12:39:18.484547 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:18.484525    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8a3573f-fa00-4b9d-9ea5-b961e6efd4f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cdc017bf-9swb5\" (UID: \"a8a3573f-fa00-4b9d-9ea5-b961e6efd4f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdc017bf-9swb5"
Oct 31 12:39:18.486057 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:18.486032    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8a3573f-fa00-4b9d-9ea5-b961e6efd4f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cdc017bf-9swb5\" (UID: \"a8a3573f-fa00-4b9d-9ea5-b961e6efd4f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdc017bf-9swb5"
Oct 31 12:39:18.495701 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:18.495675    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgqnb\" (UniqueName: \"kubernetes.io/projected/a8a3573f-fa00-4b9d-9ea5-b961e6efd4f6-kube-api-access-rgqnb\") pod \"maas-api-revoke-sub-cdc017bf-9swb5\" (UID: \"a8a3573f-fa00-4b9d-9ea5-b961e6efd4f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdc017bf-9swb5"
Oct 31 12:39:18.593981 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:18.593897    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdc017bf-9swb5"
Oct 31 12:39:18.815054 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:18.815019    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9545da26-bbzlv"]
Oct 31 12:39:18.820275 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:18.820255    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9545da26-bbzlv"
Oct 31 12:39:18.828879 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:18.828848    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9545da26-bbzlv"]
Oct 31 12:39:18.880857 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:18.880826    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cdc017bf-9swb5"]
Oct 31 12:39:18.882376 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:39:18.882343    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8a3573f_fa00_4b9d_9ea5_b961e6efd4f6.slice/crio-a0ea5ef837f0d2654229cec4d1bf030e1977a4b55d2a16b266e42be91e424de7 WatchSource:0}: Error finding container a0ea5ef837f0d2654229cec4d1bf030e1977a4b55d2a16b266e42be91e424de7: Status 404 returned error can't find the container with id a0ea5ef837f0d2654229cec4d1bf030e1977a4b55d2a16b266e42be91e424de7
Oct 31 12:39:18.971711 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:18.971682    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdc017bf-9swb5" event={"ID":"a8a3573f-fa00-4b9d-9ea5-b961e6efd4f6","Type":"ContainerStarted","Data":"a0ea5ef837f0d2654229cec4d1bf030e1977a4b55d2a16b266e42be91e424de7"}
Oct 31 12:39:18.989977 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:18.989952    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5206e1f-8f34-4035-b26f-3ba2cc158edd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9545da26-bbzlv\" (UID: \"d5206e1f-8f34-4035-b26f-3ba2cc158edd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9545da26-bbzlv"
Oct 31 12:39:18.990101 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:18.990034    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrdr7\" (UniqueName: \"kubernetes.io/projected/d5206e1f-8f34-4035-b26f-3ba2cc158edd-kube-api-access-qrdr7\") pod \"maas-api-revoke-sub-9545da26-bbzlv\" (UID: \"d5206e1f-8f34-4035-b26f-3ba2cc158edd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9545da26-bbzlv"
Oct 31 12:39:18.990101 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:18.990082    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5206e1f-8f34-4035-b26f-3ba2cc158edd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9545da26-bbzlv\" (UID: \"d5206e1f-8f34-4035-b26f-3ba2cc158edd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9545da26-bbzlv"
Oct 31 12:39:19.091023 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:19.090977    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrdr7\" (UniqueName: \"kubernetes.io/projected/d5206e1f-8f34-4035-b26f-3ba2cc158edd-kube-api-access-qrdr7\") pod \"maas-api-revoke-sub-9545da26-bbzlv\" (UID: \"d5206e1f-8f34-4035-b26f-3ba2cc158edd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9545da26-bbzlv"
Oct 31 12:39:19.091202 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:19.091064    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5206e1f-8f34-4035-b26f-3ba2cc158edd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9545da26-bbzlv\" (UID: \"d5206e1f-8f34-4035-b26f-3ba2cc158edd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9545da26-bbzlv"
Oct 31 12:39:19.091202 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:19.091188    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5206e1f-8f34-4035-b26f-3ba2cc158edd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9545da26-bbzlv\" (UID: \"d5206e1f-8f34-4035-b26f-3ba2cc158edd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9545da26-bbzlv"
Oct 31 12:39:19.091791 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:19.091764    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5206e1f-8f34-4035-b26f-3ba2cc158edd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9545da26-bbzlv\" (UID: \"d5206e1f-8f34-4035-b26f-3ba2cc158edd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9545da26-bbzlv"
Oct 31 12:39:19.093644 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:19.093617    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5206e1f-8f34-4035-b26f-3ba2cc158edd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9545da26-bbzlv\" (UID: \"d5206e1f-8f34-4035-b26f-3ba2cc158edd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9545da26-bbzlv"
Oct 31 12:39:19.103787 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:19.103756    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrdr7\" (UniqueName: \"kubernetes.io/projected/d5206e1f-8f34-4035-b26f-3ba2cc158edd-kube-api-access-qrdr7\") pod \"maas-api-revoke-sub-9545da26-bbzlv\" (UID: \"d5206e1f-8f34-4035-b26f-3ba2cc158edd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9545da26-bbzlv"
Oct 31 12:39:19.138764 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:19.138701    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9545da26-bbzlv"
Oct 31 12:39:19.422598 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:19.422560    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9545da26-bbzlv"]
Oct 31 12:39:19.423042 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:39:19.422607    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5206e1f_8f34_4035_b26f_3ba2cc158edd.slice/crio-b818184dc0f6f3af2be3a9517c8f2a82b7c9b3996c08f3e2742c9e70969093e0 WatchSource:0}: Error finding container b818184dc0f6f3af2be3a9517c8f2a82b7c9b3996c08f3e2742c9e70969093e0: Status 404 returned error can't find the container with id b818184dc0f6f3af2be3a9517c8f2a82b7c9b3996c08f3e2742c9e70969093e0
Oct 31 12:39:19.980309 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:19.980272    5060 generic.go:356] "Generic (PLEG): container finished" podID="a8a3573f-fa00-4b9d-9ea5-b961e6efd4f6" containerID="af85ee4d4436585ba9405c88b237ab12bf0900688f701c39b74fa1c1f57e3ae5" exitCode=0
Oct 31 12:39:19.980523 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:19.980356    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdc017bf-9swb5" event={"ID":"a8a3573f-fa00-4b9d-9ea5-b961e6efd4f6","Type":"ContainerDied","Data":"af85ee4d4436585ba9405c88b237ab12bf0900688f701c39b74fa1c1f57e3ae5"}
Oct 31 12:39:19.982197 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:19.982173    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9545da26-bbzlv" event={"ID":"d5206e1f-8f34-4035-b26f-3ba2cc158edd","Type":"ContainerStarted","Data":"c00c16c2d00eaf1343b63e6f816406d706d9049151d496c49ffb5f5aeed849e9"}
Oct 31 12:39:19.982329 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:19.982222    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9545da26-bbzlv" event={"ID":"d5206e1f-8f34-4035-b26f-3ba2cc158edd","Type":"ContainerStarted","Data":"b818184dc0f6f3af2be3a9517c8f2a82b7c9b3996c08f3e2742c9e70969093e0"}
Oct 31 12:39:20.035359 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:20.035307    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9545da26-bbzlv" podStartSLOduration=2.035290426 podStartE2EDuration="2.035290426s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:39:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:39:20.032260241 +0000 UTC m=+723.135327955" watchObservedRunningTime="2026-10-31 12:39:20.035290426 +0000 UTC m=+723.138358139"
Oct 31 12:39:20.992770 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:20.992736    5060 generic.go:356] "Generic (PLEG): container finished" podID="d5206e1f-8f34-4035-b26f-3ba2cc158edd" containerID="c00c16c2d00eaf1343b63e6f816406d706d9049151d496c49ffb5f5aeed849e9" exitCode=0
Oct 31 12:39:20.993263 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:20.992820    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9545da26-bbzlv" event={"ID":"d5206e1f-8f34-4035-b26f-3ba2cc158edd","Type":"ContainerDied","Data":"c00c16c2d00eaf1343b63e6f816406d706d9049151d496c49ffb5f5aeed849e9"}
Oct 31 12:39:21.144183 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:21.144157    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdc017bf-9swb5"
Oct 31 12:39:21.316899 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:21.316867    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8a3573f-fa00-4b9d-9ea5-b961e6efd4f6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8a3573f-fa00-4b9d-9ea5-b961e6efd4f6-maas-api-openshift-ca\") pod \"a8a3573f-fa00-4b9d-9ea5-b961e6efd4f6\" (UID: \"a8a3573f-fa00-4b9d-9ea5-b961e6efd4f6\") "
Oct 31 12:39:21.316899 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:21.316905    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a8a3573f-fa00-4b9d-9ea5-b961e6efd4f6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8a3573f-fa00-4b9d-9ea5-b961e6efd4f6-maas-api-xks-ca\") pod \"a8a3573f-fa00-4b9d-9ea5-b961e6efd4f6\" (UID: \"a8a3573f-fa00-4b9d-9ea5-b961e6efd4f6\") "
Oct 31 12:39:21.317126 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:21.317032    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8a3573f-fa00-4b9d-9ea5-b961e6efd4f6-kube-api-access-rgqnb\" (UniqueName: \"kubernetes.io/projected/a8a3573f-fa00-4b9d-9ea5-b961e6efd4f6-kube-api-access-rgqnb\") pod \"a8a3573f-fa00-4b9d-9ea5-b961e6efd4f6\" (UID: \"a8a3573f-fa00-4b9d-9ea5-b961e6efd4f6\") "
Oct 31 12:39:21.317259 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:21.317237    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8a3573f-fa00-4b9d-9ea5-b961e6efd4f6-maas-api-openshift-ca" pod "a8a3573f-fa00-4b9d-9ea5-b961e6efd4f6" (UID: "a8a3573f-fa00-4b9d-9ea5-b961e6efd4f6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:39:21.317518 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:21.317477    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8a3573f-fa00-4b9d-9ea5-b961e6efd4f6-maas-api-openshift-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:39:21.318960 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:21.318940    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8a3573f-fa00-4b9d-9ea5-b961e6efd4f6-maas-api-xks-ca" pod "a8a3573f-fa00-4b9d-9ea5-b961e6efd4f6" (UID: "a8a3573f-fa00-4b9d-9ea5-b961e6efd4f6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:39:21.319118 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:21.319098    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8a3573f-fa00-4b9d-9ea5-b961e6efd4f6-kube-api-access-rgqnb" pod "a8a3573f-fa00-4b9d-9ea5-b961e6efd4f6" (UID: "a8a3573f-fa00-4b9d-9ea5-b961e6efd4f6"). InnerVolumeSpecName "kube-api-access-rgqnb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:39:21.418259 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:21.418227    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rgqnb\" (UniqueName: \"kubernetes.io/projected/a8a3573f-fa00-4b9d-9ea5-b961e6efd4f6-kube-api-access-rgqnb\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:39:21.418259 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:21.418257    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8a3573f-fa00-4b9d-9ea5-b961e6efd4f6-maas-api-xks-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:39:22.006309 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:22.006275    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdc017bf-9swb5"
Oct 31 12:39:22.006765 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:22.006316    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdc017bf-9swb5" event={"ID":"a8a3573f-fa00-4b9d-9ea5-b961e6efd4f6","Type":"ContainerDied","Data":"a0ea5ef837f0d2654229cec4d1bf030e1977a4b55d2a16b266e42be91e424de7"}
Oct 31 12:39:22.006765 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:22.006365    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a0ea5ef837f0d2654229cec4d1bf030e1977a4b55d2a16b266e42be91e424de7"
Oct 31 12:39:22.163928 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:22.163899    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9545da26-bbzlv"
Oct 31 12:39:22.327791 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:22.327761    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5206e1f-8f34-4035-b26f-3ba2cc158edd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5206e1f-8f34-4035-b26f-3ba2cc158edd-maas-api-openshift-ca\") pod \"d5206e1f-8f34-4035-b26f-3ba2cc158edd\" (UID: \"d5206e1f-8f34-4035-b26f-3ba2cc158edd\") "
Oct 31 12:39:22.328006 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:22.327811    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d5206e1f-8f34-4035-b26f-3ba2cc158edd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5206e1f-8f34-4035-b26f-3ba2cc158edd-maas-api-xks-ca\") pod \"d5206e1f-8f34-4035-b26f-3ba2cc158edd\" (UID: \"d5206e1f-8f34-4035-b26f-3ba2cc158edd\") "
Oct 31 12:39:22.328006 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:22.327902    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5206e1f-8f34-4035-b26f-3ba2cc158edd-kube-api-access-qrdr7\" (UniqueName: \"kubernetes.io/projected/d5206e1f-8f34-4035-b26f-3ba2cc158edd-kube-api-access-qrdr7\") pod \"d5206e1f-8f34-4035-b26f-3ba2cc158edd\" (UID: \"d5206e1f-8f34-4035-b26f-3ba2cc158edd\") "
Oct 31 12:39:22.328195 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:22.328170    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5206e1f-8f34-4035-b26f-3ba2cc158edd-maas-api-openshift-ca" pod "d5206e1f-8f34-4035-b26f-3ba2cc158edd" (UID: "d5206e1f-8f34-4035-b26f-3ba2cc158edd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:39:22.330160 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:22.330132    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5206e1f-8f34-4035-b26f-3ba2cc158edd-kube-api-access-qrdr7" pod "d5206e1f-8f34-4035-b26f-3ba2cc158edd" (UID: "d5206e1f-8f34-4035-b26f-3ba2cc158edd"). InnerVolumeSpecName "kube-api-access-qrdr7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:39:22.330301 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:22.330282    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5206e1f-8f34-4035-b26f-3ba2cc158edd-maas-api-xks-ca" pod "d5206e1f-8f34-4035-b26f-3ba2cc158edd" (UID: "d5206e1f-8f34-4035-b26f-3ba2cc158edd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:39:22.361112 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:22.361081    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-r8l72"]
Oct 31 12:39:22.361699 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:22.361682    5060 cpu_manager.go:414] "containerMap: removing container" podUID="d5206e1f-8f34-4035-b26f-3ba2cc158edd" containerName="revoke-keys"
Oct 31 12:39:22.361699 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:22.361699    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5206e1f-8f34-4035-b26f-3ba2cc158edd" containerName="revoke-keys"
Oct 31 12:39:22.361862 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:22.361711    5060 cpu_manager.go:414] "containerMap: removing container" podUID="a8a3573f-fa00-4b9d-9ea5-b961e6efd4f6" containerName="revoke-keys"
Oct 31 12:39:22.361862 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:22.361717    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8a3573f-fa00-4b9d-9ea5-b961e6efd4f6" containerName="revoke-keys"
Oct 31 12:39:22.361862 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:22.361809    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8a3573f-fa00-4b9d-9ea5-b961e6efd4f6" containerName="revoke-keys"
Oct 31 12:39:22.361862 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:22.361828    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5206e1f-8f34-4035-b26f-3ba2cc158edd" containerName="revoke-keys"
Oct 31 12:39:22.365255 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:22.365235    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-r8l72"
Oct 31 12:39:22.375006 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:22.374981    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-e6c79d-openshift-default-dockercfg-hwsts\""
Oct 31 12:39:22.433413 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:22.429537    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-r8l72"]
Oct 31 12:39:22.433413 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:22.430173    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5206e1f-8f34-4035-b26f-3ba2cc158edd-maas-api-xks-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:39:22.433413 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:22.430192    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrdr7\" (UniqueName: \"kubernetes.io/projected/d5206e1f-8f34-4035-b26f-3ba2cc158edd-kube-api-access-qrdr7\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:39:22.433413 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:22.430204    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5206e1f-8f34-4035-b26f-3ba2cc158edd-maas-api-openshift-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:39:22.531093 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:22.531054    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c765a671-664f-40d2-9b62-67c746c397d0-workload-certs\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-r8l72\" (UID: \"c765a671-664f-40d2-9b62-67c746c397d0\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-r8l72"
Oct 31 12:39:22.531258 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:22.531130    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c765a671-664f-40d2-9b62-67c746c397d0-istio-podinfo\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-r8l72\" (UID: \"c765a671-664f-40d2-9b62-67c746c397d0\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-r8l72"
Oct 31 12:39:22.531258 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:22.531202    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c765a671-664f-40d2-9b62-67c746c397d0-istio-token\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-r8l72\" (UID: \"c765a671-664f-40d2-9b62-67c746c397d0\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-r8l72"
Oct 31 12:39:22.531258 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:22.531224    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c765a671-664f-40d2-9b62-67c746c397d0-istiod-ca-cert\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-r8l72\" (UID: \"c765a671-664f-40d2-9b62-67c746c397d0\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-r8l72"
Oct 31 12:39:22.531258 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:22.531245    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c765a671-664f-40d2-9b62-67c746c397d0-credential-socket\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-r8l72\" (UID: \"c765a671-664f-40d2-9b62-67c746c397d0\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-r8l72"
Oct 31 12:39:22.531390 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:22.531260    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c765a671-664f-40d2-9b62-67c746c397d0-istio-envoy\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-r8l72\" (UID: \"c765a671-664f-40d2-9b62-67c746c397d0\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-r8l72"
Oct 31 12:39:22.531390 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:22.531326    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5nj7\" (UniqueName: \"kubernetes.io/projected/c765a671-664f-40d2-9b62-67c746c397d0-kube-api-access-b5nj7\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-r8l72\" (UID: \"c765a671-664f-40d2-9b62-67c746c397d0\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-r8l72"
Oct 31 12:39:22.531461 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:22.531447    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c765a671-664f-40d2-9b62-67c746c397d0-istio-data\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-r8l72\" (UID: \"c765a671-664f-40d2-9b62-67c746c397d0\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-r8l72"
Oct 31 12:39:22.531529 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:22.531482    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c765a671-664f-40d2-9b62-67c746c397d0-workload-socket\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-r8l72\" (UID: \"c765a671-664f-40d2-9b62-67c746c397d0\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-r8l72"
Oct 31 12:39:22.632622 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:22.632532    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c765a671-664f-40d2-9b62-67c746c397d0-istio-podinfo\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-r8l72\" (UID: \"c765a671-664f-40d2-9b62-67c746c397d0\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-r8l72"
Oct 31 12:39:22.632622 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:22.632592    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c765a671-664f-40d2-9b62-67c746c397d0-istio-token\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-r8l72\" (UID: \"c765a671-664f-40d2-9b62-67c746c397d0\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-r8l72"
Oct 31 12:39:22.632816 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:22.632653    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c765a671-664f-40d2-9b62-67c746c397d0-istiod-ca-cert\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-r8l72\" (UID: \"c765a671-664f-40d2-9b62-67c746c397d0\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-r8l72"
Oct 31 12:39:22.632816 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:22.632715    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c765a671-664f-40d2-9b62-67c746c397d0-credential-socket\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-r8l72\" (UID: \"c765a671-664f-40d2-9b62-67c746c397d0\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-r8l72"
Oct 31 12:39:22.632816 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:22.632738    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c765a671-664f-40d2-9b62-67c746c397d0-istio-envoy\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-r8l72\" (UID: \"c765a671-664f-40d2-9b62-67c746c397d0\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-r8l72"
Oct 31 12:39:22.632816 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:22.632776    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5nj7\" (UniqueName: \"kubernetes.io/projected/c765a671-664f-40d2-9b62-67c746c397d0-kube-api-access-b5nj7\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-r8l72\" (UID: \"c765a671-664f-40d2-9b62-67c746c397d0\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-r8l72"
Oct 31 12:39:22.633017 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:22.632913    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c765a671-664f-40d2-9b62-67c746c397d0-istio-data\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-r8l72\" (UID: \"c765a671-664f-40d2-9b62-67c746c397d0\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-r8l72"
Oct 31 12:39:22.633017 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:22.632946    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c765a671-664f-40d2-9b62-67c746c397d0-workload-socket\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-r8l72\" (UID: \"c765a671-664f-40d2-9b62-67c746c397d0\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-r8l72"
Oct 31 12:39:22.633121 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:22.633016    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c765a671-664f-40d2-9b62-67c746c397d0-workload-certs\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-r8l72\" (UID: \"c765a671-664f-40d2-9b62-67c746c397d0\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-r8l72"
Oct 31 12:39:22.633205 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:22.633173    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c765a671-664f-40d2-9b62-67c746c397d0-credential-socket\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-r8l72\" (UID: \"c765a671-664f-40d2-9b62-67c746c397d0\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-r8l72"
Oct 31 12:39:22.633465 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:22.633367    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c765a671-664f-40d2-9b62-67c746c397d0-workload-certs\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-r8l72\" (UID: \"c765a671-664f-40d2-9b62-67c746c397d0\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-r8l72"
Oct 31 12:39:22.633465 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:22.633409    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c765a671-664f-40d2-9b62-67c746c397d0-workload-socket\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-r8l72\" (UID: \"c765a671-664f-40d2-9b62-67c746c397d0\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-r8l72"
Oct 31 12:39:22.633670 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:22.633473    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c765a671-664f-40d2-9b62-67c746c397d0-istio-data\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-r8l72\" (UID: \"c765a671-664f-40d2-9b62-67c746c397d0\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-r8l72"
Oct 31 12:39:22.633897 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:22.633871    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c765a671-664f-40d2-9b62-67c746c397d0-istiod-ca-cert\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-r8l72\" (UID: \"c765a671-664f-40d2-9b62-67c746c397d0\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-r8l72"
Oct 31 12:39:22.635577 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:22.635550    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c765a671-664f-40d2-9b62-67c746c397d0-istio-podinfo\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-r8l72\" (UID: \"c765a671-664f-40d2-9b62-67c746c397d0\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-r8l72"
Oct 31 12:39:22.635660 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:22.635581    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c765a671-664f-40d2-9b62-67c746c397d0-istio-envoy\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-r8l72\" (UID: \"c765a671-664f-40d2-9b62-67c746c397d0\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-r8l72"
Oct 31 12:39:22.644953 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:22.644925    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c765a671-664f-40d2-9b62-67c746c397d0-istio-token\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-r8l72\" (UID: \"c765a671-664f-40d2-9b62-67c746c397d0\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-r8l72"
Oct 31 12:39:22.645111 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:22.645091    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5nj7\" (UniqueName: \"kubernetes.io/projected/c765a671-664f-40d2-9b62-67c746c397d0-kube-api-access-b5nj7\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-r8l72\" (UID: \"c765a671-664f-40d2-9b62-67c746c397d0\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-r8l72"
Oct 31 12:39:22.738344 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:22.735364    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-r8l72"
Oct 31 12:39:23.019583 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:23.018403    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9545da26-bbzlv"
Oct 31 12:39:23.019583 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:23.018426    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9545da26-bbzlv" event={"ID":"d5206e1f-8f34-4035-b26f-3ba2cc158edd","Type":"ContainerDied","Data":"b818184dc0f6f3af2be3a9517c8f2a82b7c9b3996c08f3e2742c9e70969093e0"}
Oct 31 12:39:23.019583 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:23.018475    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b818184dc0f6f3af2be3a9517c8f2a82b7c9b3996c08f3e2742c9e70969093e0"
Oct 31 12:39:23.028071 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:23.028002    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-r8l72"]
Oct 31 12:39:23.030272 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:39:23.030247    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc765a671_664f_40d2_9b62_67c746c397d0.slice/crio-c436d512f8983338a9fc0597b49cd83845e5eb80ad9e212191b73c8ec5a7fceb WatchSource:0}: Error finding container c436d512f8983338a9fc0597b49cd83845e5eb80ad9e212191b73c8ec5a7fceb: Status 404 returned error can't find the container with id c436d512f8983338a9fc0597b49cd83845e5eb80ad9e212191b73c8ec5a7fceb
Oct 31 12:39:23.033081 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:23.033052    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:39:23.033145 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:23.033114    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:39:23.033145 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:23.033139    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:39:24.043539 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:24.038242    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-r8l72" event={"ID":"c765a671-664f-40d2-9b62-67c746c397d0","Type":"ContainerStarted","Data":"6ff27078d42d2f5f7081688c661ea5d22f9d7670552eddd47fd71785f09598e4"}
Oct 31 12:39:24.043539 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:24.038352    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-r8l72" event={"ID":"c765a671-664f-40d2-9b62-67c746c397d0","Type":"ContainerStarted","Data":"c436d512f8983338a9fc0597b49cd83845e5eb80ad9e212191b73c8ec5a7fceb"}
Oct 31 12:39:24.091011 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:24.090558    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-r8l72" podStartSLOduration=2.090538016 podStartE2EDuration="2.090538016s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:39:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:39:24.086925249 +0000 UTC m=+727.189992957" watchObservedRunningTime="2026-10-31 12:39:24.090538016 +0000 UTC m=+727.193605730"
Oct 31 12:39:24.739525 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:24.739270    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-r8l72"
Oct 31 12:39:24.739525 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:24.739435    5060 patch_prober.go:29] interesting pod/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-r8l72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Oct 31 12:39:24.739525 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:24.739482    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-r8l72" podUID="c765a671-664f-40d2-9b62-67c746c397d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Oct 31 12:39:25.742511 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:25.738860    5060 patch_prober.go:29] interesting pod/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-r8l72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Oct 31 12:39:25.742511 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:25.738959    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-r8l72" podUID="c765a671-664f-40d2-9b62-67c746c397d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Oct 31 12:39:26.430097 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:26.430059    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-361536d3-gk9cp"]
Oct 31 12:39:26.476945 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:26.476911    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-361536d3-gk9cp"]
Oct 31 12:39:26.483513 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:26.477264    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-361536d3-gk9cp"
Oct 31 12:39:26.510893 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:26.510862    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 12:39:26.524896 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:26.513161    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7wsmb\""
Oct 31 12:39:26.525005 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:26.513337    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 12:39:26.596539 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:26.513404    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 12:39:26.596539 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:26.588046    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vx75h\" (UniqueName: \"kubernetes.io/projected/f993e3b5-0533-45de-bf21-1797b50ac34d-kube-api-access-vx75h\") pod \"maas-api-revoke-sub-361536d3-gk9cp\" (UID: \"f993e3b5-0533-45de-bf21-1797b50ac34d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-361536d3-gk9cp"
Oct 31 12:39:26.596539 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:26.588244    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f993e3b5-0533-45de-bf21-1797b50ac34d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-361536d3-gk9cp\" (UID: \"f993e3b5-0533-45de-bf21-1797b50ac34d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-361536d3-gk9cp"
Oct 31 12:39:26.596539 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:26.588331    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f993e3b5-0533-45de-bf21-1797b50ac34d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-361536d3-gk9cp\" (UID: \"f993e3b5-0533-45de-bf21-1797b50ac34d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-361536d3-gk9cp"
Oct 31 12:39:26.697580 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:26.689194    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vx75h\" (UniqueName: \"kubernetes.io/projected/f993e3b5-0533-45de-bf21-1797b50ac34d-kube-api-access-vx75h\") pod \"maas-api-revoke-sub-361536d3-gk9cp\" (UID: \"f993e3b5-0533-45de-bf21-1797b50ac34d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-361536d3-gk9cp"
Oct 31 12:39:26.728569 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:26.724869    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f993e3b5-0533-45de-bf21-1797b50ac34d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-361536d3-gk9cp\" (UID: \"f993e3b5-0533-45de-bf21-1797b50ac34d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-361536d3-gk9cp"
Oct 31 12:39:26.728569 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:26.725766    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f993e3b5-0533-45de-bf21-1797b50ac34d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-361536d3-gk9cp\" (UID: \"f993e3b5-0533-45de-bf21-1797b50ac34d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-361536d3-gk9cp"
Oct 31 12:39:26.728569 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:26.725640    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f993e3b5-0533-45de-bf21-1797b50ac34d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-361536d3-gk9cp\" (UID: \"f993e3b5-0533-45de-bf21-1797b50ac34d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-361536d3-gk9cp"
Oct 31 12:39:26.747085 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:26.746055    5060 patch_prober.go:29] interesting pod/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-r8l72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Oct 31 12:39:26.747085 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:26.746115    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-r8l72" podUID="c765a671-664f-40d2-9b62-67c746c397d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Oct 31 12:39:26.775666 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:26.775629    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f993e3b5-0533-45de-bf21-1797b50ac34d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-361536d3-gk9cp\" (UID: \"f993e3b5-0533-45de-bf21-1797b50ac34d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-361536d3-gk9cp"
Oct 31 12:39:26.778195 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:26.778166    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vx75h\" (UniqueName: \"kubernetes.io/projected/f993e3b5-0533-45de-bf21-1797b50ac34d-kube-api-access-vx75h\") pod \"maas-api-revoke-sub-361536d3-gk9cp\" (UID: \"f993e3b5-0533-45de-bf21-1797b50ac34d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-361536d3-gk9cp"
Oct 31 12:39:26.857687 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:26.857652    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-361536d3-gk9cp"
Oct 31 12:39:27.549096 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:27.549052    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-361536d3-gk9cp"]
Oct 31 12:39:27.554195 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:39:27.554164    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf993e3b5_0533_45de_bf21_1797b50ac34d.slice/crio-04f554970861180073bdb9dfbc582200990aabdf84d864a69aab06c513c433a2 WatchSource:0}: Error finding container 04f554970861180073bdb9dfbc582200990aabdf84d864a69aab06c513c433a2: Status 404 returned error can't find the container with id 04f554970861180073bdb9dfbc582200990aabdf84d864a69aab06c513c433a2
Oct 31 12:39:27.738276 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:27.737786    5060 patch_prober.go:29] interesting pod/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-r8l72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Oct 31 12:39:27.738276 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:27.737862    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-r8l72" podUID="c765a671-664f-40d2-9b62-67c746c397d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Oct 31 12:39:28.229514 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:28.228978    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-361536d3-gk9cp" event={"ID":"f993e3b5-0533-45de-bf21-1797b50ac34d","Type":"ContainerStarted","Data":"60adbef124b5b188954d00615a395b2f99d5f6c748acbd35b0dbd48d211d40b7"}
Oct 31 12:39:28.229514 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:28.229048    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-361536d3-gk9cp" event={"ID":"f993e3b5-0533-45de-bf21-1797b50ac34d","Type":"ContainerStarted","Data":"04f554970861180073bdb9dfbc582200990aabdf84d864a69aab06c513c433a2"}
Oct 31 12:39:28.738512 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:28.737842    5060 patch_prober.go:29] interesting pod/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-r8l72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Oct 31 12:39:28.738512 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:28.737906    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-r8l72" podUID="c765a671-664f-40d2-9b62-67c746c397d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Oct 31 12:39:29.135838 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:29.135769    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-361536d3-gk9cp" podStartSLOduration=3.135750596 podStartE2EDuration="3.135750596s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:39:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:39:28.268255234 +0000 UTC m=+731.371322949" watchObservedRunningTime="2026-10-31 12:39:29.135750596 +0000 UTC m=+732.238818310"
Oct 31 12:39:29.136532 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:29.136483    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e2af189-ntpnr"]
Oct 31 12:39:29.140558 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:29.140537    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e2af189-ntpnr"
Oct 31 12:39:29.149963 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:29.149928    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e2af189-ntpnr"]
Oct 31 12:39:29.230393 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:29.230353    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wt5xz\" (UniqueName: \"kubernetes.io/projected/2aad7d1b-a9a7-41ba-8016-6a66f72f4cb3-kube-api-access-wt5xz\") pod \"maas-api-revoke-sub-4e2af189-ntpnr\" (UID: \"2aad7d1b-a9a7-41ba-8016-6a66f72f4cb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e2af189-ntpnr"
Oct 31 12:39:29.230921 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:29.230473    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2aad7d1b-a9a7-41ba-8016-6a66f72f4cb3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4e2af189-ntpnr\" (UID: \"2aad7d1b-a9a7-41ba-8016-6a66f72f4cb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e2af189-ntpnr"
Oct 31 12:39:29.230921 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:29.230531    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2aad7d1b-a9a7-41ba-8016-6a66f72f4cb3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4e2af189-ntpnr\" (UID: \"2aad7d1b-a9a7-41ba-8016-6a66f72f4cb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e2af189-ntpnr"
Oct 31 12:39:29.239620 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:29.239584    5060 generic.go:356] "Generic (PLEG): container finished" podID="f993e3b5-0533-45de-bf21-1797b50ac34d" containerID="60adbef124b5b188954d00615a395b2f99d5f6c748acbd35b0dbd48d211d40b7" exitCode=0
Oct 31 12:39:29.239836 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:29.239804    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-361536d3-gk9cp" event={"ID":"f993e3b5-0533-45de-bf21-1797b50ac34d","Type":"ContainerDied","Data":"60adbef124b5b188954d00615a395b2f99d5f6c748acbd35b0dbd48d211d40b7"}
Oct 31 12:39:29.332078 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:29.332040    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wt5xz\" (UniqueName: \"kubernetes.io/projected/2aad7d1b-a9a7-41ba-8016-6a66f72f4cb3-kube-api-access-wt5xz\") pod \"maas-api-revoke-sub-4e2af189-ntpnr\" (UID: \"2aad7d1b-a9a7-41ba-8016-6a66f72f4cb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e2af189-ntpnr"
Oct 31 12:39:29.332250 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:29.332090    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2aad7d1b-a9a7-41ba-8016-6a66f72f4cb3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4e2af189-ntpnr\" (UID: \"2aad7d1b-a9a7-41ba-8016-6a66f72f4cb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e2af189-ntpnr"
Oct 31 12:39:29.332250 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:29.332113    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2aad7d1b-a9a7-41ba-8016-6a66f72f4cb3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4e2af189-ntpnr\" (UID: \"2aad7d1b-a9a7-41ba-8016-6a66f72f4cb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e2af189-ntpnr"
Oct 31 12:39:29.333010 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:29.332982    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2aad7d1b-a9a7-41ba-8016-6a66f72f4cb3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4e2af189-ntpnr\" (UID: \"2aad7d1b-a9a7-41ba-8016-6a66f72f4cb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e2af189-ntpnr"
Oct 31 12:39:29.335117 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:29.335095    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2aad7d1b-a9a7-41ba-8016-6a66f72f4cb3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4e2af189-ntpnr\" (UID: \"2aad7d1b-a9a7-41ba-8016-6a66f72f4cb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e2af189-ntpnr"
Oct 31 12:39:29.344411 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:29.344381    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wt5xz\" (UniqueName: \"kubernetes.io/projected/2aad7d1b-a9a7-41ba-8016-6a66f72f4cb3-kube-api-access-wt5xz\") pod \"maas-api-revoke-sub-4e2af189-ntpnr\" (UID: \"2aad7d1b-a9a7-41ba-8016-6a66f72f4cb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e2af189-ntpnr"
Oct 31 12:39:29.458266 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:29.458175    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e2af189-ntpnr"
Oct 31 12:39:29.759257 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:29.759228    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e2af189-ntpnr"]
Oct 31 12:39:29.761153 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:39:29.761120    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2aad7d1b_a9a7_41ba_8016_6a66f72f4cb3.slice/crio-e32169ef8899e3f34600cc591c28761390e87653180ed065760f52e8ff4a7b9b WatchSource:0}: Error finding container e32169ef8899e3f34600cc591c28761390e87653180ed065760f52e8ff4a7b9b: Status 404 returned error can't find the container with id e32169ef8899e3f34600cc591c28761390e87653180ed065760f52e8ff4a7b9b
Oct 31 12:39:29.924650 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:39:29.924612    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7phb" containerName="extractor"
Oct 31 12:39:29.961156 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:29.961122    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-r8l72"
Oct 31 12:39:29.961543 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:29.961518    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-r8l72"
Oct 31 12:39:29.962315 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:29.962290    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-r8l72"
Oct 31 12:39:30.249875 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:30.249834    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e2af189-ntpnr" event={"ID":"2aad7d1b-a9a7-41ba-8016-6a66f72f4cb3","Type":"ContainerStarted","Data":"eb11c6394e5500487e31ba4aec826f0dabf80bc3cf568948907ae57effedb6a2"}
Oct 31 12:39:30.250348 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:30.249918    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e2af189-ntpnr" event={"ID":"2aad7d1b-a9a7-41ba-8016-6a66f72f4cb3","Type":"ContainerStarted","Data":"e32169ef8899e3f34600cc591c28761390e87653180ed065760f52e8ff4a7b9b"}
Oct 31 12:39:30.272004 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:30.271955    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e2af189-ntpnr" podStartSLOduration=1.271941555 podStartE2EDuration="1.271941555s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:39:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:39:30.269092865 +0000 UTC m=+733.372160577" watchObservedRunningTime="2026-10-31 12:39:30.271941555 +0000 UTC m=+733.375009267"
Oct 31 12:39:30.422886 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:30.422857    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-361536d3-gk9cp"
Oct 31 12:39:30.457118 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:30.457086    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f993e3b5-0533-45de-bf21-1797b50ac34d-kube-api-access-vx75h\" (UniqueName: \"kubernetes.io/projected/f993e3b5-0533-45de-bf21-1797b50ac34d-kube-api-access-vx75h\") pod \"f993e3b5-0533-45de-bf21-1797b50ac34d\" (UID: \"f993e3b5-0533-45de-bf21-1797b50ac34d\") "
Oct 31 12:39:30.457281 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:30.457232    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f993e3b5-0533-45de-bf21-1797b50ac34d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f993e3b5-0533-45de-bf21-1797b50ac34d-maas-api-openshift-ca\") pod \"f993e3b5-0533-45de-bf21-1797b50ac34d\" (UID: \"f993e3b5-0533-45de-bf21-1797b50ac34d\") "
Oct 31 12:39:30.457400 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:30.457370    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f993e3b5-0533-45de-bf21-1797b50ac34d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f993e3b5-0533-45de-bf21-1797b50ac34d-maas-api-xks-ca\") pod \"f993e3b5-0533-45de-bf21-1797b50ac34d\" (UID: \"f993e3b5-0533-45de-bf21-1797b50ac34d\") "
Oct 31 12:39:30.458143 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:30.458111    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f993e3b5-0533-45de-bf21-1797b50ac34d-maas-api-openshift-ca" pod "f993e3b5-0533-45de-bf21-1797b50ac34d" (UID: "f993e3b5-0533-45de-bf21-1797b50ac34d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:39:30.461653 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:30.459927    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f993e3b5-0533-45de-bf21-1797b50ac34d-kube-api-access-vx75h" pod "f993e3b5-0533-45de-bf21-1797b50ac34d" (UID: "f993e3b5-0533-45de-bf21-1797b50ac34d"). InnerVolumeSpecName "kube-api-access-vx75h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:39:30.461653 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:30.460202    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f993e3b5-0533-45de-bf21-1797b50ac34d-maas-api-xks-ca" pod "f993e3b5-0533-45de-bf21-1797b50ac34d" (UID: "f993e3b5-0533-45de-bf21-1797b50ac34d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:39:30.558820 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:30.558785    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f993e3b5-0533-45de-bf21-1797b50ac34d-maas-api-xks-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:39:30.558820 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:30.558814    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vx75h\" (UniqueName: \"kubernetes.io/projected/f993e3b5-0533-45de-bf21-1797b50ac34d-kube-api-access-vx75h\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:39:30.558820 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:30.558826    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f993e3b5-0533-45de-bf21-1797b50ac34d-maas-api-openshift-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:39:31.259649 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:31.259614    5060 generic.go:356] "Generic (PLEG): container finished" podID="2aad7d1b-a9a7-41ba-8016-6a66f72f4cb3" containerID="eb11c6394e5500487e31ba4aec826f0dabf80bc3cf568948907ae57effedb6a2" exitCode=0
Oct 31 12:39:31.260131 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:31.259695    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e2af189-ntpnr" event={"ID":"2aad7d1b-a9a7-41ba-8016-6a66f72f4cb3","Type":"ContainerDied","Data":"eb11c6394e5500487e31ba4aec826f0dabf80bc3cf568948907ae57effedb6a2"}
Oct 31 12:39:31.261352 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:31.261332    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-361536d3-gk9cp"
Oct 31 12:39:31.261508 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:31.261346    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-361536d3-gk9cp" event={"ID":"f993e3b5-0533-45de-bf21-1797b50ac34d","Type":"ContainerDied","Data":"04f554970861180073bdb9dfbc582200990aabdf84d864a69aab06c513c433a2"}
Oct 31 12:39:31.261508 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:31.261397    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="04f554970861180073bdb9dfbc582200990aabdf84d864a69aab06c513c433a2"
Oct 31 12:39:32.316712 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:32.316673    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08422da2-298fq"]
Oct 31 12:39:32.317430 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:32.317403    5060 cpu_manager.go:414] "containerMap: removing container" podUID="f993e3b5-0533-45de-bf21-1797b50ac34d" containerName="revoke-keys"
Oct 31 12:39:32.317430 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:32.317424    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f993e3b5-0533-45de-bf21-1797b50ac34d" containerName="revoke-keys"
Oct 31 12:39:32.317608 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:32.317594    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="f993e3b5-0533-45de-bf21-1797b50ac34d" containerName="revoke-keys"
Oct 31 12:39:32.321406 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:32.321385    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08422da2-298fq"
Oct 31 12:39:32.347793 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:32.347757    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08422da2-298fq"]
Oct 31 12:39:32.377398 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:32.377359    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9dde715b-q58kx"]
Oct 31 12:39:32.380364 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:32.380014    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44hd8\" (UniqueName: \"kubernetes.io/projected/ed4c92cc-3b80-4016-bdb3-6092fb0ecfdd-kube-api-access-44hd8\") pod \"maas-api-revoke-sub-08422da2-298fq\" (UID: \"ed4c92cc-3b80-4016-bdb3-6092fb0ecfdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08422da2-298fq"
Oct 31 12:39:32.380364 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:32.380077    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed4c92cc-3b80-4016-bdb3-6092fb0ecfdd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-08422da2-298fq\" (UID: \"ed4c92cc-3b80-4016-bdb3-6092fb0ecfdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08422da2-298fq"
Oct 31 12:39:32.380364 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:32.380215    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed4c92cc-3b80-4016-bdb3-6092fb0ecfdd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-08422da2-298fq\" (UID: \"ed4c92cc-3b80-4016-bdb3-6092fb0ecfdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08422da2-298fq"
Oct 31 12:39:32.383830 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:32.383782    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dde715b-q58kx"
Oct 31 12:39:32.389597 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:32.389572    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9dde715b-q58kx"]
Oct 31 12:39:32.452418 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:32.452386    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e2af189-ntpnr"
Oct 31 12:39:32.481360 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:32.481330    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed4c92cc-3b80-4016-bdb3-6092fb0ecfdd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-08422da2-298fq\" (UID: \"ed4c92cc-3b80-4016-bdb3-6092fb0ecfdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08422da2-298fq"
Oct 31 12:39:32.481562 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:32.481434    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29aae726-4558-4b6d-ab74-770c96cac741-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9dde715b-q58kx\" (UID: \"29aae726-4558-4b6d-ab74-770c96cac741\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dde715b-q58kx"
Oct 31 12:39:32.481562 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:32.481529    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44hd8\" (UniqueName: \"kubernetes.io/projected/ed4c92cc-3b80-4016-bdb3-6092fb0ecfdd-kube-api-access-44hd8\") pod \"maas-api-revoke-sub-08422da2-298fq\" (UID: \"ed4c92cc-3b80-4016-bdb3-6092fb0ecfdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08422da2-298fq"
Oct 31 12:39:32.481700 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:32.481639    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29aae726-4558-4b6d-ab74-770c96cac741-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9dde715b-q58kx\" (UID: \"29aae726-4558-4b6d-ab74-770c96cac741\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dde715b-q58kx"
Oct 31 12:39:32.481700 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:32.481688    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed4c92cc-3b80-4016-bdb3-6092fb0ecfdd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-08422da2-298fq\" (UID: \"ed4c92cc-3b80-4016-bdb3-6092fb0ecfdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08422da2-298fq"
Oct 31 12:39:32.481800 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:32.481732    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhr98\" (UniqueName: \"kubernetes.io/projected/29aae726-4558-4b6d-ab74-770c96cac741-kube-api-access-nhr98\") pod \"maas-api-revoke-sub-9dde715b-q58kx\" (UID: \"29aae726-4558-4b6d-ab74-770c96cac741\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dde715b-q58kx"
Oct 31 12:39:32.482443 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:32.482407    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed4c92cc-3b80-4016-bdb3-6092fb0ecfdd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-08422da2-298fq\" (UID: \"ed4c92cc-3b80-4016-bdb3-6092fb0ecfdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08422da2-298fq"
Oct 31 12:39:32.484571 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:32.484549    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed4c92cc-3b80-4016-bdb3-6092fb0ecfdd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-08422da2-298fq\" (UID: \"ed4c92cc-3b80-4016-bdb3-6092fb0ecfdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08422da2-298fq"
Oct 31 12:39:32.494000 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:32.493970    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-44hd8\" (UniqueName: \"kubernetes.io/projected/ed4c92cc-3b80-4016-bdb3-6092fb0ecfdd-kube-api-access-44hd8\") pod \"maas-api-revoke-sub-08422da2-298fq\" (UID: \"ed4c92cc-3b80-4016-bdb3-6092fb0ecfdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08422da2-298fq"
Oct 31 12:39:32.582805 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:32.582765    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2aad7d1b-a9a7-41ba-8016-6a66f72f4cb3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2aad7d1b-a9a7-41ba-8016-6a66f72f4cb3-maas-api-xks-ca\") pod \"2aad7d1b-a9a7-41ba-8016-6a66f72f4cb3\" (UID: \"2aad7d1b-a9a7-41ba-8016-6a66f72f4cb3\") "
Oct 31 12:39:32.582979 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:32.582831    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2aad7d1b-a9a7-41ba-8016-6a66f72f4cb3-kube-api-access-wt5xz\" (UniqueName: \"kubernetes.io/projected/2aad7d1b-a9a7-41ba-8016-6a66f72f4cb3-kube-api-access-wt5xz\") pod \"2aad7d1b-a9a7-41ba-8016-6a66f72f4cb3\" (UID: \"2aad7d1b-a9a7-41ba-8016-6a66f72f4cb3\") "
Oct 31 12:39:32.583059 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:32.583034    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2aad7d1b-a9a7-41ba-8016-6a66f72f4cb3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2aad7d1b-a9a7-41ba-8016-6a66f72f4cb3-maas-api-openshift-ca\") pod \"2aad7d1b-a9a7-41ba-8016-6a66f72f4cb3\" (UID: \"2aad7d1b-a9a7-41ba-8016-6a66f72f4cb3\") "
Oct 31 12:39:32.583299 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:32.583272    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29aae726-4558-4b6d-ab74-770c96cac741-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9dde715b-q58kx\" (UID: \"29aae726-4558-4b6d-ab74-770c96cac741\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dde715b-q58kx"
Oct 31 12:39:32.583436 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:32.583362    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2aad7d1b-a9a7-41ba-8016-6a66f72f4cb3-maas-api-openshift-ca" pod "2aad7d1b-a9a7-41ba-8016-6a66f72f4cb3" (UID: "2aad7d1b-a9a7-41ba-8016-6a66f72f4cb3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:39:32.583436 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:32.583374    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29aae726-4558-4b6d-ab74-770c96cac741-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9dde715b-q58kx\" (UID: \"29aae726-4558-4b6d-ab74-770c96cac741\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dde715b-q58kx"
Oct 31 12:39:32.583572 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:32.583462    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhr98\" (UniqueName: \"kubernetes.io/projected/29aae726-4558-4b6d-ab74-770c96cac741-kube-api-access-nhr98\") pod \"maas-api-revoke-sub-9dde715b-q58kx\" (UID: \"29aae726-4558-4b6d-ab74-770c96cac741\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dde715b-q58kx"
Oct 31 12:39:32.583684 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:32.583662    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2aad7d1b-a9a7-41ba-8016-6a66f72f4cb3-maas-api-openshift-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:39:32.584105 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:32.584079    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29aae726-4558-4b6d-ab74-770c96cac741-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9dde715b-q58kx\" (UID: \"29aae726-4558-4b6d-ab74-770c96cac741\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dde715b-q58kx"
Oct 31 12:39:32.585095 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:32.585052    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2aad7d1b-a9a7-41ba-8016-6a66f72f4cb3-maas-api-xks-ca" pod "2aad7d1b-a9a7-41ba-8016-6a66f72f4cb3" (UID: "2aad7d1b-a9a7-41ba-8016-6a66f72f4cb3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:39:32.585193 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:32.585106    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2aad7d1b-a9a7-41ba-8016-6a66f72f4cb3-kube-api-access-wt5xz" pod "2aad7d1b-a9a7-41ba-8016-6a66f72f4cb3" (UID: "2aad7d1b-a9a7-41ba-8016-6a66f72f4cb3"). InnerVolumeSpecName "kube-api-access-wt5xz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:39:32.585850 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:32.585833    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29aae726-4558-4b6d-ab74-770c96cac741-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9dde715b-q58kx\" (UID: \"29aae726-4558-4b6d-ab74-770c96cac741\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dde715b-q58kx"
Oct 31 12:39:32.596239 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:32.596213    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhr98\" (UniqueName: \"kubernetes.io/projected/29aae726-4558-4b6d-ab74-770c96cac741-kube-api-access-nhr98\") pod \"maas-api-revoke-sub-9dde715b-q58kx\" (UID: \"29aae726-4558-4b6d-ab74-770c96cac741\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dde715b-q58kx"
Oct 31 12:39:32.655231 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:32.655193    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08422da2-298fq"
Oct 31 12:39:32.684423 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:32.684383    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2aad7d1b-a9a7-41ba-8016-6a66f72f4cb3-maas-api-xks-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:39:32.684423 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:32.684419    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wt5xz\" (UniqueName: \"kubernetes.io/projected/2aad7d1b-a9a7-41ba-8016-6a66f72f4cb3-kube-api-access-wt5xz\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:39:32.704248 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:32.704209    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dde715b-q58kx"
Oct 31 12:39:32.947779 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:32.947750    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08422da2-298fq"]
Oct 31 12:39:32.948551 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:39:32.948517    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded4c92cc_3b80_4016_bdb3_6092fb0ecfdd.slice/crio-2e9c9eaa22a147fafa4897b4c486b28bde08eb85f2fc025b5319399efdfbfb81 WatchSource:0}: Error finding container 2e9c9eaa22a147fafa4897b4c486b28bde08eb85f2fc025b5319399efdfbfb81: Status 404 returned error can't find the container with id 2e9c9eaa22a147fafa4897b4c486b28bde08eb85f2fc025b5319399efdfbfb81
Oct 31 12:39:33.022359 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:33.022328    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9dde715b-q58kx"]
Oct 31 12:39:33.022913 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:39:33.022888    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29aae726_4558_4b6d_ab74_770c96cac741.slice/crio-2460cd889c1b48a42d6b859042c7b30fd5420bdd2f38ffb217ce9cb8c09d5595 WatchSource:0}: Error finding container 2460cd889c1b48a42d6b859042c7b30fd5420bdd2f38ffb217ce9cb8c09d5595: Status 404 returned error can't find the container with id 2460cd889c1b48a42d6b859042c7b30fd5420bdd2f38ffb217ce9cb8c09d5595
Oct 31 12:39:33.279611 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:33.279084    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dde715b-q58kx" event={"ID":"29aae726-4558-4b6d-ab74-770c96cac741","Type":"ContainerStarted","Data":"2460cd889c1b48a42d6b859042c7b30fd5420bdd2f38ffb217ce9cb8c09d5595"}
Oct 31 12:39:33.281124 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:33.281097    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e2af189-ntpnr" event={"ID":"2aad7d1b-a9a7-41ba-8016-6a66f72f4cb3","Type":"ContainerDied","Data":"e32169ef8899e3f34600cc591c28761390e87653180ed065760f52e8ff4a7b9b"}
Oct 31 12:39:33.281256 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:33.281145    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e32169ef8899e3f34600cc591c28761390e87653180ed065760f52e8ff4a7b9b"
Oct 31 12:39:33.281256 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:33.281219    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e2af189-ntpnr"
Oct 31 12:39:33.283297 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:33.283237    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08422da2-298fq" event={"ID":"ed4c92cc-3b80-4016-bdb3-6092fb0ecfdd","Type":"ContainerStarted","Data":"2e9c9eaa22a147fafa4897b4c486b28bde08eb85f2fc025b5319399efdfbfb81"}
Oct 31 12:39:34.046056 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:34.046021    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9384727b-z9rvm"]
Oct 31 12:39:34.046589 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:34.046575    5060 cpu_manager.go:414] "containerMap: removing container" podUID="2aad7d1b-a9a7-41ba-8016-6a66f72f4cb3" containerName="revoke-keys"
Oct 31 12:39:34.046643 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:34.046590    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2aad7d1b-a9a7-41ba-8016-6a66f72f4cb3" containerName="revoke-keys"
Oct 31 12:39:34.046718 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:34.046709    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="2aad7d1b-a9a7-41ba-8016-6a66f72f4cb3" containerName="revoke-keys"
Oct 31 12:39:34.050194 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:34.050170    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9384727b-z9rvm"
Oct 31 12:39:34.062041 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:34.061081    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9384727b-z9rvm"]
Oct 31 12:39:34.106795 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:34.106759    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5f2l2\" (UniqueName: \"kubernetes.io/projected/66856f9b-041f-4251-997c-be59be05384c-kube-api-access-5f2l2\") pod \"maas-api-revoke-sub-9384727b-z9rvm\" (UID: \"66856f9b-041f-4251-997c-be59be05384c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9384727b-z9rvm"
Oct 31 12:39:34.106967 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:34.106819    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66856f9b-041f-4251-997c-be59be05384c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9384727b-z9rvm\" (UID: \"66856f9b-041f-4251-997c-be59be05384c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9384727b-z9rvm"
Oct 31 12:39:34.106967 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:34.106922    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66856f9b-041f-4251-997c-be59be05384c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9384727b-z9rvm\" (UID: \"66856f9b-041f-4251-997c-be59be05384c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9384727b-z9rvm"
Oct 31 12:39:34.207859 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:34.207823    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5f2l2\" (UniqueName: \"kubernetes.io/projected/66856f9b-041f-4251-997c-be59be05384c-kube-api-access-5f2l2\") pod \"maas-api-revoke-sub-9384727b-z9rvm\" (UID: \"66856f9b-041f-4251-997c-be59be05384c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9384727b-z9rvm"
Oct 31 12:39:34.208027 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:34.207887    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66856f9b-041f-4251-997c-be59be05384c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9384727b-z9rvm\" (UID: \"66856f9b-041f-4251-997c-be59be05384c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9384727b-z9rvm"
Oct 31 12:39:34.208027 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:34.207924    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66856f9b-041f-4251-997c-be59be05384c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9384727b-z9rvm\" (UID: \"66856f9b-041f-4251-997c-be59be05384c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9384727b-z9rvm"
Oct 31 12:39:34.208641 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:34.208616    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66856f9b-041f-4251-997c-be59be05384c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9384727b-z9rvm\" (UID: \"66856f9b-041f-4251-997c-be59be05384c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9384727b-z9rvm"
Oct 31 12:39:34.210442 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:34.210412    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66856f9b-041f-4251-997c-be59be05384c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9384727b-z9rvm\" (UID: \"66856f9b-041f-4251-997c-be59be05384c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9384727b-z9rvm"
Oct 31 12:39:34.219890 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:34.219826    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5f2l2\" (UniqueName: \"kubernetes.io/projected/66856f9b-041f-4251-997c-be59be05384c-kube-api-access-5f2l2\") pod \"maas-api-revoke-sub-9384727b-z9rvm\" (UID: \"66856f9b-041f-4251-997c-be59be05384c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9384727b-z9rvm"
Oct 31 12:39:34.293767 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:34.293729    5060 generic.go:356] "Generic (PLEG): container finished" podID="29aae726-4558-4b6d-ab74-770c96cac741" containerID="958a9da68ee1eaa86eb1aa54cd25322fe09534702a7b3935b0868ccb6d47fbe8" exitCode=0
Oct 31 12:39:34.293959 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:34.293812    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dde715b-q58kx" event={"ID":"29aae726-4558-4b6d-ab74-770c96cac741","Type":"ContainerDied","Data":"958a9da68ee1eaa86eb1aa54cd25322fe09534702a7b3935b0868ccb6d47fbe8"}
Oct 31 12:39:34.295686 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:34.295654    5060 generic.go:356] "Generic (PLEG): container finished" podID="ed4c92cc-3b80-4016-bdb3-6092fb0ecfdd" containerID="efc110ed18fc5e62df112f661b077fcb7d25c830d7c26bb00fc648c666d99570" exitCode=0
Oct 31 12:39:34.295822 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:34.295739    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08422da2-298fq" event={"ID":"ed4c92cc-3b80-4016-bdb3-6092fb0ecfdd","Type":"ContainerDied","Data":"efc110ed18fc5e62df112f661b077fcb7d25c830d7c26bb00fc648c666d99570"}
Oct 31 12:39:34.427762 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:34.427729    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9384727b-z9rvm"
Oct 31 12:39:34.714356 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:34.714323    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9384727b-z9rvm"]
Oct 31 12:39:34.715792 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:39:34.715761    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66856f9b_041f_4251_997c_be59be05384c.slice/crio-de9348441fc15184c9e6551fdac5e137301f48011f3c63549ad91ef7be36ce7e WatchSource:0}: Error finding container de9348441fc15184c9e6551fdac5e137301f48011f3c63549ad91ef7be36ce7e: Status 404 returned error can't find the container with id de9348441fc15184c9e6551fdac5e137301f48011f3c63549ad91ef7be36ce7e
Oct 31 12:39:35.305230 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:35.305187    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9384727b-z9rvm" event={"ID":"66856f9b-041f-4251-997c-be59be05384c","Type":"ContainerStarted","Data":"8311b665530e2a7cca9867d0458a473402c5caeec7a027460d7b0cf1a6d09422"}
Oct 31 12:39:35.305772 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:35.305258    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9384727b-z9rvm" event={"ID":"66856f9b-041f-4251-997c-be59be05384c","Type":"ContainerStarted","Data":"de9348441fc15184c9e6551fdac5e137301f48011f3c63549ad91ef7be36ce7e"}
Oct 31 12:39:35.326543 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:35.326409    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9384727b-z9rvm" podStartSLOduration=1.3263937590000001 podStartE2EDuration="1.326393759s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:39:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:39:35.323692119 +0000 UTC m=+738.426759833" watchObservedRunningTime="2026-10-31 12:39:35.326393759 +0000 UTC m=+738.429461497"
Oct 31 12:39:35.495560 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:35.495536    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dde715b-q58kx"
Oct 31 12:39:35.528952 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:35.527852    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08422da2-298fq"
Oct 31 12:39:35.626806 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:35.626728    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29aae726-4558-4b6d-ab74-770c96cac741-kube-api-access-nhr98\" (UniqueName: \"kubernetes.io/projected/29aae726-4558-4b6d-ab74-770c96cac741-kube-api-access-nhr98\") pod \"29aae726-4558-4b6d-ab74-770c96cac741\" (UID: \"29aae726-4558-4b6d-ab74-770c96cac741\") "
Oct 31 12:39:35.626958 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:35.626812    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/29aae726-4558-4b6d-ab74-770c96cac741-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29aae726-4558-4b6d-ab74-770c96cac741-maas-api-xks-ca\") pod \"29aae726-4558-4b6d-ab74-770c96cac741\" (UID: \"29aae726-4558-4b6d-ab74-770c96cac741\") "
Oct 31 12:39:35.626958 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:35.626859    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed4c92cc-3b80-4016-bdb3-6092fb0ecfdd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed4c92cc-3b80-4016-bdb3-6092fb0ecfdd-maas-api-xks-ca\") pod \"ed4c92cc-3b80-4016-bdb3-6092fb0ecfdd\" (UID: \"ed4c92cc-3b80-4016-bdb3-6092fb0ecfdd\") "
Oct 31 12:39:35.626958 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:35.626902    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed4c92cc-3b80-4016-bdb3-6092fb0ecfdd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed4c92cc-3b80-4016-bdb3-6092fb0ecfdd-maas-api-openshift-ca\") pod \"ed4c92cc-3b80-4016-bdb3-6092fb0ecfdd\" (UID: \"ed4c92cc-3b80-4016-bdb3-6092fb0ecfdd\") "
Oct 31 12:39:35.627100 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:35.626998    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed4c92cc-3b80-4016-bdb3-6092fb0ecfdd-kube-api-access-44hd8\" (UniqueName: \"kubernetes.io/projected/ed4c92cc-3b80-4016-bdb3-6092fb0ecfdd-kube-api-access-44hd8\") pod \"ed4c92cc-3b80-4016-bdb3-6092fb0ecfdd\" (UID: \"ed4c92cc-3b80-4016-bdb3-6092fb0ecfdd\") "
Oct 31 12:39:35.627100 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:35.627090    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29aae726-4558-4b6d-ab74-770c96cac741-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29aae726-4558-4b6d-ab74-770c96cac741-maas-api-openshift-ca\") pod \"29aae726-4558-4b6d-ab74-770c96cac741\" (UID: \"29aae726-4558-4b6d-ab74-770c96cac741\") "
Oct 31 12:39:35.627474 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:35.627369    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed4c92cc-3b80-4016-bdb3-6092fb0ecfdd-maas-api-openshift-ca" pod "ed4c92cc-3b80-4016-bdb3-6092fb0ecfdd" (UID: "ed4c92cc-3b80-4016-bdb3-6092fb0ecfdd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:39:35.627633 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:35.627543    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29aae726-4558-4b6d-ab74-770c96cac741-maas-api-openshift-ca" pod "29aae726-4558-4b6d-ab74-770c96cac741" (UID: "29aae726-4558-4b6d-ab74-770c96cac741"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:39:35.627633 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:35.627604    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed4c92cc-3b80-4016-bdb3-6092fb0ecfdd-maas-api-openshift-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:39:35.629057 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:35.629035    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed4c92cc-3b80-4016-bdb3-6092fb0ecfdd-maas-api-xks-ca" pod "ed4c92cc-3b80-4016-bdb3-6092fb0ecfdd" (UID: "ed4c92cc-3b80-4016-bdb3-6092fb0ecfdd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:39:35.629187 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:35.629167    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed4c92cc-3b80-4016-bdb3-6092fb0ecfdd-kube-api-access-44hd8" pod "ed4c92cc-3b80-4016-bdb3-6092fb0ecfdd" (UID: "ed4c92cc-3b80-4016-bdb3-6092fb0ecfdd"). InnerVolumeSpecName "kube-api-access-44hd8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:39:35.629355 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:35.629329    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29aae726-4558-4b6d-ab74-770c96cac741-maas-api-xks-ca" pod "29aae726-4558-4b6d-ab74-770c96cac741" (UID: "29aae726-4558-4b6d-ab74-770c96cac741"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:39:35.629448 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:35.629362    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29aae726-4558-4b6d-ab74-770c96cac741-kube-api-access-nhr98" pod "29aae726-4558-4b6d-ab74-770c96cac741" (UID: "29aae726-4558-4b6d-ab74-770c96cac741"). InnerVolumeSpecName "kube-api-access-nhr98". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:39:35.728985 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:35.728948    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhr98\" (UniqueName: \"kubernetes.io/projected/29aae726-4558-4b6d-ab74-770c96cac741-kube-api-access-nhr98\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:39:35.728985 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:35.728979    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29aae726-4558-4b6d-ab74-770c96cac741-maas-api-xks-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:39:35.728985 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:35.728990    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed4c92cc-3b80-4016-bdb3-6092fb0ecfdd-maas-api-xks-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:39:35.729212 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:35.728999    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-44hd8\" (UniqueName: \"kubernetes.io/projected/ed4c92cc-3b80-4016-bdb3-6092fb0ecfdd-kube-api-access-44hd8\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:39:35.729212 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:35.729008    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29aae726-4558-4b6d-ab74-770c96cac741-maas-api-openshift-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:39:36.314199 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:36.314164    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dde715b-q58kx"
Oct 31 12:39:36.314199 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:36.314171    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dde715b-q58kx" event={"ID":"29aae726-4558-4b6d-ab74-770c96cac741","Type":"ContainerDied","Data":"2460cd889c1b48a42d6b859042c7b30fd5420bdd2f38ffb217ce9cb8c09d5595"}
Oct 31 12:39:36.314673 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:36.314221    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2460cd889c1b48a42d6b859042c7b30fd5420bdd2f38ffb217ce9cb8c09d5595"
Oct 31 12:39:36.315747 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:36.315714    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08422da2-298fq"
Oct 31 12:39:36.315747 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:36.315733    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08422da2-298fq" event={"ID":"ed4c92cc-3b80-4016-bdb3-6092fb0ecfdd","Type":"ContainerDied","Data":"2e9c9eaa22a147fafa4897b4c486b28bde08eb85f2fc025b5319399efdfbfb81"}
Oct 31 12:39:36.315936 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:36.315776    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2e9c9eaa22a147fafa4897b4c486b28bde08eb85f2fc025b5319399efdfbfb81"
Oct 31 12:39:36.317387 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:36.317366    5060 generic.go:356] "Generic (PLEG): container finished" podID="66856f9b-041f-4251-997c-be59be05384c" containerID="8311b665530e2a7cca9867d0458a473402c5caeec7a027460d7b0cf1a6d09422" exitCode=0
Oct 31 12:39:36.317532 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:36.317454    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9384727b-z9rvm" event={"ID":"66856f9b-041f-4251-997c-be59be05384c","Type":"ContainerDied","Data":"8311b665530e2a7cca9867d0458a473402c5caeec7a027460d7b0cf1a6d09422"}
Oct 31 12:39:37.513644 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:37.513618    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9384727b-z9rvm"
Oct 31 12:39:37.521107 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:37.521081    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-ffvcd"]
Oct 31 12:39:37.521702 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:37.521684    5060 cpu_manager.go:414] "containerMap: removing container" podUID="ed4c92cc-3b80-4016-bdb3-6092fb0ecfdd" containerName="revoke-keys"
Oct 31 12:39:37.521798 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:37.521703    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed4c92cc-3b80-4016-bdb3-6092fb0ecfdd" containerName="revoke-keys"
Oct 31 12:39:37.521798 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:37.521718    5060 cpu_manager.go:414] "containerMap: removing container" podUID="29aae726-4558-4b6d-ab74-770c96cac741" containerName="revoke-keys"
Oct 31 12:39:37.521798 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:37.521725    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29aae726-4558-4b6d-ab74-770c96cac741" containerName="revoke-keys"
Oct 31 12:39:37.521798 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:37.521735    5060 cpu_manager.go:414] "containerMap: removing container" podUID="66856f9b-041f-4251-997c-be59be05384c" containerName="revoke-keys"
Oct 31 12:39:37.521798 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:37.521742    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66856f9b-041f-4251-997c-be59be05384c" containerName="revoke-keys"
Oct 31 12:39:37.522039 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:37.521881    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="29aae726-4558-4b6d-ab74-770c96cac741" containerName="revoke-keys"
Oct 31 12:39:37.522039 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:37.521895    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed4c92cc-3b80-4016-bdb3-6092fb0ecfdd" containerName="revoke-keys"
Oct 31 12:39:37.522039 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:37.521904    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="66856f9b-041f-4251-997c-be59be05384c" containerName="revoke-keys"
Oct 31 12:39:37.525316 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:37.525298    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-ffvcd"
Oct 31 12:39:37.529039 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:37.529020    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-b25d3b-openshift-default-dockercfg-5r5zv\""
Oct 31 12:39:37.538855 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:37.538833    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-ffvcd"]
Oct 31 12:39:37.650614 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:37.650531    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66856f9b-041f-4251-997c-be59be05384c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66856f9b-041f-4251-997c-be59be05384c-maas-api-openshift-ca\") pod \"66856f9b-041f-4251-997c-be59be05384c\" (UID: \"66856f9b-041f-4251-997c-be59be05384c\") "
Oct 31 12:39:37.650782 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:37.650760    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66856f9b-041f-4251-997c-be59be05384c-kube-api-access-5f2l2\" (UniqueName: \"kubernetes.io/projected/66856f9b-041f-4251-997c-be59be05384c-kube-api-access-5f2l2\") pod \"66856f9b-041f-4251-997c-be59be05384c\" (UID: \"66856f9b-041f-4251-997c-be59be05384c\") "
Oct 31 12:39:37.650859 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:37.650805    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/66856f9b-041f-4251-997c-be59be05384c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66856f9b-041f-4251-997c-be59be05384c-maas-api-xks-ca\") pod \"66856f9b-041f-4251-997c-be59be05384c\" (UID: \"66856f9b-041f-4251-997c-be59be05384c\") "
Oct 31 12:39:37.650916 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:37.650872    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66856f9b-041f-4251-997c-be59be05384c-maas-api-openshift-ca" pod "66856f9b-041f-4251-997c-be59be05384c" (UID: "66856f9b-041f-4251-997c-be59be05384c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:39:37.651038 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:37.651019    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a72336d5-83db-484a-a300-de68dc5a37c4-workload-certs\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-ffvcd\" (UID: \"a72336d5-83db-484a-a300-de68dc5a37c4\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-ffvcd"
Oct 31 12:39:37.651102 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:37.651085    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25tcp\" (UniqueName: \"kubernetes.io/projected/a72336d5-83db-484a-a300-de68dc5a37c4-kube-api-access-25tcp\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-ffvcd\" (UID: \"a72336d5-83db-484a-a300-de68dc5a37c4\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-ffvcd"
Oct 31 12:39:37.651249 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:37.651220    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a72336d5-83db-484a-a300-de68dc5a37c4-workload-socket\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-ffvcd\" (UID: \"a72336d5-83db-484a-a300-de68dc5a37c4\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-ffvcd"
Oct 31 12:39:37.651309 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:37.651276    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a72336d5-83db-484a-a300-de68dc5a37c4-istio-envoy\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-ffvcd\" (UID: \"a72336d5-83db-484a-a300-de68dc5a37c4\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-ffvcd"
Oct 31 12:39:37.651373 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:37.651351    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a72336d5-83db-484a-a300-de68dc5a37c4-istio-data\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-ffvcd\" (UID: \"a72336d5-83db-484a-a300-de68dc5a37c4\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-ffvcd"
Oct 31 12:39:37.651459 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:37.651443    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a72336d5-83db-484a-a300-de68dc5a37c4-credential-socket\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-ffvcd\" (UID: \"a72336d5-83db-484a-a300-de68dc5a37c4\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-ffvcd"
Oct 31 12:39:37.651528 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:37.651471    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a72336d5-83db-484a-a300-de68dc5a37c4-istio-token\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-ffvcd\" (UID: \"a72336d5-83db-484a-a300-de68dc5a37c4\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-ffvcd"
Oct 31 12:39:37.651597 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:37.651569    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a72336d5-83db-484a-a300-de68dc5a37c4-istiod-ca-cert\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-ffvcd\" (UID: \"a72336d5-83db-484a-a300-de68dc5a37c4\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-ffvcd"
Oct 31 12:39:37.651655 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:37.651613    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a72336d5-83db-484a-a300-de68dc5a37c4-istio-podinfo\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-ffvcd\" (UID: \"a72336d5-83db-484a-a300-de68dc5a37c4\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-ffvcd"
Oct 31 12:39:37.651712 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:37.651688    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66856f9b-041f-4251-997c-be59be05384c-maas-api-openshift-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:39:37.652959 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:37.652938    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66856f9b-041f-4251-997c-be59be05384c-kube-api-access-5f2l2" pod "66856f9b-041f-4251-997c-be59be05384c" (UID: "66856f9b-041f-4251-997c-be59be05384c"). InnerVolumeSpecName "kube-api-access-5f2l2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:39:37.653053 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:37.652976    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66856f9b-041f-4251-997c-be59be05384c-maas-api-xks-ca" pod "66856f9b-041f-4251-997c-be59be05384c" (UID: "66856f9b-041f-4251-997c-be59be05384c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:39:37.752898 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:37.752861    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a72336d5-83db-484a-a300-de68dc5a37c4-workload-certs\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-ffvcd\" (UID: \"a72336d5-83db-484a-a300-de68dc5a37c4\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-ffvcd"
Oct 31 12:39:37.753064 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:37.752914    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25tcp\" (UniqueName: \"kubernetes.io/projected/a72336d5-83db-484a-a300-de68dc5a37c4-kube-api-access-25tcp\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-ffvcd\" (UID: \"a72336d5-83db-484a-a300-de68dc5a37c4\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-ffvcd"
Oct 31 12:39:37.753064 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:37.752938    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a72336d5-83db-484a-a300-de68dc5a37c4-workload-socket\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-ffvcd\" (UID: \"a72336d5-83db-484a-a300-de68dc5a37c4\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-ffvcd"
Oct 31 12:39:37.753064 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:37.752960    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a72336d5-83db-484a-a300-de68dc5a37c4-istio-envoy\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-ffvcd\" (UID: \"a72336d5-83db-484a-a300-de68dc5a37c4\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-ffvcd"
Oct 31 12:39:37.753064 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:37.753001    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a72336d5-83db-484a-a300-de68dc5a37c4-istio-data\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-ffvcd\" (UID: \"a72336d5-83db-484a-a300-de68dc5a37c4\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-ffvcd"
Oct 31 12:39:37.753064 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:37.753044    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a72336d5-83db-484a-a300-de68dc5a37c4-credential-socket\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-ffvcd\" (UID: \"a72336d5-83db-484a-a300-de68dc5a37c4\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-ffvcd"
Oct 31 12:39:37.753064 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:37.753063    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a72336d5-83db-484a-a300-de68dc5a37c4-istio-token\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-ffvcd\" (UID: \"a72336d5-83db-484a-a300-de68dc5a37c4\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-ffvcd"
Oct 31 12:39:37.753363 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:37.753091    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a72336d5-83db-484a-a300-de68dc5a37c4-istiod-ca-cert\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-ffvcd\" (UID: \"a72336d5-83db-484a-a300-de68dc5a37c4\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-ffvcd"
Oct 31 12:39:37.753363 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:37.753135    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a72336d5-83db-484a-a300-de68dc5a37c4-istio-podinfo\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-ffvcd\" (UID: \"a72336d5-83db-484a-a300-de68dc5a37c4\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-ffvcd"
Oct 31 12:39:37.753363 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:37.753224    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5f2l2\" (UniqueName: \"kubernetes.io/projected/66856f9b-041f-4251-997c-be59be05384c-kube-api-access-5f2l2\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:39:37.753363 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:37.753240    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66856f9b-041f-4251-997c-be59be05384c-maas-api-xks-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:39:37.753587 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:37.753403    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a72336d5-83db-484a-a300-de68dc5a37c4-workload-socket\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-ffvcd\" (UID: \"a72336d5-83db-484a-a300-de68dc5a37c4\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-ffvcd"
Oct 31 12:39:37.753587 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:37.753550    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a72336d5-83db-484a-a300-de68dc5a37c4-istio-data\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-ffvcd\" (UID: \"a72336d5-83db-484a-a300-de68dc5a37c4\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-ffvcd"
Oct 31 12:39:37.753683 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:37.753582    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a72336d5-83db-484a-a300-de68dc5a37c4-workload-certs\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-ffvcd\" (UID: \"a72336d5-83db-484a-a300-de68dc5a37c4\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-ffvcd"
Oct 31 12:39:37.753683 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:37.753618    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a72336d5-83db-484a-a300-de68dc5a37c4-credential-socket\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-ffvcd\" (UID: \"a72336d5-83db-484a-a300-de68dc5a37c4\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-ffvcd"
Oct 31 12:39:37.753993 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:37.753968    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a72336d5-83db-484a-a300-de68dc5a37c4-istiod-ca-cert\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-ffvcd\" (UID: \"a72336d5-83db-484a-a300-de68dc5a37c4\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-ffvcd"
Oct 31 12:39:37.755425 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:37.755401    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a72336d5-83db-484a-a300-de68dc5a37c4-istio-envoy\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-ffvcd\" (UID: \"a72336d5-83db-484a-a300-de68dc5a37c4\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-ffvcd"
Oct 31 12:39:37.755845 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:37.755824    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a72336d5-83db-484a-a300-de68dc5a37c4-istio-podinfo\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-ffvcd\" (UID: \"a72336d5-83db-484a-a300-de68dc5a37c4\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-ffvcd"
Oct 31 12:39:37.765594 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:37.765567    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a72336d5-83db-484a-a300-de68dc5a37c4-istio-token\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-ffvcd\" (UID: \"a72336d5-83db-484a-a300-de68dc5a37c4\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-ffvcd"
Oct 31 12:39:37.765846 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:37.765823    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-25tcp\" (UniqueName: \"kubernetes.io/projected/a72336d5-83db-484a-a300-de68dc5a37c4-kube-api-access-25tcp\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-ffvcd\" (UID: \"a72336d5-83db-484a-a300-de68dc5a37c4\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-ffvcd"
Oct 31 12:39:37.842691 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:37.842656    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-ffvcd"
Oct 31 12:39:38.158573 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:38.158467    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-ffvcd"]
Oct 31 12:39:38.161356 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:39:38.161330    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda72336d5_83db_484a_a300_de68dc5a37c4.slice/crio-d7e26be7fecc6dc60a542fe2e45be74ddd09159a9adaa7d4b8d3f39657a7a2fc WatchSource:0}: Error finding container d7e26be7fecc6dc60a542fe2e45be74ddd09159a9adaa7d4b8d3f39657a7a2fc: Status 404 returned error can't find the container with id d7e26be7fecc6dc60a542fe2e45be74ddd09159a9adaa7d4b8d3f39657a7a2fc
Oct 31 12:39:38.163835 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:38.163802    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:39:38.163929 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:38.163869    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:39:38.163929 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:38.163898    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:39:38.333651 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:38.333617    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9384727b-z9rvm" event={"ID":"66856f9b-041f-4251-997c-be59be05384c","Type":"ContainerDied","Data":"de9348441fc15184c9e6551fdac5e137301f48011f3c63549ad91ef7be36ce7e"}
Oct 31 12:39:38.333841 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:38.333668    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="de9348441fc15184c9e6551fdac5e137301f48011f3c63549ad91ef7be36ce7e"
Oct 31 12:39:38.333841 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:38.333698    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9384727b-z9rvm"
Oct 31 12:39:38.344582 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:38.344534    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-ffvcd" event={"ID":"a72336d5-83db-484a-a300-de68dc5a37c4","Type":"ContainerStarted","Data":"26c609efe5737af2f6c4b4b2a014ebee99b278f61bb417df55fd22f54f40072a"}
Oct 31 12:39:38.344760 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:38.344637    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-ffvcd" event={"ID":"a72336d5-83db-484a-a300-de68dc5a37c4","Type":"ContainerStarted","Data":"d7e26be7fecc6dc60a542fe2e45be74ddd09159a9adaa7d4b8d3f39657a7a2fc"}
Oct 31 12:39:38.377124 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:38.377071    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-ffvcd" podStartSLOduration=1.377056523 podStartE2EDuration="1.377056523s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:39:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:39:38.373159012 +0000 UTC m=+741.476226725" watchObservedRunningTime="2026-10-31 12:39:38.377056523 +0000 UTC m=+741.480124234"
Oct 31 12:39:38.843054 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:38.843020    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-ffvcd"
Oct 31 12:39:39.845088 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:39.845054    5060 patch_prober.go:29] interesting pod/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-ffvcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Oct 31 12:39:39.845558 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:39.845126    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-ffvcd" podUID="a72336d5-83db-484a-a300-de68dc5a37c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Oct 31 12:39:40.178565 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:40.177531    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-174b5a58-b42hp"]
Oct 31 12:39:40.198135 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:40.197814    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-174b5a58-b42hp"
Oct 31 12:39:40.222734 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:40.222702    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-174b5a58-b42hp"]
Oct 31 12:39:40.224852 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:40.224808    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 12:39:40.224974 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:40.224901    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 12:39:40.224974 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:40.224952    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 12:39:40.225172 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:40.225148    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7wsmb\""
Oct 31 12:39:40.311734 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:40.311173    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63d60111-09da-4dc5-9c83-4b715c93d5aa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-174b5a58-b42hp\" (UID: \"63d60111-09da-4dc5-9c83-4b715c93d5aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-174b5a58-b42hp"
Oct 31 12:39:40.311734 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:40.311331    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63d60111-09da-4dc5-9c83-4b715c93d5aa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-174b5a58-b42hp\" (UID: \"63d60111-09da-4dc5-9c83-4b715c93d5aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-174b5a58-b42hp"
Oct 31 12:39:40.311734 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:40.311377    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bc4gz\" (UniqueName: \"kubernetes.io/projected/63d60111-09da-4dc5-9c83-4b715c93d5aa-kube-api-access-bc4gz\") pod \"maas-api-revoke-sub-174b5a58-b42hp\" (UID: \"63d60111-09da-4dc5-9c83-4b715c93d5aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-174b5a58-b42hp"
Oct 31 12:39:40.414586 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:40.414547    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63d60111-09da-4dc5-9c83-4b715c93d5aa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-174b5a58-b42hp\" (UID: \"63d60111-09da-4dc5-9c83-4b715c93d5aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-174b5a58-b42hp"
Oct 31 12:39:40.414780 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:40.414667    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63d60111-09da-4dc5-9c83-4b715c93d5aa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-174b5a58-b42hp\" (UID: \"63d60111-09da-4dc5-9c83-4b715c93d5aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-174b5a58-b42hp"
Oct 31 12:39:40.414780 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:40.414711    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bc4gz\" (UniqueName: \"kubernetes.io/projected/63d60111-09da-4dc5-9c83-4b715c93d5aa-kube-api-access-bc4gz\") pod \"maas-api-revoke-sub-174b5a58-b42hp\" (UID: \"63d60111-09da-4dc5-9c83-4b715c93d5aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-174b5a58-b42hp"
Oct 31 12:39:40.415528 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:40.415476    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63d60111-09da-4dc5-9c83-4b715c93d5aa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-174b5a58-b42hp\" (UID: \"63d60111-09da-4dc5-9c83-4b715c93d5aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-174b5a58-b42hp"
Oct 31 12:39:40.419711 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:40.419689    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63d60111-09da-4dc5-9c83-4b715c93d5aa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-174b5a58-b42hp\" (UID: \"63d60111-09da-4dc5-9c83-4b715c93d5aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-174b5a58-b42hp"
Oct 31 12:39:40.430472 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:40.430416    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bc4gz\" (UniqueName: \"kubernetes.io/projected/63d60111-09da-4dc5-9c83-4b715c93d5aa-kube-api-access-bc4gz\") pod \"maas-api-revoke-sub-174b5a58-b42hp\" (UID: \"63d60111-09da-4dc5-9c83-4b715c93d5aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-174b5a58-b42hp"
Oct 31 12:39:40.517445 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:40.517410    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-174b5a58-b42hp"
Oct 31 12:39:40.598074 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:40.598017    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86358a2b-26tlt"]
Oct 31 12:39:40.605739 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:40.605714    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86358a2b-26tlt"
Oct 31 12:39:40.622387 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:40.615409    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86358a2b-26tlt"]
Oct 31 12:39:40.722389 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:40.722301    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/060d9865-c13f-4247-b639-e6b1260314a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-86358a2b-26tlt\" (UID: \"060d9865-c13f-4247-b639-e6b1260314a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86358a2b-26tlt"
Oct 31 12:39:40.722389 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:40.722358    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwc75\" (UniqueName: \"kubernetes.io/projected/060d9865-c13f-4247-b639-e6b1260314a2-kube-api-access-mwc75\") pod \"maas-api-revoke-sub-86358a2b-26tlt\" (UID: \"060d9865-c13f-4247-b639-e6b1260314a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86358a2b-26tlt"
Oct 31 12:39:40.722639 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:40.722416    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/060d9865-c13f-4247-b639-e6b1260314a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-86358a2b-26tlt\" (UID: \"060d9865-c13f-4247-b639-e6b1260314a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86358a2b-26tlt"
Oct 31 12:39:40.823149 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:40.823112    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/060d9865-c13f-4247-b639-e6b1260314a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-86358a2b-26tlt\" (UID: \"060d9865-c13f-4247-b639-e6b1260314a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86358a2b-26tlt"
Oct 31 12:39:40.823334 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:40.823298    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/060d9865-c13f-4247-b639-e6b1260314a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-86358a2b-26tlt\" (UID: \"060d9865-c13f-4247-b639-e6b1260314a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86358a2b-26tlt"
Oct 31 12:39:40.823400 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:40.823346    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwc75\" (UniqueName: \"kubernetes.io/projected/060d9865-c13f-4247-b639-e6b1260314a2-kube-api-access-mwc75\") pod \"maas-api-revoke-sub-86358a2b-26tlt\" (UID: \"060d9865-c13f-4247-b639-e6b1260314a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86358a2b-26tlt"
Oct 31 12:39:40.823927 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:40.823897    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/060d9865-c13f-4247-b639-e6b1260314a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-86358a2b-26tlt\" (UID: \"060d9865-c13f-4247-b639-e6b1260314a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86358a2b-26tlt"
Oct 31 12:39:40.825484 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:40.825457    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/060d9865-c13f-4247-b639-e6b1260314a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-86358a2b-26tlt\" (UID: \"060d9865-c13f-4247-b639-e6b1260314a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86358a2b-26tlt"
Oct 31 12:39:40.837348 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:40.837324    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwc75\" (UniqueName: \"kubernetes.io/projected/060d9865-c13f-4247-b639-e6b1260314a2-kube-api-access-mwc75\") pod \"maas-api-revoke-sub-86358a2b-26tlt\" (UID: \"060d9865-c13f-4247-b639-e6b1260314a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86358a2b-26tlt"
Oct 31 12:39:40.839167 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:40.839143    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-174b5a58-b42hp"]
Oct 31 12:39:40.839754 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:39:40.839730    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63d60111_09da_4dc5_9c83_4b715c93d5aa.slice/crio-fd85ab0f3a342cc6db872eddd4a30b43a719a9a98abcd461a3c8d02f21c6b3c3 WatchSource:0}: Error finding container fd85ab0f3a342cc6db872eddd4a30b43a719a9a98abcd461a3c8d02f21c6b3c3: Status 404 returned error can't find the container with id fd85ab0f3a342cc6db872eddd4a30b43a719a9a98abcd461a3c8d02f21c6b3c3
Oct 31 12:39:40.843949 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:40.843772    5060 patch_prober.go:29] interesting pod/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-ffvcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Oct 31 12:39:40.843949 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:40.843817    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-ffvcd" podUID="a72336d5-83db-484a-a300-de68dc5a37c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Oct 31 12:39:40.949991 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:40.949951    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86358a2b-26tlt"
Oct 31 12:39:41.389171 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:39:41.389131    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod060d9865_c13f_4247_b639_e6b1260314a2.slice/crio-a8aedf9e762d141dba95711a0969262e36df3f29ffb0d5c9155b8655ea4d429e WatchSource:0}: Error finding container a8aedf9e762d141dba95711a0969262e36df3f29ffb0d5c9155b8655ea4d429e: Status 404 returned error can't find the container with id a8aedf9e762d141dba95711a0969262e36df3f29ffb0d5c9155b8655ea4d429e
Oct 31 12:39:41.390302 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:41.390260    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86358a2b-26tlt"]
Oct 31 12:39:41.407880 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:41.407849    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86358a2b-26tlt" event={"ID":"060d9865-c13f-4247-b639-e6b1260314a2","Type":"ContainerStarted","Data":"a8aedf9e762d141dba95711a0969262e36df3f29ffb0d5c9155b8655ea4d429e"}
Oct 31 12:39:41.410584 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:41.410549    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-174b5a58-b42hp" event={"ID":"63d60111-09da-4dc5-9c83-4b715c93d5aa","Type":"ContainerStarted","Data":"93ef9ce56f011215080b2b1c3b037c0dfc2d437e16e9f0b44496e0b65f614e5d"}
Oct 31 12:39:41.410684 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:41.410606    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-174b5a58-b42hp" event={"ID":"63d60111-09da-4dc5-9c83-4b715c93d5aa","Type":"ContainerStarted","Data":"fd85ab0f3a342cc6db872eddd4a30b43a719a9a98abcd461a3c8d02f21c6b3c3"}
Oct 31 12:39:41.430397 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:41.430371    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38082411-ccdx6"]
Oct 31 12:39:41.436311 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:41.436292    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38082411-ccdx6"
Oct 31 12:39:41.442397 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:41.442350    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-174b5a58-b42hp" podStartSLOduration=1.442335868 podStartE2EDuration="1.442335868s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:39:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:39:41.44153248 +0000 UTC m=+744.544600193" watchObservedRunningTime="2026-10-31 12:39:41.442335868 +0000 UTC m=+744.545403581"
Oct 31 12:39:41.449293 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:41.449267    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38082411-ccdx6"]
Oct 31 12:39:41.533915 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:41.533877    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54x2r\" (UniqueName: \"kubernetes.io/projected/de3e8b02-e9f6-4a9d-90c1-ea2a3d901fbc-kube-api-access-54x2r\") pod \"maas-api-revoke-sub-38082411-ccdx6\" (UID: \"de3e8b02-e9f6-4a9d-90c1-ea2a3d901fbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38082411-ccdx6"
Oct 31 12:39:41.534107 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:41.533918    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de3e8b02-e9f6-4a9d-90c1-ea2a3d901fbc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-38082411-ccdx6\" (UID: \"de3e8b02-e9f6-4a9d-90c1-ea2a3d901fbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38082411-ccdx6"
Oct 31 12:39:41.534107 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:41.534068    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de3e8b02-e9f6-4a9d-90c1-ea2a3d901fbc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-38082411-ccdx6\" (UID: \"de3e8b02-e9f6-4a9d-90c1-ea2a3d901fbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38082411-ccdx6"
Oct 31 12:39:41.634855 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:41.634784    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54x2r\" (UniqueName: \"kubernetes.io/projected/de3e8b02-e9f6-4a9d-90c1-ea2a3d901fbc-kube-api-access-54x2r\") pod \"maas-api-revoke-sub-38082411-ccdx6\" (UID: \"de3e8b02-e9f6-4a9d-90c1-ea2a3d901fbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38082411-ccdx6"
Oct 31 12:39:41.634855 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:41.634822    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de3e8b02-e9f6-4a9d-90c1-ea2a3d901fbc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-38082411-ccdx6\" (UID: \"de3e8b02-e9f6-4a9d-90c1-ea2a3d901fbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38082411-ccdx6"
Oct 31 12:39:41.635034 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:41.634899    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de3e8b02-e9f6-4a9d-90c1-ea2a3d901fbc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-38082411-ccdx6\" (UID: \"de3e8b02-e9f6-4a9d-90c1-ea2a3d901fbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38082411-ccdx6"
Oct 31 12:39:41.635547 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:41.635484    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de3e8b02-e9f6-4a9d-90c1-ea2a3d901fbc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-38082411-ccdx6\" (UID: \"de3e8b02-e9f6-4a9d-90c1-ea2a3d901fbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38082411-ccdx6"
Oct 31 12:39:41.637387 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:41.637368    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de3e8b02-e9f6-4a9d-90c1-ea2a3d901fbc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-38082411-ccdx6\" (UID: \"de3e8b02-e9f6-4a9d-90c1-ea2a3d901fbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38082411-ccdx6"
Oct 31 12:39:41.647361 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:41.647334    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-54x2r\" (UniqueName: \"kubernetes.io/projected/de3e8b02-e9f6-4a9d-90c1-ea2a3d901fbc-kube-api-access-54x2r\") pod \"maas-api-revoke-sub-38082411-ccdx6\" (UID: \"de3e8b02-e9f6-4a9d-90c1-ea2a3d901fbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38082411-ccdx6"
Oct 31 12:39:41.799047 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:41.799015    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38082411-ccdx6"
Oct 31 12:39:41.843615 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:41.843443    5060 patch_prober.go:29] interesting pod/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-ffvcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Oct 31 12:39:41.843615 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:41.843516    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-ffvcd" podUID="a72336d5-83db-484a-a300-de68dc5a37c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Oct 31 12:39:42.112591 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:42.111201    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38082411-ccdx6"]
Oct 31 12:39:42.114412 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:39:42.114384    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde3e8b02_e9f6_4a9d_90c1_ea2a3d901fbc.slice/crio-8882e3cc215e7a5c95ec825c872124650e278d2cf1d4afea8e04b56de6927aad WatchSource:0}: Error finding container 8882e3cc215e7a5c95ec825c872124650e278d2cf1d4afea8e04b56de6927aad: Status 404 returned error can't find the container with id 8882e3cc215e7a5c95ec825c872124650e278d2cf1d4afea8e04b56de6927aad
Oct 31 12:39:42.401717 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:42.401684    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e50d2485-49m8z"]
Oct 31 12:39:42.409610 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:42.409581    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e50d2485-49m8z"
Oct 31 12:39:42.415898 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:42.415870    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e50d2485-49m8z"]
Oct 31 12:39:42.421901 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:42.421868    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86358a2b-26tlt" event={"ID":"060d9865-c13f-4247-b639-e6b1260314a2","Type":"ContainerStarted","Data":"f9dd97d39421e34001bc73f6bedf1b26da6c1ada560389d20c81b52a07da4a6e"}
Oct 31 12:39:42.423740 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:42.423714    5060 generic.go:356] "Generic (PLEG): container finished" podID="63d60111-09da-4dc5-9c83-4b715c93d5aa" containerID="93ef9ce56f011215080b2b1c3b037c0dfc2d437e16e9f0b44496e0b65f614e5d" exitCode=0
Oct 31 12:39:42.423834 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:42.423805    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-174b5a58-b42hp" event={"ID":"63d60111-09da-4dc5-9c83-4b715c93d5aa","Type":"ContainerDied","Data":"93ef9ce56f011215080b2b1c3b037c0dfc2d437e16e9f0b44496e0b65f614e5d"}
Oct 31 12:39:42.425570 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:42.425546    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38082411-ccdx6" event={"ID":"de3e8b02-e9f6-4a9d-90c1-ea2a3d901fbc","Type":"ContainerStarted","Data":"8882e3cc215e7a5c95ec825c872124650e278d2cf1d4afea8e04b56de6927aad"}
Oct 31 12:39:42.448979 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:42.448912    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86358a2b-26tlt" podStartSLOduration=2.448894501 podStartE2EDuration="2.448894501s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:39:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:39:42.446283236 +0000 UTC m=+745.549350955" watchObservedRunningTime="2026-10-31 12:39:42.448894501 +0000 UTC m=+745.551962221"
Oct 31 12:39:42.549303 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:42.547990    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18cf4a1e-a2da-4f86-871c-8da52d904b8c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e50d2485-49m8z\" (UID: \"18cf4a1e-a2da-4f86-871c-8da52d904b8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e50d2485-49m8z"
Oct 31 12:39:42.549303 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:42.548088    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18cf4a1e-a2da-4f86-871c-8da52d904b8c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e50d2485-49m8z\" (UID: \"18cf4a1e-a2da-4f86-871c-8da52d904b8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e50d2485-49m8z"
Oct 31 12:39:42.549303 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:42.548128    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpcw4\" (UniqueName: \"kubernetes.io/projected/18cf4a1e-a2da-4f86-871c-8da52d904b8c-kube-api-access-xpcw4\") pod \"maas-api-revoke-sub-e50d2485-49m8z\" (UID: \"18cf4a1e-a2da-4f86-871c-8da52d904b8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e50d2485-49m8z"
Oct 31 12:39:42.649315 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:42.649274    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18cf4a1e-a2da-4f86-871c-8da52d904b8c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e50d2485-49m8z\" (UID: \"18cf4a1e-a2da-4f86-871c-8da52d904b8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e50d2485-49m8z"
Oct 31 12:39:42.649476 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:42.649327    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpcw4\" (UniqueName: \"kubernetes.io/projected/18cf4a1e-a2da-4f86-871c-8da52d904b8c-kube-api-access-xpcw4\") pod \"maas-api-revoke-sub-e50d2485-49m8z\" (UID: \"18cf4a1e-a2da-4f86-871c-8da52d904b8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e50d2485-49m8z"
Oct 31 12:39:42.649572 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:42.649537    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18cf4a1e-a2da-4f86-871c-8da52d904b8c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e50d2485-49m8z\" (UID: \"18cf4a1e-a2da-4f86-871c-8da52d904b8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e50d2485-49m8z"
Oct 31 12:39:42.649964 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:42.649920    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18cf4a1e-a2da-4f86-871c-8da52d904b8c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e50d2485-49m8z\" (UID: \"18cf4a1e-a2da-4f86-871c-8da52d904b8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e50d2485-49m8z"
Oct 31 12:39:42.651815 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:42.651797    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18cf4a1e-a2da-4f86-871c-8da52d904b8c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e50d2485-49m8z\" (UID: \"18cf4a1e-a2da-4f86-871c-8da52d904b8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e50d2485-49m8z"
Oct 31 12:39:42.686253 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:42.686177    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpcw4\" (UniqueName: \"kubernetes.io/projected/18cf4a1e-a2da-4f86-871c-8da52d904b8c-kube-api-access-xpcw4\") pod \"maas-api-revoke-sub-e50d2485-49m8z\" (UID: \"18cf4a1e-a2da-4f86-871c-8da52d904b8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e50d2485-49m8z"
Oct 31 12:39:42.732241 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:42.732210    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e50d2485-49m8z"
Oct 31 12:39:42.843972 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:42.843939    5060 patch_prober.go:29] interesting pod/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-ffvcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Oct 31 12:39:42.844097 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:42.844011    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-ffvcd" podUID="a72336d5-83db-484a-a300-de68dc5a37c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Oct 31 12:39:43.093827 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:43.093794    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e50d2485-49m8z"]
Oct 31 12:39:43.433644 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:43.433610    5060 generic.go:356] "Generic (PLEG): container finished" podID="060d9865-c13f-4247-b639-e6b1260314a2" containerID="f9dd97d39421e34001bc73f6bedf1b26da6c1ada560389d20c81b52a07da4a6e" exitCode=0
Oct 31 12:39:43.434026 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:43.433706    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86358a2b-26tlt" event={"ID":"060d9865-c13f-4247-b639-e6b1260314a2","Type":"ContainerDied","Data":"f9dd97d39421e34001bc73f6bedf1b26da6c1ada560389d20c81b52a07da4a6e"}
Oct 31 12:39:43.435347 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:43.435323    5060 generic.go:356] "Generic (PLEG): container finished" podID="de3e8b02-e9f6-4a9d-90c1-ea2a3d901fbc" containerID="11f365ff9233dad78c06d763b6d868b1fcc373c8d03beb179875edcaf8e8a23a" exitCode=0
Oct 31 12:39:43.435462 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:43.435397    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38082411-ccdx6" event={"ID":"de3e8b02-e9f6-4a9d-90c1-ea2a3d901fbc","Type":"ContainerDied","Data":"11f365ff9233dad78c06d763b6d868b1fcc373c8d03beb179875edcaf8e8a23a"}
Oct 31 12:39:43.436778 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:43.436753    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e50d2485-49m8z" event={"ID":"18cf4a1e-a2da-4f86-871c-8da52d904b8c","Type":"ContainerStarted","Data":"1ce80ecb9eb1153e4bea82f25692bf16b99c5afed7cccd86a4d049dfcf258b15"}
Oct 31 12:39:43.683141 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:43.683098    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-174b5a58-b42hp"
Oct 31 12:39:43.724234 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:43.723746    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/63d60111-09da-4dc5-9c83-4b715c93d5aa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63d60111-09da-4dc5-9c83-4b715c93d5aa-maas-api-xks-ca\") pod \"63d60111-09da-4dc5-9c83-4b715c93d5aa\" (UID: \"63d60111-09da-4dc5-9c83-4b715c93d5aa\") "
Oct 31 12:39:43.724234 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:43.723781    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63d60111-09da-4dc5-9c83-4b715c93d5aa-kube-api-access-bc4gz\" (UniqueName: \"kubernetes.io/projected/63d60111-09da-4dc5-9c83-4b715c93d5aa-kube-api-access-bc4gz\") pod \"63d60111-09da-4dc5-9c83-4b715c93d5aa\" (UID: \"63d60111-09da-4dc5-9c83-4b715c93d5aa\") "
Oct 31 12:39:43.724234 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:43.723812    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63d60111-09da-4dc5-9c83-4b715c93d5aa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63d60111-09da-4dc5-9c83-4b715c93d5aa-maas-api-openshift-ca\") pod \"63d60111-09da-4dc5-9c83-4b715c93d5aa\" (UID: \"63d60111-09da-4dc5-9c83-4b715c93d5aa\") "
Oct 31 12:39:43.724904 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:43.724611    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63d60111-09da-4dc5-9c83-4b715c93d5aa-maas-api-openshift-ca" pod "63d60111-09da-4dc5-9c83-4b715c93d5aa" (UID: "63d60111-09da-4dc5-9c83-4b715c93d5aa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:39:43.727306 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:43.727271    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63d60111-09da-4dc5-9c83-4b715c93d5aa-kube-api-access-bc4gz" pod "63d60111-09da-4dc5-9c83-4b715c93d5aa" (UID: "63d60111-09da-4dc5-9c83-4b715c93d5aa"). InnerVolumeSpecName "kube-api-access-bc4gz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:39:43.728714 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:43.728682    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63d60111-09da-4dc5-9c83-4b715c93d5aa-maas-api-xks-ca" pod "63d60111-09da-4dc5-9c83-4b715c93d5aa" (UID: "63d60111-09da-4dc5-9c83-4b715c93d5aa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:39:43.825349 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:43.825304    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63d60111-09da-4dc5-9c83-4b715c93d5aa-maas-api-xks-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:39:43.825349 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:43.825343    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bc4gz\" (UniqueName: \"kubernetes.io/projected/63d60111-09da-4dc5-9c83-4b715c93d5aa-kube-api-access-bc4gz\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:39:43.825597 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:43.825358    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63d60111-09da-4dc5-9c83-4b715c93d5aa-maas-api-openshift-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:39:43.843723 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:43.843691    5060 patch_prober.go:29] interesting pod/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-ffvcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Oct 31 12:39:43.843882 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:43.843745    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-ffvcd" podUID="a72336d5-83db-484a-a300-de68dc5a37c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Oct 31 12:39:44.364484 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:44.364449    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8138d6b-7r46x"]
Oct 31 12:39:44.365064 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:44.365050    5060 cpu_manager.go:414] "containerMap: removing container" podUID="63d60111-09da-4dc5-9c83-4b715c93d5aa" containerName="revoke-keys"
Oct 31 12:39:44.365114 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:44.365066    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63d60111-09da-4dc5-9c83-4b715c93d5aa" containerName="revoke-keys"
Oct 31 12:39:44.365150 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:44.365146    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="63d60111-09da-4dc5-9c83-4b715c93d5aa" containerName="revoke-keys"
Oct 31 12:39:44.368674 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:44.368647    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8138d6b-7r46x"
Oct 31 12:39:44.385976 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:44.379615    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8138d6b-7r46x"]
Oct 31 12:39:44.432261 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:44.432216    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e805d884-0f36-4b61-b17d-b829c423ce95-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c8138d6b-7r46x\" (UID: \"e805d884-0f36-4b61-b17d-b829c423ce95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8138d6b-7r46x"
Oct 31 12:39:44.432431 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:44.432369    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e805d884-0f36-4b61-b17d-b829c423ce95-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c8138d6b-7r46x\" (UID: \"e805d884-0f36-4b61-b17d-b829c423ce95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8138d6b-7r46x"
Oct 31 12:39:44.432478 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:44.432427    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d989g\" (UniqueName: \"kubernetes.io/projected/e805d884-0f36-4b61-b17d-b829c423ce95-kube-api-access-d989g\") pod \"maas-api-revoke-sub-c8138d6b-7r46x\" (UID: \"e805d884-0f36-4b61-b17d-b829c423ce95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8138d6b-7r46x"
Oct 31 12:39:44.446058 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:44.446032    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-174b5a58-b42hp"
Oct 31 12:39:44.446429 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:44.446060    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-174b5a58-b42hp" event={"ID":"63d60111-09da-4dc5-9c83-4b715c93d5aa","Type":"ContainerDied","Data":"fd85ab0f3a342cc6db872eddd4a30b43a719a9a98abcd461a3c8d02f21c6b3c3"}
Oct 31 12:39:44.446429 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:44.446117    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fd85ab0f3a342cc6db872eddd4a30b43a719a9a98abcd461a3c8d02f21c6b3c3"
Oct 31 12:39:44.447828 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:44.447807    5060 generic.go:356] "Generic (PLEG): container finished" podID="18cf4a1e-a2da-4f86-871c-8da52d904b8c" containerID="6cd6703dd1c2cad435ce7e7555cabf53dc58af7c4777b656b6a9816458d77a52" exitCode=0
Oct 31 12:39:44.447924 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:44.447860    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e50d2485-49m8z" event={"ID":"18cf4a1e-a2da-4f86-871c-8da52d904b8c","Type":"ContainerDied","Data":"6cd6703dd1c2cad435ce7e7555cabf53dc58af7c4777b656b6a9816458d77a52"}
Oct 31 12:39:44.534546 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:44.534331    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e805d884-0f36-4b61-b17d-b829c423ce95-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c8138d6b-7r46x\" (UID: \"e805d884-0f36-4b61-b17d-b829c423ce95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8138d6b-7r46x"
Oct 31 12:39:44.534546 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:44.534409    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d989g\" (UniqueName: \"kubernetes.io/projected/e805d884-0f36-4b61-b17d-b829c423ce95-kube-api-access-d989g\") pod \"maas-api-revoke-sub-c8138d6b-7r46x\" (UID: \"e805d884-0f36-4b61-b17d-b829c423ce95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8138d6b-7r46x"
Oct 31 12:39:44.534782 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:44.534678    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e805d884-0f36-4b61-b17d-b829c423ce95-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c8138d6b-7r46x\" (UID: \"e805d884-0f36-4b61-b17d-b829c423ce95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8138d6b-7r46x"
Oct 31 12:39:44.535337 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:44.535306    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e805d884-0f36-4b61-b17d-b829c423ce95-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c8138d6b-7r46x\" (UID: \"e805d884-0f36-4b61-b17d-b829c423ce95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8138d6b-7r46x"
Oct 31 12:39:44.537282 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:44.537256    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e805d884-0f36-4b61-b17d-b829c423ce95-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c8138d6b-7r46x\" (UID: \"e805d884-0f36-4b61-b17d-b829c423ce95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8138d6b-7r46x"
Oct 31 12:39:44.549323 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:44.549297    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d989g\" (UniqueName: \"kubernetes.io/projected/e805d884-0f36-4b61-b17d-b829c423ce95-kube-api-access-d989g\") pod \"maas-api-revoke-sub-c8138d6b-7r46x\" (UID: \"e805d884-0f36-4b61-b17d-b829c423ce95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8138d6b-7r46x"
Oct 31 12:39:44.644331 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:44.644286    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38082411-ccdx6"
Oct 31 12:39:44.648202 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:44.648174    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86358a2b-26tlt"
Oct 31 12:39:44.687923 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:44.687881    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8138d6b-7r46x"
Oct 31 12:39:44.738015 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:44.737976    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/060d9865-c13f-4247-b639-e6b1260314a2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/060d9865-c13f-4247-b639-e6b1260314a2-maas-api-xks-ca\") pod \"060d9865-c13f-4247-b639-e6b1260314a2\" (UID: \"060d9865-c13f-4247-b639-e6b1260314a2\") "
Oct 31 12:39:44.738203 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:44.738077    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de3e8b02-e9f6-4a9d-90c1-ea2a3d901fbc-kube-api-access-54x2r\" (UniqueName: \"kubernetes.io/projected/de3e8b02-e9f6-4a9d-90c1-ea2a3d901fbc-kube-api-access-54x2r\") pod \"de3e8b02-e9f6-4a9d-90c1-ea2a3d901fbc\" (UID: \"de3e8b02-e9f6-4a9d-90c1-ea2a3d901fbc\") "
Oct 31 12:39:44.738203 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:44.738131    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/060d9865-c13f-4247-b639-e6b1260314a2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/060d9865-c13f-4247-b639-e6b1260314a2-maas-api-openshift-ca\") pod \"060d9865-c13f-4247-b639-e6b1260314a2\" (UID: \"060d9865-c13f-4247-b639-e6b1260314a2\") "
Oct 31 12:39:44.738203 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:44.738170    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de3e8b02-e9f6-4a9d-90c1-ea2a3d901fbc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de3e8b02-e9f6-4a9d-90c1-ea2a3d901fbc-maas-api-openshift-ca\") pod \"de3e8b02-e9f6-4a9d-90c1-ea2a3d901fbc\" (UID: \"de3e8b02-e9f6-4a9d-90c1-ea2a3d901fbc\") "
Oct 31 12:39:44.738583 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:44.738551    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/060d9865-c13f-4247-b639-e6b1260314a2-maas-api-openshift-ca" pod "060d9865-c13f-4247-b639-e6b1260314a2" (UID: "060d9865-c13f-4247-b639-e6b1260314a2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:39:44.738778 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:44.738755    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de3e8b02-e9f6-4a9d-90c1-ea2a3d901fbc-maas-api-openshift-ca" pod "de3e8b02-e9f6-4a9d-90c1-ea2a3d901fbc" (UID: "de3e8b02-e9f6-4a9d-90c1-ea2a3d901fbc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:39:44.739003 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:44.738982    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/060d9865-c13f-4247-b639-e6b1260314a2-kube-api-access-mwc75\" (UniqueName: \"kubernetes.io/projected/060d9865-c13f-4247-b639-e6b1260314a2-kube-api-access-mwc75\") pod \"060d9865-c13f-4247-b639-e6b1260314a2\" (UID: \"060d9865-c13f-4247-b639-e6b1260314a2\") "
Oct 31 12:39:44.739416 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:44.739396    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/de3e8b02-e9f6-4a9d-90c1-ea2a3d901fbc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de3e8b02-e9f6-4a9d-90c1-ea2a3d901fbc-maas-api-xks-ca\") pod \"de3e8b02-e9f6-4a9d-90c1-ea2a3d901fbc\" (UID: \"de3e8b02-e9f6-4a9d-90c1-ea2a3d901fbc\") "
Oct 31 12:39:44.739989 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:44.739963    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/060d9865-c13f-4247-b639-e6b1260314a2-maas-api-openshift-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:39:44.740100 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:44.740003    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de3e8b02-e9f6-4a9d-90c1-ea2a3d901fbc-maas-api-openshift-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:39:44.740284 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:44.740255    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/060d9865-c13f-4247-b639-e6b1260314a2-maas-api-xks-ca" pod "060d9865-c13f-4247-b639-e6b1260314a2" (UID: "060d9865-c13f-4247-b639-e6b1260314a2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:39:44.742040 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:44.741998    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/060d9865-c13f-4247-b639-e6b1260314a2-kube-api-access-mwc75" pod "060d9865-c13f-4247-b639-e6b1260314a2" (UID: "060d9865-c13f-4247-b639-e6b1260314a2"). InnerVolumeSpecName "kube-api-access-mwc75". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:39:44.742279 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:44.742244    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de3e8b02-e9f6-4a9d-90c1-ea2a3d901fbc-maas-api-xks-ca" pod "de3e8b02-e9f6-4a9d-90c1-ea2a3d901fbc" (UID: "de3e8b02-e9f6-4a9d-90c1-ea2a3d901fbc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:39:44.742996 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:44.742541    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de3e8b02-e9f6-4a9d-90c1-ea2a3d901fbc-kube-api-access-54x2r" pod "de3e8b02-e9f6-4a9d-90c1-ea2a3d901fbc" (UID: "de3e8b02-e9f6-4a9d-90c1-ea2a3d901fbc"). InnerVolumeSpecName "kube-api-access-54x2r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:39:44.841472 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:44.841351    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54x2r\" (UniqueName: \"kubernetes.io/projected/de3e8b02-e9f6-4a9d-90c1-ea2a3d901fbc-kube-api-access-54x2r\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:39:44.841472 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:44.841413    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwc75\" (UniqueName: \"kubernetes.io/projected/060d9865-c13f-4247-b639-e6b1260314a2-kube-api-access-mwc75\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:39:44.841472 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:44.841433    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de3e8b02-e9f6-4a9d-90c1-ea2a3d901fbc-maas-api-xks-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:39:44.841472 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:44.841450    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/060d9865-c13f-4247-b639-e6b1260314a2-maas-api-xks-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:39:44.845600 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:44.843878    5060 patch_prober.go:29] interesting pod/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-ffvcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Oct 31 12:39:44.845600 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:44.843937    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-ffvcd" podUID="a72336d5-83db-484a-a300-de68dc5a37c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Oct 31 12:39:44.985932 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:39:44.985894    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode805d884_0f36_4b61_b17d_b829c423ce95.slice/crio-b926ff4d667647c2d5cf5601664e84b9c9ec84481785779c651830731675357b WatchSource:0}: Error finding container b926ff4d667647c2d5cf5601664e84b9c9ec84481785779c651830731675357b: Status 404 returned error can't find the container with id b926ff4d667647c2d5cf5601664e84b9c9ec84481785779c651830731675357b
Oct 31 12:39:44.986276 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:44.986247    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8138d6b-7r46x"]
Oct 31 12:39:45.469724 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:45.468529    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8138d6b-7r46x" event={"ID":"e805d884-0f36-4b61-b17d-b829c423ce95","Type":"ContainerStarted","Data":"c17f4a0a1e6158224d7c2f0f94b95f72dc790a1edf158227943173a0c456f449"}
Oct 31 12:39:45.469724 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:45.468601    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8138d6b-7r46x" event={"ID":"e805d884-0f36-4b61-b17d-b829c423ce95","Type":"ContainerStarted","Data":"b926ff4d667647c2d5cf5601664e84b9c9ec84481785779c651830731675357b"}
Oct 31 12:39:45.470991 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:45.470967    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86358a2b-26tlt"
Oct 31 12:39:45.471131 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:45.470965    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86358a2b-26tlt" event={"ID":"060d9865-c13f-4247-b639-e6b1260314a2","Type":"ContainerDied","Data":"a8aedf9e762d141dba95711a0969262e36df3f29ffb0d5c9155b8655ea4d429e"}
Oct 31 12:39:45.471131 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:45.471100    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a8aedf9e762d141dba95711a0969262e36df3f29ffb0d5c9155b8655ea4d429e"
Oct 31 12:39:45.472428 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:45.472404    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38082411-ccdx6" event={"ID":"de3e8b02-e9f6-4a9d-90c1-ea2a3d901fbc","Type":"ContainerDied","Data":"8882e3cc215e7a5c95ec825c872124650e278d2cf1d4afea8e04b56de6927aad"}
Oct 31 12:39:45.472589 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:45.472436    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38082411-ccdx6"
Oct 31 12:39:45.472589 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:45.472446    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8882e3cc215e7a5c95ec825c872124650e278d2cf1d4afea8e04b56de6927aad"
Oct 31 12:39:45.490291 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:45.490235    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8138d6b-7r46x" podStartSLOduration=1.490222499 podStartE2EDuration="1.490222499s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:39:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:39:45.487162681 +0000 UTC m=+748.590230395" watchObservedRunningTime="2026-10-31 12:39:45.490222499 +0000 UTC m=+748.593290210"
Oct 31 12:39:45.626507 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:45.626469    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e50d2485-49m8z"
Oct 31 12:39:45.652547 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:45.652018    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18cf4a1e-a2da-4f86-871c-8da52d904b8c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18cf4a1e-a2da-4f86-871c-8da52d904b8c-maas-api-xks-ca\") pod \"18cf4a1e-a2da-4f86-871c-8da52d904b8c\" (UID: \"18cf4a1e-a2da-4f86-871c-8da52d904b8c\") "
Oct 31 12:39:45.652547 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:45.652085    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18cf4a1e-a2da-4f86-871c-8da52d904b8c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18cf4a1e-a2da-4f86-871c-8da52d904b8c-maas-api-openshift-ca\") pod \"18cf4a1e-a2da-4f86-871c-8da52d904b8c\" (UID: \"18cf4a1e-a2da-4f86-871c-8da52d904b8c\") "
Oct 31 12:39:45.652547 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:45.652141    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18cf4a1e-a2da-4f86-871c-8da52d904b8c-kube-api-access-xpcw4\" (UniqueName: \"kubernetes.io/projected/18cf4a1e-a2da-4f86-871c-8da52d904b8c-kube-api-access-xpcw4\") pod \"18cf4a1e-a2da-4f86-871c-8da52d904b8c\" (UID: \"18cf4a1e-a2da-4f86-871c-8da52d904b8c\") "
Oct 31 12:39:45.652547 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:45.652507    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18cf4a1e-a2da-4f86-871c-8da52d904b8c-maas-api-openshift-ca" pod "18cf4a1e-a2da-4f86-871c-8da52d904b8c" (UID: "18cf4a1e-a2da-4f86-871c-8da52d904b8c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:39:45.652832 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:45.652688    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18cf4a1e-a2da-4f86-871c-8da52d904b8c-maas-api-openshift-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:39:45.656812 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:45.656775    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18cf4a1e-a2da-4f86-871c-8da52d904b8c-maas-api-xks-ca" pod "18cf4a1e-a2da-4f86-871c-8da52d904b8c" (UID: "18cf4a1e-a2da-4f86-871c-8da52d904b8c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:39:45.657547 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:45.657476    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18cf4a1e-a2da-4f86-871c-8da52d904b8c-kube-api-access-xpcw4" pod "18cf4a1e-a2da-4f86-871c-8da52d904b8c" (UID: "18cf4a1e-a2da-4f86-871c-8da52d904b8c"). InnerVolumeSpecName "kube-api-access-xpcw4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:39:45.754333 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:45.754300    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18cf4a1e-a2da-4f86-871c-8da52d904b8c-maas-api-xks-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:39:45.754333 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:45.754331    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xpcw4\" (UniqueName: \"kubernetes.io/projected/18cf4a1e-a2da-4f86-871c-8da52d904b8c-kube-api-access-xpcw4\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:39:45.849172 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:45.849094    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-ffvcd"
Oct 31 12:39:45.849541 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:45.849515    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-ffvcd"
Oct 31 12:39:45.850325 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:45.850305    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-ffvcd"
Oct 31 12:39:46.480978 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:46.480944    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e50d2485-49m8z"
Oct 31 12:39:46.481379 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:46.480976    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e50d2485-49m8z" event={"ID":"18cf4a1e-a2da-4f86-871c-8da52d904b8c","Type":"ContainerDied","Data":"1ce80ecb9eb1153e4bea82f25692bf16b99c5afed7cccd86a4d049dfcf258b15"}
Oct 31 12:39:46.481379 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:46.481065    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1ce80ecb9eb1153e4bea82f25692bf16b99c5afed7cccd86a4d049dfcf258b15"
Oct 31 12:39:46.482719 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:46.482698    5060 generic.go:356] "Generic (PLEG): container finished" podID="e805d884-0f36-4b61-b17d-b829c423ce95" containerID="c17f4a0a1e6158224d7c2f0f94b95f72dc790a1edf158227943173a0c456f449" exitCode=0
Oct 31 12:39:46.482825 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:46.482754    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8138d6b-7r46x" event={"ID":"e805d884-0f36-4b61-b17d-b829c423ce95","Type":"ContainerDied","Data":"c17f4a0a1e6158224d7c2f0f94b95f72dc790a1edf158227943173a0c456f449"}
Oct 31 12:39:47.649692 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:47.649667    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8138d6b-7r46x"
Oct 31 12:39:47.676861 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:47.676231    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e805d884-0f36-4b61-b17d-b829c423ce95-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e805d884-0f36-4b61-b17d-b829c423ce95-maas-api-xks-ca\") pod \"e805d884-0f36-4b61-b17d-b829c423ce95\" (UID: \"e805d884-0f36-4b61-b17d-b829c423ce95\") "
Oct 31 12:39:47.676861 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:47.676281    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e805d884-0f36-4b61-b17d-b829c423ce95-kube-api-access-d989g\" (UniqueName: \"kubernetes.io/projected/e805d884-0f36-4b61-b17d-b829c423ce95-kube-api-access-d989g\") pod \"e805d884-0f36-4b61-b17d-b829c423ce95\" (UID: \"e805d884-0f36-4b61-b17d-b829c423ce95\") "
Oct 31 12:39:47.676861 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:47.676345    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e805d884-0f36-4b61-b17d-b829c423ce95-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e805d884-0f36-4b61-b17d-b829c423ce95-maas-api-openshift-ca\") pod \"e805d884-0f36-4b61-b17d-b829c423ce95\" (UID: \"e805d884-0f36-4b61-b17d-b829c423ce95\") "
Oct 31 12:39:47.676861 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:47.676819    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e805d884-0f36-4b61-b17d-b829c423ce95-maas-api-openshift-ca" pod "e805d884-0f36-4b61-b17d-b829c423ce95" (UID: "e805d884-0f36-4b61-b17d-b829c423ce95"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:39:47.677190 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:47.677009    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e805d884-0f36-4b61-b17d-b829c423ce95-maas-api-openshift-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:39:47.678962 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:47.678931    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e805d884-0f36-4b61-b17d-b829c423ce95-maas-api-xks-ca" pod "e805d884-0f36-4b61-b17d-b829c423ce95" (UID: "e805d884-0f36-4b61-b17d-b829c423ce95"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:39:47.679089 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:47.678965    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e805d884-0f36-4b61-b17d-b829c423ce95-kube-api-access-d989g" pod "e805d884-0f36-4b61-b17d-b829c423ce95" (UID: "e805d884-0f36-4b61-b17d-b829c423ce95"). InnerVolumeSpecName "kube-api-access-d989g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:39:47.778069 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:47.778025    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e805d884-0f36-4b61-b17d-b829c423ce95-maas-api-xks-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:39:47.778069 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:47.778068    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d989g\" (UniqueName: \"kubernetes.io/projected/e805d884-0f36-4b61-b17d-b829c423ce95-kube-api-access-d989g\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:39:48.501185 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:48.501153    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8138d6b-7r46x"
Oct 31 12:39:48.501385 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:48.501150    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8138d6b-7r46x" event={"ID":"e805d884-0f36-4b61-b17d-b829c423ce95","Type":"ContainerDied","Data":"b926ff4d667647c2d5cf5601664e84b9c9ec84481785779c651830731675357b"}
Oct 31 12:39:48.501385 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:48.501282    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b926ff4d667647c2d5cf5601664e84b9c9ec84481785779c651830731675357b"
Oct 31 12:39:50.831749 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:50.831714    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f6427de-hg4lt"]
Oct 31 12:39:50.832288 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:50.832273    5060 cpu_manager.go:414] "containerMap: removing container" podUID="060d9865-c13f-4247-b639-e6b1260314a2" containerName="revoke-keys"
Oct 31 12:39:50.832347 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:50.832289    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="060d9865-c13f-4247-b639-e6b1260314a2" containerName="revoke-keys"
Oct 31 12:39:50.832347 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:50.832306    5060 cpu_manager.go:414] "containerMap: removing container" podUID="e805d884-0f36-4b61-b17d-b829c423ce95" containerName="revoke-keys"
Oct 31 12:39:50.832347 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:50.832311    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e805d884-0f36-4b61-b17d-b829c423ce95" containerName="revoke-keys"
Oct 31 12:39:50.832347 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:50.832335    5060 cpu_manager.go:414] "containerMap: removing container" podUID="de3e8b02-e9f6-4a9d-90c1-ea2a3d901fbc" containerName="revoke-keys"
Oct 31 12:39:50.832347 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:50.832341    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de3e8b02-e9f6-4a9d-90c1-ea2a3d901fbc" containerName="revoke-keys"
Oct 31 12:39:50.832347 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:50.832348    5060 cpu_manager.go:414] "containerMap: removing container" podUID="18cf4a1e-a2da-4f86-871c-8da52d904b8c" containerName="revoke-keys"
Oct 31 12:39:50.832564 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:50.832354    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18cf4a1e-a2da-4f86-871c-8da52d904b8c" containerName="revoke-keys"
Oct 31 12:39:50.832564 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:50.832437    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="060d9865-c13f-4247-b639-e6b1260314a2" containerName="revoke-keys"
Oct 31 12:39:50.832564 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:50.832446    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="e805d884-0f36-4b61-b17d-b829c423ce95" containerName="revoke-keys"
Oct 31 12:39:50.832564 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:50.832455    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="de3e8b02-e9f6-4a9d-90c1-ea2a3d901fbc" containerName="revoke-keys"
Oct 31 12:39:50.832564 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:50.832461    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="18cf4a1e-a2da-4f86-871c-8da52d904b8c" containerName="revoke-keys"
Oct 31 12:39:50.836156 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:50.836139    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f6427de-hg4lt"
Oct 31 12:39:50.841617 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:50.841599    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7wsmb\""
Oct 31 12:39:50.846767 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:50.846744    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 12:39:50.846767 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:50.846744    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 12:39:50.846886 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:50.846763    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 12:39:50.853510 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:50.852670    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f6427de-hg4lt"]
Oct 31 12:39:50.911045 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:50.911005    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32d226db-5345-467d-9a0b-12920ae3e0e3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6f6427de-hg4lt\" (UID: \"32d226db-5345-467d-9a0b-12920ae3e0e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f6427de-hg4lt"
Oct 31 12:39:50.911211 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:50.911161    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pnhz\" (UniqueName: \"kubernetes.io/projected/32d226db-5345-467d-9a0b-12920ae3e0e3-kube-api-access-6pnhz\") pod \"maas-api-revoke-sub-6f6427de-hg4lt\" (UID: \"32d226db-5345-467d-9a0b-12920ae3e0e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f6427de-hg4lt"
Oct 31 12:39:50.911265 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:50.911214    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32d226db-5345-467d-9a0b-12920ae3e0e3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6f6427de-hg4lt\" (UID: \"32d226db-5345-467d-9a0b-12920ae3e0e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f6427de-hg4lt"
Oct 31 12:39:51.012932 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:51.012895    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pnhz\" (UniqueName: \"kubernetes.io/projected/32d226db-5345-467d-9a0b-12920ae3e0e3-kube-api-access-6pnhz\") pod \"maas-api-revoke-sub-6f6427de-hg4lt\" (UID: \"32d226db-5345-467d-9a0b-12920ae3e0e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f6427de-hg4lt"
Oct 31 12:39:51.013125 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:51.012957    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32d226db-5345-467d-9a0b-12920ae3e0e3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6f6427de-hg4lt\" (UID: \"32d226db-5345-467d-9a0b-12920ae3e0e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f6427de-hg4lt"
Oct 31 12:39:51.013125 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:51.013013    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32d226db-5345-467d-9a0b-12920ae3e0e3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6f6427de-hg4lt\" (UID: \"32d226db-5345-467d-9a0b-12920ae3e0e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f6427de-hg4lt"
Oct 31 12:39:51.013707 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:51.013679    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32d226db-5345-467d-9a0b-12920ae3e0e3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6f6427de-hg4lt\" (UID: \"32d226db-5345-467d-9a0b-12920ae3e0e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f6427de-hg4lt"
Oct 31 12:39:51.015467 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:51.015435    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32d226db-5345-467d-9a0b-12920ae3e0e3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6f6427de-hg4lt\" (UID: \"32d226db-5345-467d-9a0b-12920ae3e0e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f6427de-hg4lt"
Oct 31 12:39:51.028909 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:51.028877    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pnhz\" (UniqueName: \"kubernetes.io/projected/32d226db-5345-467d-9a0b-12920ae3e0e3-kube-api-access-6pnhz\") pod \"maas-api-revoke-sub-6f6427de-hg4lt\" (UID: \"32d226db-5345-467d-9a0b-12920ae3e0e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f6427de-hg4lt"
Oct 31 12:39:51.152072 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:51.151988    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f6427de-hg4lt"
Oct 31 12:39:51.482657 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:51.470187    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f6427de-hg4lt"]
Oct 31 12:39:51.528592 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:51.528552    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f6427de-hg4lt" event={"ID":"32d226db-5345-467d-9a0b-12920ae3e0e3","Type":"ContainerStarted","Data":"96ef213dd83edfeebf0b853c7a5ab199b41afc2ead2b33023e71c4533fd21e84"}
Oct 31 12:39:51.864524 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:51.862651    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-rr5tg"]
Oct 31 12:39:52.011242 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:52.011100    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-rr5tg"]
Oct 31 12:39:52.011517 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:52.011211    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-rr5tg"
Oct 31 12:39:52.072522 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:52.044623    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8873fe2c-839e-4d23-83d1-1fe3a451b5cd-istio-data\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-rr5tg\" (UID: \"8873fe2c-839e-4d23-83d1-1fe3a451b5cd\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-rr5tg"
Oct 31 12:39:52.072962 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:52.072936    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8873fe2c-839e-4d23-83d1-1fe3a451b5cd-credential-socket\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-rr5tg\" (UID: \"8873fe2c-839e-4d23-83d1-1fe3a451b5cd\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-rr5tg"
Oct 31 12:39:52.073167 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:52.069356    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-67d410-openshift-default-dockercfg-p2p5s\""
Oct 31 12:39:52.073324 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:52.073305    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8873fe2c-839e-4d23-83d1-1fe3a451b5cd-istio-token\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-rr5tg\" (UID: \"8873fe2c-839e-4d23-83d1-1fe3a451b5cd\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-rr5tg"
Oct 31 12:39:52.073479 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:52.073465    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8873fe2c-839e-4d23-83d1-1fe3a451b5cd-istio-envoy\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-rr5tg\" (UID: \"8873fe2c-839e-4d23-83d1-1fe3a451b5cd\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-rr5tg"
Oct 31 12:39:52.073671 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:52.073654    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8873fe2c-839e-4d23-83d1-1fe3a451b5cd-workload-socket\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-rr5tg\" (UID: \"8873fe2c-839e-4d23-83d1-1fe3a451b5cd\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-rr5tg"
Oct 31 12:39:52.073815 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:52.073799    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8873fe2c-839e-4d23-83d1-1fe3a451b5cd-istiod-ca-cert\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-rr5tg\" (UID: \"8873fe2c-839e-4d23-83d1-1fe3a451b5cd\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-rr5tg"
Oct 31 12:39:52.073991 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:52.073975    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8jcz\" (UniqueName: \"kubernetes.io/projected/8873fe2c-839e-4d23-83d1-1fe3a451b5cd-kube-api-access-v8jcz\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-rr5tg\" (UID: \"8873fe2c-839e-4d23-83d1-1fe3a451b5cd\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-rr5tg"
Oct 31 12:39:52.074207 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:52.074190    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8873fe2c-839e-4d23-83d1-1fe3a451b5cd-istio-podinfo\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-rr5tg\" (UID: \"8873fe2c-839e-4d23-83d1-1fe3a451b5cd\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-rr5tg"
Oct 31 12:39:52.074365 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:52.074351    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8873fe2c-839e-4d23-83d1-1fe3a451b5cd-workload-certs\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-rr5tg\" (UID: \"8873fe2c-839e-4d23-83d1-1fe3a451b5cd\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-rr5tg"
Oct 31 12:39:52.223226 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:52.223121    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8873fe2c-839e-4d23-83d1-1fe3a451b5cd-istiod-ca-cert\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-rr5tg\" (UID: \"8873fe2c-839e-4d23-83d1-1fe3a451b5cd\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-rr5tg"
Oct 31 12:39:52.223483 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:52.223461    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8jcz\" (UniqueName: \"kubernetes.io/projected/8873fe2c-839e-4d23-83d1-1fe3a451b5cd-kube-api-access-v8jcz\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-rr5tg\" (UID: \"8873fe2c-839e-4d23-83d1-1fe3a451b5cd\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-rr5tg"
Oct 31 12:39:52.223699 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:52.223679    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8873fe2c-839e-4d23-83d1-1fe3a451b5cd-istio-podinfo\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-rr5tg\" (UID: \"8873fe2c-839e-4d23-83d1-1fe3a451b5cd\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-rr5tg"
Oct 31 12:39:52.223807 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:52.223793    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8873fe2c-839e-4d23-83d1-1fe3a451b5cd-workload-certs\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-rr5tg\" (UID: \"8873fe2c-839e-4d23-83d1-1fe3a451b5cd\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-rr5tg"
Oct 31 12:39:52.223969 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:52.223955    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8873fe2c-839e-4d23-83d1-1fe3a451b5cd-istio-data\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-rr5tg\" (UID: \"8873fe2c-839e-4d23-83d1-1fe3a451b5cd\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-rr5tg"
Oct 31 12:39:52.224082 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:52.224067    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8873fe2c-839e-4d23-83d1-1fe3a451b5cd-credential-socket\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-rr5tg\" (UID: \"8873fe2c-839e-4d23-83d1-1fe3a451b5cd\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-rr5tg"
Oct 31 12:39:52.224216 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:52.224204    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8873fe2c-839e-4d23-83d1-1fe3a451b5cd-istio-token\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-rr5tg\" (UID: \"8873fe2c-839e-4d23-83d1-1fe3a451b5cd\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-rr5tg"
Oct 31 12:39:52.224321 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:52.224308    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8873fe2c-839e-4d23-83d1-1fe3a451b5cd-istio-envoy\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-rr5tg\" (UID: \"8873fe2c-839e-4d23-83d1-1fe3a451b5cd\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-rr5tg"
Oct 31 12:39:52.248850 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:52.248814    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8873fe2c-839e-4d23-83d1-1fe3a451b5cd-workload-socket\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-rr5tg\" (UID: \"8873fe2c-839e-4d23-83d1-1fe3a451b5cd\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-rr5tg"
Oct 31 12:39:52.249160 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:52.233523    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8873fe2c-839e-4d23-83d1-1fe3a451b5cd-istiod-ca-cert\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-rr5tg\" (UID: \"8873fe2c-839e-4d23-83d1-1fe3a451b5cd\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-rr5tg"
Oct 31 12:39:52.253514 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:52.249530    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8873fe2c-839e-4d23-83d1-1fe3a451b5cd-workload-socket\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-rr5tg\" (UID: \"8873fe2c-839e-4d23-83d1-1fe3a451b5cd\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-rr5tg"
Oct 31 12:39:52.307149 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:52.282999    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8873fe2c-839e-4d23-83d1-1fe3a451b5cd-istio-data\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-rr5tg\" (UID: \"8873fe2c-839e-4d23-83d1-1fe3a451b5cd\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-rr5tg"
Oct 31 12:39:52.307149 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:52.283270    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8873fe2c-839e-4d23-83d1-1fe3a451b5cd-workload-certs\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-rr5tg\" (UID: \"8873fe2c-839e-4d23-83d1-1fe3a451b5cd\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-rr5tg"
Oct 31 12:39:52.307149 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:52.283529    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8873fe2c-839e-4d23-83d1-1fe3a451b5cd-credential-socket\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-rr5tg\" (UID: \"8873fe2c-839e-4d23-83d1-1fe3a451b5cd\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-rr5tg"
Oct 31 12:39:52.374798 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:52.373559    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8jcz\" (UniqueName: \"kubernetes.io/projected/8873fe2c-839e-4d23-83d1-1fe3a451b5cd-kube-api-access-v8jcz\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-rr5tg\" (UID: \"8873fe2c-839e-4d23-83d1-1fe3a451b5cd\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-rr5tg"
Oct 31 12:39:52.430526 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:52.423057    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8873fe2c-839e-4d23-83d1-1fe3a451b5cd-istio-podinfo\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-rr5tg\" (UID: \"8873fe2c-839e-4d23-83d1-1fe3a451b5cd\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-rr5tg"
Oct 31 12:39:52.465834 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:52.465797    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8873fe2c-839e-4d23-83d1-1fe3a451b5cd-istio-envoy\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-rr5tg\" (UID: \"8873fe2c-839e-4d23-83d1-1fe3a451b5cd\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-rr5tg"
Oct 31 12:39:52.509264 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:52.509158    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8873fe2c-839e-4d23-83d1-1fe3a451b5cd-istio-token\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-rr5tg\" (UID: \"8873fe2c-839e-4d23-83d1-1fe3a451b5cd\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-rr5tg"
Oct 31 12:39:52.583654 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:52.583599    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f6427de-hg4lt" event={"ID":"32d226db-5345-467d-9a0b-12920ae3e0e3","Type":"ContainerStarted","Data":"ba1fe25266ab8f7ab9ac332357a5de8f0049823d424b9d00dd2d34f896a52ddf"}
Oct 31 12:39:52.663515 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:52.659726    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f6427de-hg4lt" podStartSLOduration=2.65970334 podStartE2EDuration="2.65970334s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:39:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:39:52.642032054 +0000 UTC m=+755.745099767" watchObservedRunningTime="2026-10-31 12:39:52.65970334 +0000 UTC m=+755.762771054"
Oct 31 12:39:52.765941 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:52.765856    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-rr5tg"
Oct 31 12:39:53.524403 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:53.524363    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-rr5tg"]
Oct 31 12:39:53.539684 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:53.539638    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:39:53.539958 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:53.539931    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:39:53.540104 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:53.540083    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:39:53.619217 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:53.619180    5060 generic.go:356] "Generic (PLEG): container finished" podID="32d226db-5345-467d-9a0b-12920ae3e0e3" containerID="ba1fe25266ab8f7ab9ac332357a5de8f0049823d424b9d00dd2d34f896a52ddf" exitCode=0
Oct 31 12:39:53.619503 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:53.619464    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f6427de-hg4lt" event={"ID":"32d226db-5345-467d-9a0b-12920ae3e0e3","Type":"ContainerDied","Data":"ba1fe25266ab8f7ab9ac332357a5de8f0049823d424b9d00dd2d34f896a52ddf"}
Oct 31 12:39:53.622031 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:53.622006    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-rr5tg" event={"ID":"8873fe2c-839e-4d23-83d1-1fe3a451b5cd","Type":"ContainerStarted","Data":"3475b16fe741a8d8cadfea0dbcb62e45d397853074e63dd362b8710f5bfa6b00"}
Oct 31 12:39:54.638084 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:54.638042    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-rr5tg" event={"ID":"8873fe2c-839e-4d23-83d1-1fe3a451b5cd","Type":"ContainerStarted","Data":"e0c1db5b4653ba85849774353da9a1a5d2c1fd751be6dca22e10e7926e304778"}
Oct 31 12:39:54.732702 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:54.717949    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-rr5tg" podStartSLOduration=3.717927203 podStartE2EDuration="3.717927203s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:39:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:39:54.674829193 +0000 UTC m=+757.777896909" watchObservedRunningTime="2026-10-31 12:39:54.717927203 +0000 UTC m=+757.820994917"
Oct 31 12:39:54.838310 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:54.775934    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-rr5tg"
Oct 31 12:39:54.838310 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:54.794313    5060 patch_prober.go:29] interesting pod/e2e-worker-w6-67d410-openshift-default-676b978549-rr5tg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Oct 31 12:39:54.838310 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:54.794374    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-rr5tg" podUID="8873fe2c-839e-4d23-83d1-1fe3a451b5cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Oct 31 12:39:55.120572 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:55.120541    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e622406c-mq2c9"]
Oct 31 12:39:55.126722 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:55.126694    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f6427de-hg4lt"
Oct 31 12:39:55.131685 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:55.131656    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e622406c-mq2c9"
Oct 31 12:39:55.133249 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:55.133222    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e622406c-mq2c9"]
Oct 31 12:39:55.322134 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:55.322104    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/32d226db-5345-467d-9a0b-12920ae3e0e3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32d226db-5345-467d-9a0b-12920ae3e0e3-maas-api-xks-ca\") pod \"32d226db-5345-467d-9a0b-12920ae3e0e3\" (UID: \"32d226db-5345-467d-9a0b-12920ae3e0e3\") "
Oct 31 12:39:55.322423 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:55.322406    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32d226db-5345-467d-9a0b-12920ae3e0e3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32d226db-5345-467d-9a0b-12920ae3e0e3-maas-api-openshift-ca\") pod \"32d226db-5345-467d-9a0b-12920ae3e0e3\" (UID: \"32d226db-5345-467d-9a0b-12920ae3e0e3\") "
Oct 31 12:39:55.322628 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:55.322611    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32d226db-5345-467d-9a0b-12920ae3e0e3-kube-api-access-6pnhz\" (UniqueName: \"kubernetes.io/projected/32d226db-5345-467d-9a0b-12920ae3e0e3-kube-api-access-6pnhz\") pod \"32d226db-5345-467d-9a0b-12920ae3e0e3\" (UID: \"32d226db-5345-467d-9a0b-12920ae3e0e3\") "
Oct 31 12:39:55.322814 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:55.322790    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32d226db-5345-467d-9a0b-12920ae3e0e3-maas-api-openshift-ca" pod "32d226db-5345-467d-9a0b-12920ae3e0e3" (UID: "32d226db-5345-467d-9a0b-12920ae3e0e3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:39:55.323146 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:55.323128    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b44cd3d-8334-452f-937f-a3dd8904d3f2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e622406c-mq2c9\" (UID: \"7b44cd3d-8334-452f-937f-a3dd8904d3f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e622406c-mq2c9"
Oct 31 12:39:55.323263 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:55.323250    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b44cd3d-8334-452f-937f-a3dd8904d3f2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e622406c-mq2c9\" (UID: \"7b44cd3d-8334-452f-937f-a3dd8904d3f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e622406c-mq2c9"
Oct 31 12:39:55.323421 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:55.323398    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbtvj\" (UniqueName: \"kubernetes.io/projected/7b44cd3d-8334-452f-937f-a3dd8904d3f2-kube-api-access-gbtvj\") pod \"maas-api-revoke-sub-e622406c-mq2c9\" (UID: \"7b44cd3d-8334-452f-937f-a3dd8904d3f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e622406c-mq2c9"
Oct 31 12:39:55.323593 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:55.323575    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32d226db-5345-467d-9a0b-12920ae3e0e3-maas-api-openshift-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:39:55.325210 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:55.325179    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32d226db-5345-467d-9a0b-12920ae3e0e3-maas-api-xks-ca" pod "32d226db-5345-467d-9a0b-12920ae3e0e3" (UID: "32d226db-5345-467d-9a0b-12920ae3e0e3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:39:55.325352 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:55.325334    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32d226db-5345-467d-9a0b-12920ae3e0e3-kube-api-access-6pnhz" pod "32d226db-5345-467d-9a0b-12920ae3e0e3" (UID: "32d226db-5345-467d-9a0b-12920ae3e0e3"). InnerVolumeSpecName "kube-api-access-6pnhz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:39:55.426560 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:55.426520    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b44cd3d-8334-452f-937f-a3dd8904d3f2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e622406c-mq2c9\" (UID: \"7b44cd3d-8334-452f-937f-a3dd8904d3f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e622406c-mq2c9"
Oct 31 12:39:55.426734 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:55.426577    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b44cd3d-8334-452f-937f-a3dd8904d3f2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e622406c-mq2c9\" (UID: \"7b44cd3d-8334-452f-937f-a3dd8904d3f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e622406c-mq2c9"
Oct 31 12:39:55.426734 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:55.426645    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbtvj\" (UniqueName: \"kubernetes.io/projected/7b44cd3d-8334-452f-937f-a3dd8904d3f2-kube-api-access-gbtvj\") pod \"maas-api-revoke-sub-e622406c-mq2c9\" (UID: \"7b44cd3d-8334-452f-937f-a3dd8904d3f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e622406c-mq2c9"
Oct 31 12:39:55.426883 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:55.426794    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32d226db-5345-467d-9a0b-12920ae3e0e3-maas-api-xks-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:39:55.426883 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:55.426812    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6pnhz\" (UniqueName: \"kubernetes.io/projected/32d226db-5345-467d-9a0b-12920ae3e0e3-kube-api-access-6pnhz\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:39:55.427869 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:55.427814    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b44cd3d-8334-452f-937f-a3dd8904d3f2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e622406c-mq2c9\" (UID: \"7b44cd3d-8334-452f-937f-a3dd8904d3f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e622406c-mq2c9"
Oct 31 12:39:55.429333 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:55.429299    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b44cd3d-8334-452f-937f-a3dd8904d3f2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e622406c-mq2c9\" (UID: \"7b44cd3d-8334-452f-937f-a3dd8904d3f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e622406c-mq2c9"
Oct 31 12:39:55.441932 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:55.441909    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbtvj\" (UniqueName: \"kubernetes.io/projected/7b44cd3d-8334-452f-937f-a3dd8904d3f2-kube-api-access-gbtvj\") pod \"maas-api-revoke-sub-e622406c-mq2c9\" (UID: \"7b44cd3d-8334-452f-937f-a3dd8904d3f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e622406c-mq2c9"
Oct 31 12:39:55.533930 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:55.533893    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e622406c-mq2c9"
Oct 31 12:39:55.540794 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:55.540768    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-97464201-lbnl5"]
Oct 31 12:39:55.541367 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:55.541353    5060 cpu_manager.go:414] "containerMap: removing container" podUID="32d226db-5345-467d-9a0b-12920ae3e0e3" containerName="revoke-keys"
Oct 31 12:39:55.541411 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:55.541370    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32d226db-5345-467d-9a0b-12920ae3e0e3" containerName="revoke-keys"
Oct 31 12:39:55.541482 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:55.541472    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="32d226db-5345-467d-9a0b-12920ae3e0e3" containerName="revoke-keys"
Oct 31 12:39:55.545957 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:55.545940    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97464201-lbnl5"
Oct 31 12:39:55.612442 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:55.612310    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-97464201-lbnl5"]
Oct 31 12:39:55.628234 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:55.628012    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c469bb8f-7712-4c3d-8bd4-af409971ef16-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-97464201-lbnl5\" (UID: \"c469bb8f-7712-4c3d-8bd4-af409971ef16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97464201-lbnl5"
Oct 31 12:39:55.628234 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:55.628046    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c469bb8f-7712-4c3d-8bd4-af409971ef16-maas-api-xks-ca\") pod \"maas-api-revoke-sub-97464201-lbnl5\" (UID: \"c469bb8f-7712-4c3d-8bd4-af409971ef16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97464201-lbnl5"
Oct 31 12:39:55.628234 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:55.628140    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pp8r\" (UniqueName: \"kubernetes.io/projected/c469bb8f-7712-4c3d-8bd4-af409971ef16-kube-api-access-7pp8r\") pod \"maas-api-revoke-sub-97464201-lbnl5\" (UID: \"c469bb8f-7712-4c3d-8bd4-af409971ef16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97464201-lbnl5"
Oct 31 12:39:55.637631 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:55.637607    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f07afdef-x7w7t"]
Oct 31 12:39:55.679516 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:55.653188    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f6427de-hg4lt"
Oct 31 12:39:55.679516 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:55.653331    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f07afdef-x7w7t"
Oct 31 12:39:55.679516 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:55.654294    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f6427de-hg4lt" event={"ID":"32d226db-5345-467d-9a0b-12920ae3e0e3","Type":"ContainerDied","Data":"96ef213dd83edfeebf0b853c7a5ab199b41afc2ead2b33023e71c4533fd21e84"}
Oct 31 12:39:55.679516 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:55.654407    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="96ef213dd83edfeebf0b853c7a5ab199b41afc2ead2b33023e71c4533fd21e84"
Oct 31 12:39:55.705338 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:55.705255    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f07afdef-x7w7t"]
Oct 31 12:39:55.728972 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:55.728940    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c469bb8f-7712-4c3d-8bd4-af409971ef16-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-97464201-lbnl5\" (UID: \"c469bb8f-7712-4c3d-8bd4-af409971ef16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97464201-lbnl5"
Oct 31 12:39:55.729116 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:55.728986    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c469bb8f-7712-4c3d-8bd4-af409971ef16-maas-api-xks-ca\") pod \"maas-api-revoke-sub-97464201-lbnl5\" (UID: \"c469bb8f-7712-4c3d-8bd4-af409971ef16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97464201-lbnl5"
Oct 31 12:39:55.729116 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:55.729040    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgwcn\" (UniqueName: \"kubernetes.io/projected/1449a717-8c6d-4eed-b3e0-4475a92e75c3-kube-api-access-pgwcn\") pod \"maas-api-revoke-sub-f07afdef-x7w7t\" (UID: \"1449a717-8c6d-4eed-b3e0-4475a92e75c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f07afdef-x7w7t"
Oct 31 12:39:55.729116 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:55.729105    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1449a717-8c6d-4eed-b3e0-4475a92e75c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f07afdef-x7w7t\" (UID: \"1449a717-8c6d-4eed-b3e0-4475a92e75c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f07afdef-x7w7t"
Oct 31 12:39:55.729310 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:55.729138    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1449a717-8c6d-4eed-b3e0-4475a92e75c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f07afdef-x7w7t\" (UID: \"1449a717-8c6d-4eed-b3e0-4475a92e75c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f07afdef-x7w7t"
Oct 31 12:39:55.729310 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:55.729191    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pp8r\" (UniqueName: \"kubernetes.io/projected/c469bb8f-7712-4c3d-8bd4-af409971ef16-kube-api-access-7pp8r\") pod \"maas-api-revoke-sub-97464201-lbnl5\" (UID: \"c469bb8f-7712-4c3d-8bd4-af409971ef16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97464201-lbnl5"
Oct 31 12:39:55.731277 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:55.730433    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c469bb8f-7712-4c3d-8bd4-af409971ef16-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-97464201-lbnl5\" (UID: \"c469bb8f-7712-4c3d-8bd4-af409971ef16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97464201-lbnl5"
Oct 31 12:39:55.731925 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:55.731902    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c469bb8f-7712-4c3d-8bd4-af409971ef16-maas-api-xks-ca\") pod \"maas-api-revoke-sub-97464201-lbnl5\" (UID: \"c469bb8f-7712-4c3d-8bd4-af409971ef16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97464201-lbnl5"
Oct 31 12:39:55.753180 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:55.753156    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pp8r\" (UniqueName: \"kubernetes.io/projected/c469bb8f-7712-4c3d-8bd4-af409971ef16-kube-api-access-7pp8r\") pod \"maas-api-revoke-sub-97464201-lbnl5\" (UID: \"c469bb8f-7712-4c3d-8bd4-af409971ef16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97464201-lbnl5"
Oct 31 12:39:55.788567 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:55.786266    5060 patch_prober.go:29] interesting pod/e2e-worker-w6-67d410-openshift-default-676b978549-rr5tg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Oct 31 12:39:55.788567 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:55.786334    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-rr5tg" podUID="8873fe2c-839e-4d23-83d1-1fe3a451b5cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Oct 31 12:39:55.829795 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:55.829764    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgwcn\" (UniqueName: \"kubernetes.io/projected/1449a717-8c6d-4eed-b3e0-4475a92e75c3-kube-api-access-pgwcn\") pod \"maas-api-revoke-sub-f07afdef-x7w7t\" (UID: \"1449a717-8c6d-4eed-b3e0-4475a92e75c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f07afdef-x7w7t"
Oct 31 12:39:55.829960 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:55.829839    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1449a717-8c6d-4eed-b3e0-4475a92e75c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f07afdef-x7w7t\" (UID: \"1449a717-8c6d-4eed-b3e0-4475a92e75c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f07afdef-x7w7t"
Oct 31 12:39:55.829960 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:55.829881    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1449a717-8c6d-4eed-b3e0-4475a92e75c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f07afdef-x7w7t\" (UID: \"1449a717-8c6d-4eed-b3e0-4475a92e75c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f07afdef-x7w7t"
Oct 31 12:39:55.830538 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:55.830509    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1449a717-8c6d-4eed-b3e0-4475a92e75c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f07afdef-x7w7t\" (UID: \"1449a717-8c6d-4eed-b3e0-4475a92e75c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f07afdef-x7w7t"
Oct 31 12:39:55.832266 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:55.832247    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1449a717-8c6d-4eed-b3e0-4475a92e75c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f07afdef-x7w7t\" (UID: \"1449a717-8c6d-4eed-b3e0-4475a92e75c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f07afdef-x7w7t"
Oct 31 12:39:55.888322 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:55.888236    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgwcn\" (UniqueName: \"kubernetes.io/projected/1449a717-8c6d-4eed-b3e0-4475a92e75c3-kube-api-access-pgwcn\") pod \"maas-api-revoke-sub-f07afdef-x7w7t\" (UID: \"1449a717-8c6d-4eed-b3e0-4475a92e75c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f07afdef-x7w7t"
Oct 31 12:39:55.911076 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:55.911044    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e622406c-mq2c9"]
Oct 31 12:39:55.911836 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:39:55.911804    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b44cd3d_8334_452f_937f_a3dd8904d3f2.slice/crio-6554abeddf5e45950f763d84d971c4eb2139a21775ffe58ee24389f32e0de91e WatchSource:0}: Error finding container 6554abeddf5e45950f763d84d971c4eb2139a21775ffe58ee24389f32e0de91e: Status 404 returned error can't find the container with id 6554abeddf5e45950f763d84d971c4eb2139a21775ffe58ee24389f32e0de91e
Oct 31 12:39:55.915914 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:55.915892    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97464201-lbnl5"
Oct 31 12:39:56.030886 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:56.030857    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f07afdef-x7w7t"
Oct 31 12:39:56.234690 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:56.234608    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-97464201-lbnl5"]
Oct 31 12:39:56.236159 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:39:56.236135    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc469bb8f_7712_4c3d_8bd4_af409971ef16.slice/crio-e0fbb673b41939bbbcac73f0c7a735b58b7ca41f12352081e2726009506bad33 WatchSource:0}: Error finding container e0fbb673b41939bbbcac73f0c7a735b58b7ca41f12352081e2726009506bad33: Status 404 returned error can't find the container with id e0fbb673b41939bbbcac73f0c7a735b58b7ca41f12352081e2726009506bad33
Oct 31 12:39:56.393249 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:56.393174    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f07afdef-x7w7t"]
Oct 31 12:39:56.490525 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:39:56.489831    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1449a717_8c6d_4eed_b3e0_4475a92e75c3.slice/crio-ac88b6cfd0dffe559110146346e39ecdf73f3bf1b0153685699fa82908f8f7c6 WatchSource:0}: Error finding container ac88b6cfd0dffe559110146346e39ecdf73f3bf1b0153685699fa82908f8f7c6: Status 404 returned error can't find the container with id ac88b6cfd0dffe559110146346e39ecdf73f3bf1b0153685699fa82908f8f7c6
Oct 31 12:39:56.698105 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:56.698071    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f07afdef-x7w7t" event={"ID":"1449a717-8c6d-4eed-b3e0-4475a92e75c3","Type":"ContainerStarted","Data":"ac88b6cfd0dffe559110146346e39ecdf73f3bf1b0153685699fa82908f8f7c6"}
Oct 31 12:39:56.700002 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:56.699971    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97464201-lbnl5" event={"ID":"c469bb8f-7712-4c3d-8bd4-af409971ef16","Type":"ContainerStarted","Data":"e0fbb673b41939bbbcac73f0c7a735b58b7ca41f12352081e2726009506bad33"}
Oct 31 12:39:56.701904 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:56.701876    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e622406c-mq2c9" event={"ID":"7b44cd3d-8334-452f-937f-a3dd8904d3f2","Type":"ContainerStarted","Data":"89ef3c27df05aa7946878decc263a3f4b9de33523c2259d80108c4266465fa79"}
Oct 31 12:39:56.702077 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:56.702053    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e622406c-mq2c9" event={"ID":"7b44cd3d-8334-452f-937f-a3dd8904d3f2","Type":"ContainerStarted","Data":"6554abeddf5e45950f763d84d971c4eb2139a21775ffe58ee24389f32e0de91e"}
Oct 31 12:39:56.723250 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:56.723208    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e622406c-mq2c9" podStartSLOduration=1.723196132 podStartE2EDuration="1.723196132s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:39:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:39:56.720458104 +0000 UTC m=+759.823525816" watchObservedRunningTime="2026-10-31 12:39:56.723196132 +0000 UTC m=+759.826263878"
Oct 31 12:39:56.775721 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:56.775679    5060 patch_prober.go:29] interesting pod/e2e-worker-w6-67d410-openshift-default-676b978549-rr5tg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Oct 31 12:39:56.775979 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:56.775940    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-rr5tg" podUID="8873fe2c-839e-4d23-83d1-1fe3a451b5cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Oct 31 12:39:57.712300 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:57.712262    5060 generic.go:356] "Generic (PLEG): container finished" podID="c469bb8f-7712-4c3d-8bd4-af409971ef16" containerID="cf8416d48a62293528b93feed181db38360846c4012c673073664232cf39285d" exitCode=0
Oct 31 12:39:57.712792 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:57.712342    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97464201-lbnl5" event={"ID":"c469bb8f-7712-4c3d-8bd4-af409971ef16","Type":"ContainerDied","Data":"cf8416d48a62293528b93feed181db38360846c4012c673073664232cf39285d"}
Oct 31 12:39:57.714787 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:57.714757    5060 generic.go:356] "Generic (PLEG): container finished" podID="7b44cd3d-8334-452f-937f-a3dd8904d3f2" containerID="89ef3c27df05aa7946878decc263a3f4b9de33523c2259d80108c4266465fa79" exitCode=0
Oct 31 12:39:57.714914 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:57.714890    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e622406c-mq2c9" event={"ID":"7b44cd3d-8334-452f-937f-a3dd8904d3f2","Type":"ContainerDied","Data":"89ef3c27df05aa7946878decc263a3f4b9de33523c2259d80108c4266465fa79"}
Oct 31 12:39:57.716979 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:57.716958    5060 generic.go:356] "Generic (PLEG): container finished" podID="1449a717-8c6d-4eed-b3e0-4475a92e75c3" containerID="ecc196bdfebc5a6981bf5541dce205df3e0e9d385930dbada1d73df06c858ddb" exitCode=0
Oct 31 12:39:57.717099 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:57.717032    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f07afdef-x7w7t" event={"ID":"1449a717-8c6d-4eed-b3e0-4475a92e75c3","Type":"ContainerDied","Data":"ecc196bdfebc5a6981bf5541dce205df3e0e9d385930dbada1d73df06c858ddb"}
Oct 31 12:39:57.777571 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:57.772948    5060 patch_prober.go:29] interesting pod/e2e-worker-w6-67d410-openshift-default-676b978549-rr5tg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Oct 31 12:39:57.777571 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:57.773001    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-rr5tg" podUID="8873fe2c-839e-4d23-83d1-1fe3a451b5cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Oct 31 12:39:58.770518 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:58.769903    5060 patch_prober.go:29] interesting pod/e2e-worker-w6-67d410-openshift-default-676b978549-rr5tg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Oct 31 12:39:58.770518 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:58.769973    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-rr5tg" podUID="8873fe2c-839e-4d23-83d1-1fe3a451b5cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Oct 31 12:39:58.938356 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:58.938323    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e622406c-mq2c9"
Oct 31 12:39:58.994381 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:58.994292    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b44cd3d-8334-452f-937f-a3dd8904d3f2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b44cd3d-8334-452f-937f-a3dd8904d3f2-maas-api-openshift-ca\") pod \"7b44cd3d-8334-452f-937f-a3dd8904d3f2\" (UID: \"7b44cd3d-8334-452f-937f-a3dd8904d3f2\") "
Oct 31 12:39:58.994381 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:58.994362    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7b44cd3d-8334-452f-937f-a3dd8904d3f2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b44cd3d-8334-452f-937f-a3dd8904d3f2-maas-api-xks-ca\") pod \"7b44cd3d-8334-452f-937f-a3dd8904d3f2\" (UID: \"7b44cd3d-8334-452f-937f-a3dd8904d3f2\") "
Oct 31 12:39:58.994616 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:58.994447    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b44cd3d-8334-452f-937f-a3dd8904d3f2-kube-api-access-gbtvj\" (UniqueName: \"kubernetes.io/projected/7b44cd3d-8334-452f-937f-a3dd8904d3f2-kube-api-access-gbtvj\") pod \"7b44cd3d-8334-452f-937f-a3dd8904d3f2\" (UID: \"7b44cd3d-8334-452f-937f-a3dd8904d3f2\") "
Oct 31 12:39:58.996025 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:58.995979    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b44cd3d-8334-452f-937f-a3dd8904d3f2-maas-api-openshift-ca" pod "7b44cd3d-8334-452f-937f-a3dd8904d3f2" (UID: "7b44cd3d-8334-452f-937f-a3dd8904d3f2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:39:58.999242 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:58.999213    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b44cd3d-8334-452f-937f-a3dd8904d3f2-kube-api-access-gbtvj" pod "7b44cd3d-8334-452f-937f-a3dd8904d3f2" (UID: "7b44cd3d-8334-452f-937f-a3dd8904d3f2"). InnerVolumeSpecName "kube-api-access-gbtvj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:39:59.001778 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:59.001752    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b44cd3d-8334-452f-937f-a3dd8904d3f2-maas-api-xks-ca" pod "7b44cd3d-8334-452f-937f-a3dd8904d3f2" (UID: "7b44cd3d-8334-452f-937f-a3dd8904d3f2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:39:59.002081 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:59.002062    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97464201-lbnl5"
Oct 31 12:39:59.016864 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:59.016842    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f07afdef-x7w7t"
Oct 31 12:39:59.096174 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:59.096103    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b44cd3d-8334-452f-937f-a3dd8904d3f2-maas-api-xks-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:39:59.096174 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:59.096147    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbtvj\" (UniqueName: \"kubernetes.io/projected/7b44cd3d-8334-452f-937f-a3dd8904d3f2-kube-api-access-gbtvj\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:39:59.096480 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:59.096455    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b44cd3d-8334-452f-937f-a3dd8904d3f2-maas-api-openshift-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:39:59.200458 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:59.197187    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c469bb8f-7712-4c3d-8bd4-af409971ef16-kube-api-access-7pp8r\" (UniqueName: \"kubernetes.io/projected/c469bb8f-7712-4c3d-8bd4-af409971ef16-kube-api-access-7pp8r\") pod \"c469bb8f-7712-4c3d-8bd4-af409971ef16\" (UID: \"c469bb8f-7712-4c3d-8bd4-af409971ef16\") "
Oct 31 12:39:59.200458 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:59.197247    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1449a717-8c6d-4eed-b3e0-4475a92e75c3-kube-api-access-pgwcn\" (UniqueName: \"kubernetes.io/projected/1449a717-8c6d-4eed-b3e0-4475a92e75c3-kube-api-access-pgwcn\") pod \"1449a717-8c6d-4eed-b3e0-4475a92e75c3\" (UID: \"1449a717-8c6d-4eed-b3e0-4475a92e75c3\") "
Oct 31 12:39:59.200458 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:59.197279    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c469bb8f-7712-4c3d-8bd4-af409971ef16-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c469bb8f-7712-4c3d-8bd4-af409971ef16-maas-api-xks-ca\") pod \"c469bb8f-7712-4c3d-8bd4-af409971ef16\" (UID: \"c469bb8f-7712-4c3d-8bd4-af409971ef16\") "
Oct 31 12:39:59.200458 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:59.197341    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c469bb8f-7712-4c3d-8bd4-af409971ef16-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c469bb8f-7712-4c3d-8bd4-af409971ef16-maas-api-openshift-ca\") pod \"c469bb8f-7712-4c3d-8bd4-af409971ef16\" (UID: \"c469bb8f-7712-4c3d-8bd4-af409971ef16\") "
Oct 31 12:39:59.200458 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:59.197406    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1449a717-8c6d-4eed-b3e0-4475a92e75c3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1449a717-8c6d-4eed-b3e0-4475a92e75c3-maas-api-openshift-ca\") pod \"1449a717-8c6d-4eed-b3e0-4475a92e75c3\" (UID: \"1449a717-8c6d-4eed-b3e0-4475a92e75c3\") "
Oct 31 12:39:59.200458 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:59.197529    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1449a717-8c6d-4eed-b3e0-4475a92e75c3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1449a717-8c6d-4eed-b3e0-4475a92e75c3-maas-api-xks-ca\") pod \"1449a717-8c6d-4eed-b3e0-4475a92e75c3\" (UID: \"1449a717-8c6d-4eed-b3e0-4475a92e75c3\") "
Oct 31 12:39:59.200978 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:59.200945    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c469bb8f-7712-4c3d-8bd4-af409971ef16-maas-api-openshift-ca" pod "c469bb8f-7712-4c3d-8bd4-af409971ef16" (UID: "c469bb8f-7712-4c3d-8bd4-af409971ef16"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:39:59.205737 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:59.201823    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1449a717-8c6d-4eed-b3e0-4475a92e75c3-maas-api-openshift-ca" pod "1449a717-8c6d-4eed-b3e0-4475a92e75c3" (UID: "1449a717-8c6d-4eed-b3e0-4475a92e75c3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:39:59.205737 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:59.205605    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1449a717-8c6d-4eed-b3e0-4475a92e75c3-kube-api-access-pgwcn" pod "1449a717-8c6d-4eed-b3e0-4475a92e75c3" (UID: "1449a717-8c6d-4eed-b3e0-4475a92e75c3"). InnerVolumeSpecName "kube-api-access-pgwcn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:39:59.206403 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:59.206372    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1449a717-8c6d-4eed-b3e0-4475a92e75c3-maas-api-xks-ca" pod "1449a717-8c6d-4eed-b3e0-4475a92e75c3" (UID: "1449a717-8c6d-4eed-b3e0-4475a92e75c3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:39:59.207750 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:59.207708    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c469bb8f-7712-4c3d-8bd4-af409971ef16-kube-api-access-7pp8r" pod "c469bb8f-7712-4c3d-8bd4-af409971ef16" (UID: "c469bb8f-7712-4c3d-8bd4-af409971ef16"). InnerVolumeSpecName "kube-api-access-7pp8r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:39:59.207927 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:59.207911    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c469bb8f-7712-4c3d-8bd4-af409971ef16-maas-api-xks-ca" pod "c469bb8f-7712-4c3d-8bd4-af409971ef16" (UID: "c469bb8f-7712-4c3d-8bd4-af409971ef16"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:39:59.298671 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:59.298635    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1449a717-8c6d-4eed-b3e0-4475a92e75c3-maas-api-xks-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:39:59.298671 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:59.298672    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7pp8r\" (UniqueName: \"kubernetes.io/projected/c469bb8f-7712-4c3d-8bd4-af409971ef16-kube-api-access-7pp8r\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:39:59.298889 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:59.298688    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pgwcn\" (UniqueName: \"kubernetes.io/projected/1449a717-8c6d-4eed-b3e0-4475a92e75c3-kube-api-access-pgwcn\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:39:59.298889 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:59.298701    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c469bb8f-7712-4c3d-8bd4-af409971ef16-maas-api-xks-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:39:59.298889 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:59.298718    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c469bb8f-7712-4c3d-8bd4-af409971ef16-maas-api-openshift-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:39:59.298889 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:59.298733    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1449a717-8c6d-4eed-b3e0-4475a92e75c3-maas-api-openshift-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:39:59.609452 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:59.609358    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c51ce5e-t4xfg"]
Oct 31 12:39:59.609958 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:59.609940    5060 cpu_manager.go:414] "containerMap: removing container" podUID="c469bb8f-7712-4c3d-8bd4-af409971ef16" containerName="revoke-keys"
Oct 31 12:39:59.610030 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:59.609959    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c469bb8f-7712-4c3d-8bd4-af409971ef16" containerName="revoke-keys"
Oct 31 12:39:59.610030 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:59.609979    5060 cpu_manager.go:414] "containerMap: removing container" podUID="1449a717-8c6d-4eed-b3e0-4475a92e75c3" containerName="revoke-keys"
Oct 31 12:39:59.610030 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:59.609984    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1449a717-8c6d-4eed-b3e0-4475a92e75c3" containerName="revoke-keys"
Oct 31 12:39:59.610030 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:59.610001    5060 cpu_manager.go:414] "containerMap: removing container" podUID="7b44cd3d-8334-452f-937f-a3dd8904d3f2" containerName="revoke-keys"
Oct 31 12:39:59.610030 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:59.610006    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b44cd3d-8334-452f-937f-a3dd8904d3f2" containerName="revoke-keys"
Oct 31 12:39:59.610286 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:59.610087    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="c469bb8f-7712-4c3d-8bd4-af409971ef16" containerName="revoke-keys"
Oct 31 12:39:59.610286 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:59.610096    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="1449a717-8c6d-4eed-b3e0-4475a92e75c3" containerName="revoke-keys"
Oct 31 12:39:59.610286 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:59.610105    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b44cd3d-8334-452f-937f-a3dd8904d3f2" containerName="revoke-keys"
Oct 31 12:39:59.613642 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:59.613621    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c51ce5e-t4xfg"
Oct 31 12:39:59.623562 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:59.623518    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c51ce5e-t4xfg"]
Oct 31 12:39:59.702323 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:59.702286    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c3204bd-0aa1-4adb-877f-e435f0a1725a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6c51ce5e-t4xfg\" (UID: \"3c3204bd-0aa1-4adb-877f-e435f0a1725a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c51ce5e-t4xfg"
Oct 31 12:39:59.702481 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:59.702444    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c3204bd-0aa1-4adb-877f-e435f0a1725a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6c51ce5e-t4xfg\" (UID: \"3c3204bd-0aa1-4adb-877f-e435f0a1725a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c51ce5e-t4xfg"
Oct 31 12:39:59.702481 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:59.702475    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zr7zj\" (UniqueName: \"kubernetes.io/projected/3c3204bd-0aa1-4adb-877f-e435f0a1725a-kube-api-access-zr7zj\") pod \"maas-api-revoke-sub-6c51ce5e-t4xfg\" (UID: \"3c3204bd-0aa1-4adb-877f-e435f0a1725a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c51ce5e-t4xfg"
Oct 31 12:39:59.735351 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:59.735312    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f07afdef-x7w7t" event={"ID":"1449a717-8c6d-4eed-b3e0-4475a92e75c3","Type":"ContainerDied","Data":"ac88b6cfd0dffe559110146346e39ecdf73f3bf1b0153685699fa82908f8f7c6"}
Oct 31 12:39:59.735562 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:59.735379    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ac88b6cfd0dffe559110146346e39ecdf73f3bf1b0153685699fa82908f8f7c6"
Oct 31 12:39:59.735562 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:59.735323    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f07afdef-x7w7t"
Oct 31 12:39:59.737220 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:59.737199    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97464201-lbnl5"
Oct 31 12:39:59.737327 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:59.737236    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97464201-lbnl5" event={"ID":"c469bb8f-7712-4c3d-8bd4-af409971ef16","Type":"ContainerDied","Data":"e0fbb673b41939bbbcac73f0c7a735b58b7ca41f12352081e2726009506bad33"}
Oct 31 12:39:59.737327 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:59.737285    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e0fbb673b41939bbbcac73f0c7a735b58b7ca41f12352081e2726009506bad33"
Oct 31 12:39:59.738820 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:59.738800    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e622406c-mq2c9" event={"ID":"7b44cd3d-8334-452f-937f-a3dd8904d3f2","Type":"ContainerDied","Data":"6554abeddf5e45950f763d84d971c4eb2139a21775ffe58ee24389f32e0de91e"}
Oct 31 12:39:59.738935 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:59.738832    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6554abeddf5e45950f763d84d971c4eb2139a21775ffe58ee24389f32e0de91e"
Oct 31 12:39:59.738935 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:59.738839    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e622406c-mq2c9"
Oct 31 12:39:59.778513 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:59.767927    5060 patch_prober.go:29] interesting pod/e2e-worker-w6-67d410-openshift-default-676b978549-rr5tg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Oct 31 12:39:59.778513 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:59.767984    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-rr5tg" podUID="8873fe2c-839e-4d23-83d1-1fe3a451b5cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Oct 31 12:39:59.803214 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:59.803175    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c3204bd-0aa1-4adb-877f-e435f0a1725a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6c51ce5e-t4xfg\" (UID: \"3c3204bd-0aa1-4adb-877f-e435f0a1725a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c51ce5e-t4xfg"
Oct 31 12:39:59.803361 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:59.803292    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c3204bd-0aa1-4adb-877f-e435f0a1725a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6c51ce5e-t4xfg\" (UID: \"3c3204bd-0aa1-4adb-877f-e435f0a1725a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c51ce5e-t4xfg"
Oct 31 12:39:59.803361 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:59.803318    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zr7zj\" (UniqueName: \"kubernetes.io/projected/3c3204bd-0aa1-4adb-877f-e435f0a1725a-kube-api-access-zr7zj\") pod \"maas-api-revoke-sub-6c51ce5e-t4xfg\" (UID: \"3c3204bd-0aa1-4adb-877f-e435f0a1725a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c51ce5e-t4xfg"
Oct 31 12:39:59.803900 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:59.803875    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c3204bd-0aa1-4adb-877f-e435f0a1725a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6c51ce5e-t4xfg\" (UID: \"3c3204bd-0aa1-4adb-877f-e435f0a1725a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c51ce5e-t4xfg"
Oct 31 12:39:59.805720 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:59.805696    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c3204bd-0aa1-4adb-877f-e435f0a1725a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6c51ce5e-t4xfg\" (UID: \"3c3204bd-0aa1-4adb-877f-e435f0a1725a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c51ce5e-t4xfg"
Oct 31 12:39:59.817804 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:59.817776    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zr7zj\" (UniqueName: \"kubernetes.io/projected/3c3204bd-0aa1-4adb-877f-e435f0a1725a-kube-api-access-zr7zj\") pod \"maas-api-revoke-sub-6c51ce5e-t4xfg\" (UID: \"3c3204bd-0aa1-4adb-877f-e435f0a1725a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c51ce5e-t4xfg"
Oct 31 12:39:59.922047 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:39:59.922015    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7phb" containerName="extractor"
Oct 31 12:39:59.929347 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:39:59.929323    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c51ce5e-t4xfg"
Oct 31 12:40:00.272640 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:00.272608    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c51ce5e-t4xfg"]
Oct 31 12:40:00.273067 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:40:00.273045    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c3204bd_0aa1_4adb_877f_e435f0a1725a.slice/crio-83d9e141789d01edc55413875e972201f16849faaeeaa28b4a87b205e3557b30 WatchSource:0}: Error finding container 83d9e141789d01edc55413875e972201f16849faaeeaa28b4a87b205e3557b30: Status 404 returned error can't find the container with id 83d9e141789d01edc55413875e972201f16849faaeeaa28b4a87b205e3557b30
Oct 31 12:40:00.749323 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:00.749270    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c51ce5e-t4xfg" event={"ID":"3c3204bd-0aa1-4adb-877f-e435f0a1725a","Type":"ContainerStarted","Data":"4433fc46a6142233b4f8c08806a0b6c4e9c3c05946e4b8537f34ef2128958efc"}
Oct 31 12:40:00.749475 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:00.749353    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c51ce5e-t4xfg" event={"ID":"3c3204bd-0aa1-4adb-877f-e435f0a1725a","Type":"ContainerStarted","Data":"83d9e141789d01edc55413875e972201f16849faaeeaa28b4a87b205e3557b30"}
Oct 31 12:40:00.767439 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:00.767408    5060 patch_prober.go:29] interesting pod/e2e-worker-w6-67d410-openshift-default-676b978549-rr5tg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Oct 31 12:40:00.767603 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:00.767460    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-rr5tg" podUID="8873fe2c-839e-4d23-83d1-1fe3a451b5cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Oct 31 12:40:00.778011 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:00.777799    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c51ce5e-t4xfg" podStartSLOduration=1.777777458 podStartE2EDuration="1.777777458s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:39:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:40:00.776528133 +0000 UTC m=+763.879595845" watchObservedRunningTime="2026-10-31 12:40:00.777777458 +0000 UTC m=+763.880845171"
Oct 31 12:40:01.759475 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:01.759428    5060 generic.go:356] "Generic (PLEG): container finished" podID="3c3204bd-0aa1-4adb-877f-e435f0a1725a" containerID="4433fc46a6142233b4f8c08806a0b6c4e9c3c05946e4b8537f34ef2128958efc" exitCode=0
Oct 31 12:40:01.759874 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:01.759586    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c51ce5e-t4xfg" event={"ID":"3c3204bd-0aa1-4adb-877f-e435f0a1725a","Type":"ContainerDied","Data":"4433fc46a6142233b4f8c08806a0b6c4e9c3c05946e4b8537f34ef2128958efc"}
Oct 31 12:40:01.772002 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:01.771973    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-rr5tg"
Oct 31 12:40:01.772271 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:01.772244    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-rr5tg"
Oct 31 12:40:01.773226 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:01.773206    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-rr5tg"
Oct 31 12:40:02.942533 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:02.942473    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c51ce5e-t4xfg"
Oct 31 12:40:03.042186 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:03.042139    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c3204bd-0aa1-4adb-877f-e435f0a1725a-kube-api-access-zr7zj\" (UniqueName: \"kubernetes.io/projected/3c3204bd-0aa1-4adb-877f-e435f0a1725a-kube-api-access-zr7zj\") pod \"3c3204bd-0aa1-4adb-877f-e435f0a1725a\" (UID: \"3c3204bd-0aa1-4adb-877f-e435f0a1725a\") "
Oct 31 12:40:03.042363 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:03.042272    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3c3204bd-0aa1-4adb-877f-e435f0a1725a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c3204bd-0aa1-4adb-877f-e435f0a1725a-maas-api-xks-ca\") pod \"3c3204bd-0aa1-4adb-877f-e435f0a1725a\" (UID: \"3c3204bd-0aa1-4adb-877f-e435f0a1725a\") "
Oct 31 12:40:03.042363 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:03.042336    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c3204bd-0aa1-4adb-877f-e435f0a1725a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c3204bd-0aa1-4adb-877f-e435f0a1725a-maas-api-openshift-ca\") pod \"3c3204bd-0aa1-4adb-877f-e435f0a1725a\" (UID: \"3c3204bd-0aa1-4adb-877f-e435f0a1725a\") "
Oct 31 12:40:03.042767 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:03.042734    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c3204bd-0aa1-4adb-877f-e435f0a1725a-maas-api-openshift-ca" pod "3c3204bd-0aa1-4adb-877f-e435f0a1725a" (UID: "3c3204bd-0aa1-4adb-877f-e435f0a1725a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:40:03.044925 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:03.044895    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c3204bd-0aa1-4adb-877f-e435f0a1725a-maas-api-xks-ca" pod "3c3204bd-0aa1-4adb-877f-e435f0a1725a" (UID: "3c3204bd-0aa1-4adb-877f-e435f0a1725a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:40:03.045021 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:03.044921    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c3204bd-0aa1-4adb-877f-e435f0a1725a-kube-api-access-zr7zj" pod "3c3204bd-0aa1-4adb-877f-e435f0a1725a" (UID: "3c3204bd-0aa1-4adb-877f-e435f0a1725a"). InnerVolumeSpecName "kube-api-access-zr7zj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:40:03.144251 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:03.144211    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zr7zj\" (UniqueName: \"kubernetes.io/projected/3c3204bd-0aa1-4adb-877f-e435f0a1725a-kube-api-access-zr7zj\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:40:03.144251 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:03.144254    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c3204bd-0aa1-4adb-877f-e435f0a1725a-maas-api-xks-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:40:03.144251 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:03.144271    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c3204bd-0aa1-4adb-877f-e435f0a1725a-maas-api-openshift-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:40:03.572909 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:03.572876    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3eb4bba9-52x2t"]
Oct 31 12:40:03.573434 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:03.573421    5060 cpu_manager.go:414] "containerMap: removing container" podUID="3c3204bd-0aa1-4adb-877f-e435f0a1725a" containerName="revoke-keys"
Oct 31 12:40:03.573521 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:03.573435    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c3204bd-0aa1-4adb-877f-e435f0a1725a" containerName="revoke-keys"
Oct 31 12:40:03.573572 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:03.573557    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c3204bd-0aa1-4adb-877f-e435f0a1725a" containerName="revoke-keys"
Oct 31 12:40:03.577208 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:03.577186    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3eb4bba9-52x2t"
Oct 31 12:40:03.590341 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:03.590315    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3eb4bba9-52x2t"]
Oct 31 12:40:03.649461 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:03.649426    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmbfn\" (UniqueName: \"kubernetes.io/projected/627354b5-bbf0-46b0-80f6-5a0ce90b377c-kube-api-access-nmbfn\") pod \"maas-api-revoke-sub-3eb4bba9-52x2t\" (UID: \"627354b5-bbf0-46b0-80f6-5a0ce90b377c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3eb4bba9-52x2t"
Oct 31 12:40:03.649628 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:03.649521    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/627354b5-bbf0-46b0-80f6-5a0ce90b377c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3eb4bba9-52x2t\" (UID: \"627354b5-bbf0-46b0-80f6-5a0ce90b377c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3eb4bba9-52x2t"
Oct 31 12:40:03.649690 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:03.649666    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/627354b5-bbf0-46b0-80f6-5a0ce90b377c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3eb4bba9-52x2t\" (UID: \"627354b5-bbf0-46b0-80f6-5a0ce90b377c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3eb4bba9-52x2t"
Oct 31 12:40:03.750931 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:03.750888    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/627354b5-bbf0-46b0-80f6-5a0ce90b377c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3eb4bba9-52x2t\" (UID: \"627354b5-bbf0-46b0-80f6-5a0ce90b377c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3eb4bba9-52x2t"
Oct 31 12:40:03.751136 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:03.751001    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/627354b5-bbf0-46b0-80f6-5a0ce90b377c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3eb4bba9-52x2t\" (UID: \"627354b5-bbf0-46b0-80f6-5a0ce90b377c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3eb4bba9-52x2t"
Oct 31 12:40:03.751136 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:03.751055    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmbfn\" (UniqueName: \"kubernetes.io/projected/627354b5-bbf0-46b0-80f6-5a0ce90b377c-kube-api-access-nmbfn\") pod \"maas-api-revoke-sub-3eb4bba9-52x2t\" (UID: \"627354b5-bbf0-46b0-80f6-5a0ce90b377c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3eb4bba9-52x2t"
Oct 31 12:40:03.751601 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:03.751575    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/627354b5-bbf0-46b0-80f6-5a0ce90b377c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3eb4bba9-52x2t\" (UID: \"627354b5-bbf0-46b0-80f6-5a0ce90b377c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3eb4bba9-52x2t"
Oct 31 12:40:03.753437 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:03.753402    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/627354b5-bbf0-46b0-80f6-5a0ce90b377c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3eb4bba9-52x2t\" (UID: \"627354b5-bbf0-46b0-80f6-5a0ce90b377c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3eb4bba9-52x2t"
Oct 31 12:40:03.771250 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:03.771224    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmbfn\" (UniqueName: \"kubernetes.io/projected/627354b5-bbf0-46b0-80f6-5a0ce90b377c-kube-api-access-nmbfn\") pod \"maas-api-revoke-sub-3eb4bba9-52x2t\" (UID: \"627354b5-bbf0-46b0-80f6-5a0ce90b377c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3eb4bba9-52x2t"
Oct 31 12:40:03.776468 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:03.776446    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c51ce5e-t4xfg"
Oct 31 12:40:03.776630 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:03.776473    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c51ce5e-t4xfg" event={"ID":"3c3204bd-0aa1-4adb-877f-e435f0a1725a","Type":"ContainerDied","Data":"83d9e141789d01edc55413875e972201f16849faaeeaa28b4a87b205e3557b30"}
Oct 31 12:40:03.776630 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:03.776565    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="83d9e141789d01edc55413875e972201f16849faaeeaa28b4a87b205e3557b30"
Oct 31 12:40:03.895382 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:03.895289    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3eb4bba9-52x2t"
Oct 31 12:40:04.182330 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:40:04.182288    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod627354b5_bbf0_46b0_80f6_5a0ce90b377c.slice/crio-81548d1ed81f619f4ebcb942b4ff5a7cc0d861dc0bf66727282d68779ecfb594 WatchSource:0}: Error finding container 81548d1ed81f619f4ebcb942b4ff5a7cc0d861dc0bf66727282d68779ecfb594: Status 404 returned error can't find the container with id 81548d1ed81f619f4ebcb942b4ff5a7cc0d861dc0bf66727282d68779ecfb594
Oct 31 12:40:04.184437 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:04.184403    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3eb4bba9-52x2t"]
Oct 31 12:40:04.788898 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:04.788862    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3eb4bba9-52x2t" event={"ID":"627354b5-bbf0-46b0-80f6-5a0ce90b377c","Type":"ContainerStarted","Data":"f2ec50798bf391a0703c14078d553b13d6e5c5838f4fdd9f6b3faf8d75860f29"}
Oct 31 12:40:04.789077 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:04.788929    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3eb4bba9-52x2t" event={"ID":"627354b5-bbf0-46b0-80f6-5a0ce90b377c","Type":"ContainerStarted","Data":"81548d1ed81f619f4ebcb942b4ff5a7cc0d861dc0bf66727282d68779ecfb594"}
Oct 31 12:40:04.821525 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:04.821451    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3eb4bba9-52x2t" podStartSLOduration=1.821427691 podStartE2EDuration="1.821427691s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:40:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:40:04.820687647 +0000 UTC m=+767.923755358" watchObservedRunningTime="2026-10-31 12:40:04.821427691 +0000 UTC m=+767.924495405"
Oct 31 12:40:05.799936 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:05.799897    5060 generic.go:356] "Generic (PLEG): container finished" podID="627354b5-bbf0-46b0-80f6-5a0ce90b377c" containerID="f2ec50798bf391a0703c14078d553b13d6e5c5838f4fdd9f6b3faf8d75860f29" exitCode=0
Oct 31 12:40:05.800321 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:05.799979    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3eb4bba9-52x2t" event={"ID":"627354b5-bbf0-46b0-80f6-5a0ce90b377c","Type":"ContainerDied","Data":"f2ec50798bf391a0703c14078d553b13d6e5c5838f4fdd9f6b3faf8d75860f29"}
Oct 31 12:40:06.954282 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:06.954256    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3eb4bba9-52x2t"
Oct 31 12:40:06.990369 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:06.990339    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/627354b5-bbf0-46b0-80f6-5a0ce90b377c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/627354b5-bbf0-46b0-80f6-5a0ce90b377c-maas-api-xks-ca\") pod \"627354b5-bbf0-46b0-80f6-5a0ce90b377c\" (UID: \"627354b5-bbf0-46b0-80f6-5a0ce90b377c\") "
Oct 31 12:40:06.990650 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:06.990399    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/627354b5-bbf0-46b0-80f6-5a0ce90b377c-kube-api-access-nmbfn\" (UniqueName: \"kubernetes.io/projected/627354b5-bbf0-46b0-80f6-5a0ce90b377c-kube-api-access-nmbfn\") pod \"627354b5-bbf0-46b0-80f6-5a0ce90b377c\" (UID: \"627354b5-bbf0-46b0-80f6-5a0ce90b377c\") "
Oct 31 12:40:06.990650 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:06.990609    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/627354b5-bbf0-46b0-80f6-5a0ce90b377c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/627354b5-bbf0-46b0-80f6-5a0ce90b377c-maas-api-openshift-ca\") pod \"627354b5-bbf0-46b0-80f6-5a0ce90b377c\" (UID: \"627354b5-bbf0-46b0-80f6-5a0ce90b377c\") "
Oct 31 12:40:06.991285 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:06.991259    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/627354b5-bbf0-46b0-80f6-5a0ce90b377c-maas-api-openshift-ca" pod "627354b5-bbf0-46b0-80f6-5a0ce90b377c" (UID: "627354b5-bbf0-46b0-80f6-5a0ce90b377c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:40:07.004659 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:07.004632    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/627354b5-bbf0-46b0-80f6-5a0ce90b377c-maas-api-xks-ca" pod "627354b5-bbf0-46b0-80f6-5a0ce90b377c" (UID: "627354b5-bbf0-46b0-80f6-5a0ce90b377c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:40:07.004783 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:07.004667    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/627354b5-bbf0-46b0-80f6-5a0ce90b377c-kube-api-access-nmbfn" pod "627354b5-bbf0-46b0-80f6-5a0ce90b377c" (UID: "627354b5-bbf0-46b0-80f6-5a0ce90b377c"). InnerVolumeSpecName "kube-api-access-nmbfn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:40:07.091958 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:07.091924    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/627354b5-bbf0-46b0-80f6-5a0ce90b377c-maas-api-openshift-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:40:07.091958 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:07.091956    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/627354b5-bbf0-46b0-80f6-5a0ce90b377c-maas-api-xks-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:40:07.092166 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:07.091971    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nmbfn\" (UniqueName: \"kubernetes.io/projected/627354b5-bbf0-46b0-80f6-5a0ce90b377c-kube-api-access-nmbfn\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:40:07.121514 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:07.121459    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-2fn6t"]
Oct 31 12:40:07.122061 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:07.122048    5060 cpu_manager.go:414] "containerMap: removing container" podUID="627354b5-bbf0-46b0-80f6-5a0ce90b377c" containerName="revoke-keys"
Oct 31 12:40:07.122107 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:07.122062    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="627354b5-bbf0-46b0-80f6-5a0ce90b377c" containerName="revoke-keys"
Oct 31 12:40:07.122159 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:07.122151    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="627354b5-bbf0-46b0-80f6-5a0ce90b377c" containerName="revoke-keys"
Oct 31 12:40:07.125896 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:07.125875    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-2fn6t"
Oct 31 12:40:07.154012 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:07.153982    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-2fn6t"]
Oct 31 12:40:07.193471 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:07.193432    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60d395e9-e4e9-4c68-ab10-9ef5371df1b1-credential-socket\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-2fn6t\" (UID: \"60d395e9-e4e9-4c68-ab10-9ef5371df1b1\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-2fn6t"
Oct 31 12:40:07.193658 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:07.193485    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60d395e9-e4e9-4c68-ab10-9ef5371df1b1-istio-podinfo\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-2fn6t\" (UID: \"60d395e9-e4e9-4c68-ab10-9ef5371df1b1\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-2fn6t"
Oct 31 12:40:07.193658 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:07.193634    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60d395e9-e4e9-4c68-ab10-9ef5371df1b1-istio-envoy\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-2fn6t\" (UID: \"60d395e9-e4e9-4c68-ab10-9ef5371df1b1\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-2fn6t"
Oct 31 12:40:07.193783 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:07.193747    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60d395e9-e4e9-4c68-ab10-9ef5371df1b1-workload-certs\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-2fn6t\" (UID: \"60d395e9-e4e9-4c68-ab10-9ef5371df1b1\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-2fn6t"
Oct 31 12:40:07.193835 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:07.193819    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bzqq\" (UniqueName: \"kubernetes.io/projected/60d395e9-e4e9-4c68-ab10-9ef5371df1b1-kube-api-access-4bzqq\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-2fn6t\" (UID: \"60d395e9-e4e9-4c68-ab10-9ef5371df1b1\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-2fn6t"
Oct 31 12:40:07.194008 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:07.193985    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60d395e9-e4e9-4c68-ab10-9ef5371df1b1-istio-token\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-2fn6t\" (UID: \"60d395e9-e4e9-4c68-ab10-9ef5371df1b1\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-2fn6t"
Oct 31 12:40:07.194066 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:07.194051    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60d395e9-e4e9-4c68-ab10-9ef5371df1b1-istiod-ca-cert\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-2fn6t\" (UID: \"60d395e9-e4e9-4c68-ab10-9ef5371df1b1\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-2fn6t"
Oct 31 12:40:07.194126 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:07.194109    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60d395e9-e4e9-4c68-ab10-9ef5371df1b1-workload-socket\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-2fn6t\" (UID: \"60d395e9-e4e9-4c68-ab10-9ef5371df1b1\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-2fn6t"
Oct 31 12:40:07.194193 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:07.194173    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60d395e9-e4e9-4c68-ab10-9ef5371df1b1-istio-data\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-2fn6t\" (UID: \"60d395e9-e4e9-4c68-ab10-9ef5371df1b1\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-2fn6t"
Oct 31 12:40:07.265123 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:07.265089    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-l8n27"]
Oct 31 12:40:07.269728 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:07.269706    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-l8n27"
Oct 31 12:40:07.294287 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:07.294252    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-l8n27"]
Oct 31 12:40:07.294984 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:07.294945    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5c79f52-3049-4611-a935-ef2f4eb5e0bd-istio-data\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-l8n27\" (UID: \"c5c79f52-3049-4611-a935-ef2f4eb5e0bd\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-l8n27"
Oct 31 12:40:07.295207 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:07.295183    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60d395e9-e4e9-4c68-ab10-9ef5371df1b1-istiod-ca-cert\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-2fn6t\" (UID: \"60d395e9-e4e9-4c68-ab10-9ef5371df1b1\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-2fn6t"
Oct 31 12:40:07.295353 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:07.295334    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60d395e9-e4e9-4c68-ab10-9ef5371df1b1-workload-socket\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-2fn6t\" (UID: \"60d395e9-e4e9-4c68-ab10-9ef5371df1b1\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-2fn6t"
Oct 31 12:40:07.295510 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:07.295477    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60d395e9-e4e9-4c68-ab10-9ef5371df1b1-istio-data\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-2fn6t\" (UID: \"60d395e9-e4e9-4c68-ab10-9ef5371df1b1\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-2fn6t"
Oct 31 12:40:07.295643 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:07.295626    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60d395e9-e4e9-4c68-ab10-9ef5371df1b1-credential-socket\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-2fn6t\" (UID: \"60d395e9-e4e9-4c68-ab10-9ef5371df1b1\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-2fn6t"
Oct 31 12:40:07.295762 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:07.295744    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5c79f52-3049-4611-a935-ef2f4eb5e0bd-workload-certs\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-l8n27\" (UID: \"c5c79f52-3049-4611-a935-ef2f4eb5e0bd\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-l8n27"
Oct 31 12:40:07.295874 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:07.295859    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60d395e9-e4e9-4c68-ab10-9ef5371df1b1-istio-podinfo\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-2fn6t\" (UID: \"60d395e9-e4e9-4c68-ab10-9ef5371df1b1\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-2fn6t"
Oct 31 12:40:07.295995 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:07.295979    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60d395e9-e4e9-4c68-ab10-9ef5371df1b1-istio-envoy\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-2fn6t\" (UID: \"60d395e9-e4e9-4c68-ab10-9ef5371df1b1\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-2fn6t"
Oct 31 12:40:07.296116 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:07.296101    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5c79f52-3049-4611-a935-ef2f4eb5e0bd-istiod-ca-cert\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-l8n27\" (UID: \"c5c79f52-3049-4611-a935-ef2f4eb5e0bd\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-l8n27"
Oct 31 12:40:07.296235 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:07.296219    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5c79f52-3049-4611-a935-ef2f4eb5e0bd-istio-envoy\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-l8n27\" (UID: \"c5c79f52-3049-4611-a935-ef2f4eb5e0bd\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-l8n27"
Oct 31 12:40:07.296359 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:07.296341    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60d395e9-e4e9-4c68-ab10-9ef5371df1b1-workload-certs\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-2fn6t\" (UID: \"60d395e9-e4e9-4c68-ab10-9ef5371df1b1\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-2fn6t"
Oct 31 12:40:07.296539 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:07.296479    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhqld\" (UniqueName: \"kubernetes.io/projected/c5c79f52-3049-4611-a935-ef2f4eb5e0bd-kube-api-access-nhqld\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-l8n27\" (UID: \"c5c79f52-3049-4611-a935-ef2f4eb5e0bd\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-l8n27"
Oct 31 12:40:07.296650 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:07.296556    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60d395e9-e4e9-4c68-ab10-9ef5371df1b1-credential-socket\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-2fn6t\" (UID: \"60d395e9-e4e9-4c68-ab10-9ef5371df1b1\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-2fn6t"
Oct 31 12:40:07.296742 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:07.296612    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60d395e9-e4e9-4c68-ab10-9ef5371df1b1-istio-data\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-2fn6t\" (UID: \"60d395e9-e4e9-4c68-ab10-9ef5371df1b1\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-2fn6t"
Oct 31 12:40:07.296834 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:07.296690    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bzqq\" (UniqueName: \"kubernetes.io/projected/60d395e9-e4e9-4c68-ab10-9ef5371df1b1-kube-api-access-4bzqq\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-2fn6t\" (UID: \"60d395e9-e4e9-4c68-ab10-9ef5371df1b1\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-2fn6t"
Oct 31 12:40:07.296942 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:07.296927    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5c79f52-3049-4611-a935-ef2f4eb5e0bd-istio-podinfo\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-l8n27\" (UID: \"c5c79f52-3049-4611-a935-ef2f4eb5e0bd\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-l8n27"
Oct 31 12:40:07.297041 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:07.297020    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60d395e9-e4e9-4c68-ab10-9ef5371df1b1-workload-certs\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-2fn6t\" (UID: \"60d395e9-e4e9-4c68-ab10-9ef5371df1b1\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-2fn6t"
Oct 31 12:40:07.297106 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:07.296774    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60d395e9-e4e9-4c68-ab10-9ef5371df1b1-workload-socket\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-2fn6t\" (UID: \"60d395e9-e4e9-4c68-ab10-9ef5371df1b1\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-2fn6t"
Oct 31 12:40:07.297187 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:07.296229    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60d395e9-e4e9-4c68-ab10-9ef5371df1b1-istiod-ca-cert\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-2fn6t\" (UID: \"60d395e9-e4e9-4c68-ab10-9ef5371df1b1\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-2fn6t"
Oct 31 12:40:07.298629 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:07.298599    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5c79f52-3049-4611-a935-ef2f4eb5e0bd-istio-token\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-l8n27\" (UID: \"c5c79f52-3049-4611-a935-ef2f4eb5e0bd\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-l8n27"
Oct 31 12:40:07.298748 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:07.298730    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5c79f52-3049-4611-a935-ef2f4eb5e0bd-credential-socket\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-l8n27\" (UID: \"c5c79f52-3049-4611-a935-ef2f4eb5e0bd\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-l8n27"
Oct 31 12:40:07.298831 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:07.298809    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60d395e9-e4e9-4c68-ab10-9ef5371df1b1-istio-token\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-2fn6t\" (UID: \"60d395e9-e4e9-4c68-ab10-9ef5371df1b1\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-2fn6t"
Oct 31 12:40:07.299123 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:07.299098    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5c79f52-3049-4611-a935-ef2f4eb5e0bd-workload-socket\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-l8n27\" (UID: \"c5c79f52-3049-4611-a935-ef2f4eb5e0bd\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-l8n27"
Oct 31 12:40:07.309480 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:07.309425    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60d395e9-e4e9-4c68-ab10-9ef5371df1b1-istio-envoy\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-2fn6t\" (UID: \"60d395e9-e4e9-4c68-ab10-9ef5371df1b1\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-2fn6t"
Oct 31 12:40:07.309917 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:07.309879    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60d395e9-e4e9-4c68-ab10-9ef5371df1b1-istio-podinfo\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-2fn6t\" (UID: \"60d395e9-e4e9-4c68-ab10-9ef5371df1b1\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-2fn6t"
Oct 31 12:40:07.322447 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:07.322389    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60d395e9-e4e9-4c68-ab10-9ef5371df1b1-istio-token\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-2fn6t\" (UID: \"60d395e9-e4e9-4c68-ab10-9ef5371df1b1\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-2fn6t"
Oct 31 12:40:07.322729 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:07.322707    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bzqq\" (UniqueName: \"kubernetes.io/projected/60d395e9-e4e9-4c68-ab10-9ef5371df1b1-kube-api-access-4bzqq\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-2fn6t\" (UID: \"60d395e9-e4e9-4c68-ab10-9ef5371df1b1\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-2fn6t"
Oct 31 12:40:07.386035 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:07.385986    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-gxs26"]
Oct 31 12:40:07.391859 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:07.391828    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-gxs26"
Oct 31 12:40:07.399851 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:07.399819    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5c79f52-3049-4611-a935-ef2f4eb5e0bd-istio-token\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-l8n27\" (UID: \"c5c79f52-3049-4611-a935-ef2f4eb5e0bd\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-l8n27"
Oct 31 12:40:07.400011 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:07.399940    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5c79f52-3049-4611-a935-ef2f4eb5e0bd-credential-socket\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-l8n27\" (UID: \"c5c79f52-3049-4611-a935-ef2f4eb5e0bd\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-l8n27"
Oct 31 12:40:07.400011 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:07.399985    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5c79f52-3049-4611-a935-ef2f4eb5e0bd-workload-socket\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-l8n27\" (UID: \"c5c79f52-3049-4611-a935-ef2f4eb5e0bd\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-l8n27"
Oct 31 12:40:07.400148 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:07.400019    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5c79f52-3049-4611-a935-ef2f4eb5e0bd-istio-data\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-l8n27\" (UID: \"c5c79f52-3049-4611-a935-ef2f4eb5e0bd\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-l8n27"
Oct 31 12:40:07.400148 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:07.400078    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5c79f52-3049-4611-a935-ef2f4eb5e0bd-workload-certs\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-l8n27\" (UID: \"c5c79f52-3049-4611-a935-ef2f4eb5e0bd\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-l8n27"
Oct 31 12:40:07.400148 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:07.400124    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5c79f52-3049-4611-a935-ef2f4eb5e0bd-istiod-ca-cert\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-l8n27\" (UID: \"c5c79f52-3049-4611-a935-ef2f4eb5e0bd\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-l8n27"
Oct 31 12:40:07.400312 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:07.400163    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5c79f52-3049-4611-a935-ef2f4eb5e0bd-istio-envoy\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-l8n27\" (UID: \"c5c79f52-3049-4611-a935-ef2f4eb5e0bd\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-l8n27"
Oct 31 12:40:07.400312 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:07.400221    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhqld\" (UniqueName: \"kubernetes.io/projected/c5c79f52-3049-4611-a935-ef2f4eb5e0bd-kube-api-access-nhqld\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-l8n27\" (UID: \"c5c79f52-3049-4611-a935-ef2f4eb5e0bd\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-l8n27"
Oct 31 12:40:07.400312 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:07.400262    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5c79f52-3049-4611-a935-ef2f4eb5e0bd-istio-podinfo\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-l8n27\" (UID: \"c5c79f52-3049-4611-a935-ef2f4eb5e0bd\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-l8n27"
Oct 31 12:40:07.400470 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:07.400386    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5c79f52-3049-4611-a935-ef2f4eb5e0bd-credential-socket\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-l8n27\" (UID: \"c5c79f52-3049-4611-a935-ef2f4eb5e0bd\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-l8n27"
Oct 31 12:40:07.400710 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:07.400683    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5c79f52-3049-4611-a935-ef2f4eb5e0bd-workload-certs\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-l8n27\" (UID: \"c5c79f52-3049-4611-a935-ef2f4eb5e0bd\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-l8n27"
Oct 31 12:40:07.400989 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:07.400966    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5c79f52-3049-4611-a935-ef2f4eb5e0bd-workload-socket\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-l8n27\" (UID: \"c5c79f52-3049-4611-a935-ef2f4eb5e0bd\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-l8n27"
Oct 31 12:40:07.401269 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:07.401248    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5c79f52-3049-4611-a935-ef2f4eb5e0bd-istio-data\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-l8n27\" (UID: \"c5c79f52-3049-4611-a935-ef2f4eb5e0bd\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-l8n27"
Oct 31 12:40:07.401917 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:07.401889    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5c79f52-3049-4611-a935-ef2f4eb5e0bd-istiod-ca-cert\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-l8n27\" (UID: \"c5c79f52-3049-4611-a935-ef2f4eb5e0bd\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-l8n27"
Oct 31 12:40:07.403927 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:07.403884    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5c79f52-3049-4611-a935-ef2f4eb5e0bd-istio-envoy\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-l8n27\" (UID: \"c5c79f52-3049-4611-a935-ef2f4eb5e0bd\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-l8n27"
Oct 31 12:40:07.412727 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:07.412705    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-gxs26"]
Oct 31 12:40:07.413166 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:07.413147    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5c79f52-3049-4611-a935-ef2f4eb5e0bd-istio-podinfo\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-l8n27\" (UID: \"c5c79f52-3049-4611-a935-ef2f4eb5e0bd\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-l8n27"
Oct 31 12:40:07.421417 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:07.421393    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhqld\" (UniqueName: \"kubernetes.io/projected/c5c79f52-3049-4611-a935-ef2f4eb5e0bd-kube-api-access-nhqld\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-l8n27\" (UID: \"c5c79f52-3049-4611-a935-ef2f4eb5e0bd\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-l8n27"
Oct 31 12:40:07.422696 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:07.422673    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5c79f52-3049-4611-a935-ef2f4eb5e0bd-istio-token\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-l8n27\" (UID: \"c5c79f52-3049-4611-a935-ef2f4eb5e0bd\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-l8n27"
Oct 31 12:40:07.439710 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:07.439684    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-2fn6t"
Oct 31 12:40:07.502219 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:07.501634    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86c84864-127f-4507-8f60-2dbb684c687a-workload-certs\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-gxs26\" (UID: \"86c84864-127f-4507-8f60-2dbb684c687a\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-gxs26"
Oct 31 12:40:07.502219 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:07.501768    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86c84864-127f-4507-8f60-2dbb684c687a-istiod-ca-cert\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-gxs26\" (UID: \"86c84864-127f-4507-8f60-2dbb684c687a\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-gxs26"
Oct 31 12:40:07.502219 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:07.501793    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86c84864-127f-4507-8f60-2dbb684c687a-istio-podinfo\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-gxs26\" (UID: \"86c84864-127f-4507-8f60-2dbb684c687a\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-gxs26"
Oct 31 12:40:07.502219 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:07.501861    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86c84864-127f-4507-8f60-2dbb684c687a-istio-token\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-gxs26\" (UID: \"86c84864-127f-4507-8f60-2dbb684c687a\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-gxs26"
Oct 31 12:40:07.502219 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:07.501888    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86c84864-127f-4507-8f60-2dbb684c687a-istio-data\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-gxs26\" (UID: \"86c84864-127f-4507-8f60-2dbb684c687a\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-gxs26"
Oct 31 12:40:07.502219 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:07.501907    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86c84864-127f-4507-8f60-2dbb684c687a-istio-envoy\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-gxs26\" (UID: \"86c84864-127f-4507-8f60-2dbb684c687a\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-gxs26"
Oct 31 12:40:07.502219 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:07.501956    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86c84864-127f-4507-8f60-2dbb684c687a-credential-socket\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-gxs26\" (UID: \"86c84864-127f-4507-8f60-2dbb684c687a\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-gxs26"
Oct 31 12:40:07.502219 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:07.501992    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x42tb\" (UniqueName: \"kubernetes.io/projected/86c84864-127f-4507-8f60-2dbb684c687a-kube-api-access-x42tb\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-gxs26\" (UID: \"86c84864-127f-4507-8f60-2dbb684c687a\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-gxs26"
Oct 31 12:40:07.502219 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:07.502010    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86c84864-127f-4507-8f60-2dbb684c687a-workload-socket\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-gxs26\" (UID: \"86c84864-127f-4507-8f60-2dbb684c687a\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-gxs26"
Oct 31 12:40:07.585921 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:07.585828    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-l8n27"
Oct 31 12:40:07.603135 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:07.603106    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86c84864-127f-4507-8f60-2dbb684c687a-istiod-ca-cert\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-gxs26\" (UID: \"86c84864-127f-4507-8f60-2dbb684c687a\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-gxs26"
Oct 31 12:40:07.603252 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:07.603142    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86c84864-127f-4507-8f60-2dbb684c687a-istio-podinfo\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-gxs26\" (UID: \"86c84864-127f-4507-8f60-2dbb684c687a\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-gxs26"
Oct 31 12:40:07.603252 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:07.603196    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86c84864-127f-4507-8f60-2dbb684c687a-istio-token\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-gxs26\" (UID: \"86c84864-127f-4507-8f60-2dbb684c687a\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-gxs26"
Oct 31 12:40:07.603252 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:07.603214    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86c84864-127f-4507-8f60-2dbb684c687a-istio-data\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-gxs26\" (UID: \"86c84864-127f-4507-8f60-2dbb684c687a\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-gxs26"
Oct 31 12:40:07.603252 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:07.603232    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86c84864-127f-4507-8f60-2dbb684c687a-istio-envoy\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-gxs26\" (UID: \"86c84864-127f-4507-8f60-2dbb684c687a\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-gxs26"
Oct 31 12:40:07.603468 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:07.603295    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86c84864-127f-4507-8f60-2dbb684c687a-credential-socket\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-gxs26\" (UID: \"86c84864-127f-4507-8f60-2dbb684c687a\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-gxs26"
Oct 31 12:40:07.603468 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:07.603335    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x42tb\" (UniqueName: \"kubernetes.io/projected/86c84864-127f-4507-8f60-2dbb684c687a-kube-api-access-x42tb\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-gxs26\" (UID: \"86c84864-127f-4507-8f60-2dbb684c687a\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-gxs26"
Oct 31 12:40:07.603468 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:07.603361    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86c84864-127f-4507-8f60-2dbb684c687a-workload-socket\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-gxs26\" (UID: \"86c84864-127f-4507-8f60-2dbb684c687a\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-gxs26"
Oct 31 12:40:07.603468 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:07.603427    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86c84864-127f-4507-8f60-2dbb684c687a-workload-certs\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-gxs26\" (UID: \"86c84864-127f-4507-8f60-2dbb684c687a\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-gxs26"
Oct 31 12:40:07.603698 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:07.603623    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86c84864-127f-4507-8f60-2dbb684c687a-istio-data\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-gxs26\" (UID: \"86c84864-127f-4507-8f60-2dbb684c687a\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-gxs26"
Oct 31 12:40:07.603897 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:07.603871    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86c84864-127f-4507-8f60-2dbb684c687a-workload-certs\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-gxs26\" (UID: \"86c84864-127f-4507-8f60-2dbb684c687a\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-gxs26"
Oct 31 12:40:07.603971 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:07.603916    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86c84864-127f-4507-8f60-2dbb684c687a-workload-socket\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-gxs26\" (UID: \"86c84864-127f-4507-8f60-2dbb684c687a\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-gxs26"
Oct 31 12:40:07.603971 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:07.603916    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86c84864-127f-4507-8f60-2dbb684c687a-istiod-ca-cert\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-gxs26\" (UID: \"86c84864-127f-4507-8f60-2dbb684c687a\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-gxs26"
Oct 31 12:40:07.604149 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:07.604115    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86c84864-127f-4507-8f60-2dbb684c687a-credential-socket\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-gxs26\" (UID: \"86c84864-127f-4507-8f60-2dbb684c687a\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-gxs26"
Oct 31 12:40:07.605829 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:07.605801    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86c84864-127f-4507-8f60-2dbb684c687a-istio-envoy\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-gxs26\" (UID: \"86c84864-127f-4507-8f60-2dbb684c687a\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-gxs26"
Oct 31 12:40:07.605999 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:07.605981    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86c84864-127f-4507-8f60-2dbb684c687a-istio-podinfo\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-gxs26\" (UID: \"86c84864-127f-4507-8f60-2dbb684c687a\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-gxs26"
Oct 31 12:40:07.615679 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:07.615654    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86c84864-127f-4507-8f60-2dbb684c687a-istio-token\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-gxs26\" (UID: \"86c84864-127f-4507-8f60-2dbb684c687a\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-gxs26"
Oct 31 12:40:07.616288 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:07.616268    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x42tb\" (UniqueName: \"kubernetes.io/projected/86c84864-127f-4507-8f60-2dbb684c687a-kube-api-access-x42tb\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-gxs26\" (UID: \"86c84864-127f-4507-8f60-2dbb684c687a\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-gxs26"
Oct 31 12:40:07.722540 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:07.721702    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-gxs26"
Oct 31 12:40:07.759798 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:07.755444    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-2fn6t"]
Oct 31 12:40:07.765601 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:07.764118    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:40:07.765601 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:07.764209    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:40:07.765601 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:07.764252    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:40:07.827386 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:07.825712    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3eb4bba9-52x2t" event={"ID":"627354b5-bbf0-46b0-80f6-5a0ce90b377c","Type":"ContainerDied","Data":"81548d1ed81f619f4ebcb942b4ff5a7cc0d861dc0bf66727282d68779ecfb594"}
Oct 31 12:40:07.827386 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:07.825738    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3eb4bba9-52x2t"
Oct 31 12:40:07.827386 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:07.825754    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="81548d1ed81f619f4ebcb942b4ff5a7cc0d861dc0bf66727282d68779ecfb594"
Oct 31 12:40:07.828865 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:07.828793    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-2fn6t" event={"ID":"60d395e9-e4e9-4c68-ab10-9ef5371df1b1","Type":"ContainerStarted","Data":"ac559542ac0e20afb2af7ca9f91ee86a5d4ecab6810e5f14d5526e6f6ba0ab59"}
Oct 31 12:40:07.880617 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:07.880586    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-l8n27"]
Oct 31 12:40:07.880799 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:40:07.880742    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5c79f52_3049_4611_a935_ef2f4eb5e0bd.slice/crio-ac2088dd0e1a1315aa49042883f143c77917c1ecb4e308455521685e37af5a96 WatchSource:0}: Error finding container ac2088dd0e1a1315aa49042883f143c77917c1ecb4e308455521685e37af5a96: Status 404 returned error can't find the container with id ac2088dd0e1a1315aa49042883f143c77917c1ecb4e308455521685e37af5a96
Oct 31 12:40:07.884533 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:07.883709    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:40:07.884533 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:07.883787    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:40:07.884533 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:07.883831    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:40:08.032115 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:40:08.032059    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86c84864_127f_4507_8f60_2dbb684c687a.slice/crio-f5ad51ea2614a5cee4c36c4d3a879b5c62ac7cce423e7f83ae904f0d74e3d1ef WatchSource:0}: Error finding container f5ad51ea2614a5cee4c36c4d3a879b5c62ac7cce423e7f83ae904f0d74e3d1ef: Status 404 returned error can't find the container with id f5ad51ea2614a5cee4c36c4d3a879b5c62ac7cce423e7f83ae904f0d74e3d1ef
Oct 31 12:40:08.035162 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:08.035135    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-gxs26"]
Oct 31 12:40:08.035262 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:08.035222    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:40:08.035341 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:08.035323    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:40:08.035391 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:08.035371    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:40:08.896519 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:08.893755    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-l8n27" event={"ID":"c5c79f52-3049-4611-a935-ef2f4eb5e0bd","Type":"ContainerStarted","Data":"996dfd8501bc25b0bbf2d5acaacca3f2bb6ca231454aee25616f89192cb48a5d"}
Oct 31 12:40:08.896519 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:08.893824    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-l8n27" event={"ID":"c5c79f52-3049-4611-a935-ef2f4eb5e0bd","Type":"ContainerStarted","Data":"ac2088dd0e1a1315aa49042883f143c77917c1ecb4e308455521685e37af5a96"}
Oct 31 12:40:08.933520 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:08.929820    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-gxs26" event={"ID":"86c84864-127f-4507-8f60-2dbb684c687a","Type":"ContainerStarted","Data":"b0e2771f19a497662c194c4899a869898da0bab9a01207f1478d9e573f31a211"}
Oct 31 12:40:08.933520 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:08.929905    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-gxs26" event={"ID":"86c84864-127f-4507-8f60-2dbb684c687a","Type":"ContainerStarted","Data":"f5ad51ea2614a5cee4c36c4d3a879b5c62ac7cce423e7f83ae904f0d74e3d1ef"}
Oct 31 12:40:08.934513 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:08.933810    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-l8n27" podStartSLOduration=1.93379579 podStartE2EDuration="1.93379579s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:40:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:40:08.931237298 +0000 UTC m=+772.034305012" watchObservedRunningTime="2026-10-31 12:40:08.93379579 +0000 UTC m=+772.036863503"
Oct 31 12:40:08.951729 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:08.951695    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-2fn6t" event={"ID":"60d395e9-e4e9-4c68-ab10-9ef5371df1b1","Type":"ContainerStarted","Data":"6576cc6e6a88037c75e3d89b906c3e21c090af7d8a2f8e801ef2a27753a51791"}
Oct 31 12:40:09.097863 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:09.097786    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-gxs26" podStartSLOduration=2.097765342 podStartE2EDuration="2.097765342s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:40:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:40:08.99399584 +0000 UTC m=+772.097063566" watchObservedRunningTime="2026-10-31 12:40:09.097765342 +0000 UTC m=+772.200833056"
Oct 31 12:40:09.373572 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:09.371256    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-2fn6t" podStartSLOduration=2.371234193 podStartE2EDuration="2.371234193s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:40:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:40:09.129273153 +0000 UTC m=+772.232340892" watchObservedRunningTime="2026-10-31 12:40:09.371234193 +0000 UTC m=+772.474301908"
Oct 31 12:40:09.448524 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:09.447806    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-2fn6t"
Oct 31 12:40:09.452525 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:09.450041    5060 patch_prober.go:29] interesting pod/e2e-worker-w3-b218d3-openshift-default-67596dd89f-2fn6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body=
Oct 31 12:40:09.452525 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:09.450102    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-2fn6t" podUID="60d395e9-e4e9-4c68-ab10-9ef5371df1b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused"
Oct 31 12:40:09.596523 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:09.596036    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-l8n27"
Oct 31 12:40:09.600560 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:09.597984    5060 patch_prober.go:29] interesting pod/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-l8n27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body=
Oct 31 12:40:09.600560 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:09.598038    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-l8n27" podUID="c5c79f52-3049-4611-a935-ef2f4eb5e0bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused"
Oct 31 12:40:09.726639 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:09.724232    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-gxs26"
Oct 31 12:40:09.726639 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:09.724399    5060 patch_prober.go:29] interesting pod/e2e-worker-w1-85ce08-openshift-default-77d6458845-gxs26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body=
Oct 31 12:40:09.726639 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:09.724444    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-gxs26" podUID="86c84864-127f-4507-8f60-2dbb684c687a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused"
Oct 31 12:40:10.263516 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:10.260828    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x2qzv" podUID="c247ed2a-aa90-424b-9b7e-4f2936d67e3f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:40:10.375517 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:10.368581    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/simulator-w4-c67de2-kserve-868679cf9b-gqjbx" podUID="e104d7da-1fb2-4c99-a480-b4ae84c4f273" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded"
Oct 31 12:40:10.465002 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:10.461929    5060 patch_prober.go:29] interesting pod/e2e-worker-w3-b218d3-openshift-default-67596dd89f-2fn6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body=
Oct 31 12:40:10.465002 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:10.461998    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-2fn6t" podUID="60d395e9-e4e9-4c68-ab10-9ef5371df1b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused"
Oct 31 12:40:10.590337 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:10.589887    5060 patch_prober.go:29] interesting pod/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-l8n27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body=
Oct 31 12:40:10.590337 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:10.589948    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-l8n27" podUID="c5c79f52-3049-4611-a935-ef2f4eb5e0bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused"
Oct 31 12:40:10.726517 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:10.711646    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/premium-w2-e6c79d-kserve-6468bf46d4-h66f8" podUID="cd6a266f-8d12-4089-83ee-c75c716bf26c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:40:10.728642 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:10.728198    5060 patch_prober.go:29] interesting pod/e2e-worker-w1-85ce08-openshift-default-77d6458845-gxs26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body=
Oct 31 12:40:10.728642 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:10.728254    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-gxs26" podUID="86c84864-127f-4507-8f60-2dbb684c687a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused"
Oct 31 12:40:11.261664 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:11.261608    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x2qzv" podUID="c247ed2a-aa90-424b-9b7e-4f2936d67e3f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:40:11.274521 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:11.270780    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x2qzv"
Oct 31 12:40:11.433137 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:11.399528    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/simulator-w4-c67de2-kserve-868679cf9b-gqjbx" podUID="e104d7da-1fb2-4c99-a480-b4ae84c4f273" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:40:11.433137 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:11.402793    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c67de2/simulator-w4-c67de2-kserve-868679cf9b-gqjbx"
Oct 31 12:40:11.457280 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:11.457242    5060 patch_prober.go:29] interesting pod/e2e-worker-w3-b218d3-openshift-default-67596dd89f-2fn6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body=
Oct 31 12:40:11.457541 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:11.457510    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-2fn6t" podUID="60d395e9-e4e9-4c68-ab10-9ef5371df1b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused"
Oct 31 12:40:11.614666 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:11.614630    5060 patch_prober.go:29] interesting pod/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-l8n27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body=
Oct 31 12:40:11.631517 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:11.621795    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-l8n27" podUID="c5c79f52-3049-4611-a935-ef2f4eb5e0bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused"
Oct 31 12:40:11.783845 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:11.767595    5060 patch_prober.go:29] interesting pod/e2e-worker-w1-85ce08-openshift-default-77d6458845-gxs26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body=
Oct 31 12:40:11.783845 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:11.767657    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-gxs26" podUID="86c84864-127f-4507-8f60-2dbb684c687a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused"
Oct 31 12:40:11.786435 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:11.785018    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/premium-w2-e6c79d-kserve-6468bf46d4-h66f8" podUID="cd6a266f-8d12-4089-83ee-c75c716bf26c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Oct 31 12:40:11.789666 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:11.789632    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e6c79d/premium-w2-e6c79d-kserve-6468bf46d4-h66f8"
Oct 31 12:40:12.262697 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:12.262590    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x2qzv" podUID="c247ed2a-aa90-424b-9b7e-4f2936d67e3f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:40:12.396903 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:12.396849    5060 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x2qzv" podUID="c247ed2a-aa90-424b-9b7e-4f2936d67e3f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:40:12.401578 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:12.400565    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/simulator-w4-c67de2-kserve-868679cf9b-gqjbx" podUID="e104d7da-1fb2-4c99-a480-b4ae84c4f273" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:40:12.453287 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:12.453254    5060 patch_prober.go:29] interesting pod/e2e-worker-w3-b218d3-openshift-default-67596dd89f-2fn6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body=
Oct 31 12:40:12.453590 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:12.453560    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-2fn6t" podUID="60d395e9-e4e9-4c68-ab10-9ef5371df1b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused"
Oct 31 12:40:12.602516 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:12.588749    5060 patch_prober.go:29] interesting pod/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-l8n27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body=
Oct 31 12:40:12.602516 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:12.588829    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-l8n27" podUID="c5c79f52-3049-4611-a935-ef2f4eb5e0bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused"
Oct 31 12:40:12.728520 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:12.727895    5060 patch_prober.go:29] interesting pod/e2e-worker-w1-85ce08-openshift-default-77d6458845-gxs26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body=
Oct 31 12:40:12.728520 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:12.727951    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-gxs26" podUID="86c84864-127f-4507-8f60-2dbb684c687a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused"
Oct 31 12:40:12.808520 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:12.806667    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/premium-w2-e6c79d-kserve-6468bf46d4-h66f8" podUID="cd6a266f-8d12-4089-83ee-c75c716bf26c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Oct 31 12:40:13.272523 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:13.268756    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x2qzv" podUID="c247ed2a-aa90-424b-9b7e-4f2936d67e3f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded"
Oct 31 12:40:13.413866 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:13.404703    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/simulator-w4-c67de2-kserve-868679cf9b-gqjbx" podUID="e104d7da-1fb2-4c99-a480-b4ae84c4f273" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:40:13.449866 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:13.449635    5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-645cf464d7-7psf5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 12:40:13.449866 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:13.449760    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-7psf5" podUID="37aec2da-59cb-41ed-9e6b-a443f6ff84c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:40:13.468562 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:13.454771    5060 patch_prober.go:29] interesting pod/e2e-worker-w3-b218d3-openshift-default-67596dd89f-2fn6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body=
Oct 31 12:40:13.468562 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:13.454823    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-2fn6t" podUID="60d395e9-e4e9-4c68-ab10-9ef5371df1b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused"
Oct 31 12:40:13.598581 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:13.590318    5060 patch_prober.go:29] interesting pod/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-l8n27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body=
Oct 31 12:40:13.598581 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:13.590376    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-l8n27" podUID="c5c79f52-3049-4611-a935-ef2f4eb5e0bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused"
Oct 31 12:40:13.864571 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:13.766937    5060 patch_prober.go:29] interesting pod/e2e-worker-w1-85ce08-openshift-default-77d6458845-gxs26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body=
Oct 31 12:40:13.864571 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:13.767001    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-gxs26" podUID="86c84864-127f-4507-8f60-2dbb684c687a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused"
Oct 31 12:40:13.864571 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:13.835631    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/premium-w2-e6c79d-kserve-6468bf46d4-h66f8" podUID="cd6a266f-8d12-4089-83ee-c75c716bf26c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:40:14.144577 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:14.142752    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-466d864d-94qhv"]
Oct 31 12:40:14.254521 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:14.240046    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-466d864d-94qhv"]
Oct 31 12:40:14.254521 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:14.240181    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-466d864d-94qhv"
Oct 31 12:40:14.254521 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:14.247801    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 12:40:14.254521 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:14.248454    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 12:40:14.254521 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:14.248638    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7wsmb\""
Oct 31 12:40:14.254521 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:14.248788    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 12:40:14.293512 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:14.268940    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x2qzv" podUID="c247ed2a-aa90-424b-9b7e-4f2936d67e3f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:40:14.327838 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:14.327808    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c67de2/simulator-w4-c67de2-kserve-868679cf9b-gqjbx"
Oct 31 12:40:14.351141 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:14.351102    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9px66\" (UniqueName: \"kubernetes.io/projected/b50eace0-22ba-490b-9344-0722c9b6cfa0-kube-api-access-9px66\") pod \"maas-api-revoke-sub-466d864d-94qhv\" (UID: \"b50eace0-22ba-490b-9344-0722c9b6cfa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-466d864d-94qhv"
Oct 31 12:40:14.351406 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:14.351387    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b50eace0-22ba-490b-9344-0722c9b6cfa0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-466d864d-94qhv\" (UID: \"b50eace0-22ba-490b-9344-0722c9b6cfa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-466d864d-94qhv"
Oct 31 12:40:14.351561 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:14.351544    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b50eace0-22ba-490b-9344-0722c9b6cfa0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-466d864d-94qhv\" (UID: \"b50eace0-22ba-490b-9344-0722c9b6cfa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-466d864d-94qhv"
Oct 31 12:40:14.560573 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:14.478898    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9px66\" (UniqueName: \"kubernetes.io/projected/b50eace0-22ba-490b-9344-0722c9b6cfa0-kube-api-access-9px66\") pod \"maas-api-revoke-sub-466d864d-94qhv\" (UID: \"b50eace0-22ba-490b-9344-0722c9b6cfa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-466d864d-94qhv"
Oct 31 12:40:14.560573 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:14.478996    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b50eace0-22ba-490b-9344-0722c9b6cfa0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-466d864d-94qhv\" (UID: \"b50eace0-22ba-490b-9344-0722c9b6cfa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-466d864d-94qhv"
Oct 31 12:40:14.560573 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:14.479048    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b50eace0-22ba-490b-9344-0722c9b6cfa0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-466d864d-94qhv\" (UID: \"b50eace0-22ba-490b-9344-0722c9b6cfa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-466d864d-94qhv"
Oct 31 12:40:14.560573 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:14.519553    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9px66\" (UniqueName: \"kubernetes.io/projected/b50eace0-22ba-490b-9344-0722c9b6cfa0-kube-api-access-9px66\") pod \"maas-api-revoke-sub-466d864d-94qhv\" (UID: \"b50eace0-22ba-490b-9344-0722c9b6cfa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-466d864d-94qhv"
Oct 31 12:40:14.560573 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:14.520102    5060 patch_prober.go:29] interesting pod/e2e-worker-w3-b218d3-openshift-default-67596dd89f-2fn6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body=
Oct 31 12:40:14.560573 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:14.520153    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-2fn6t" podUID="60d395e9-e4e9-4c68-ab10-9ef5371df1b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused"
Oct 31 12:40:14.585946 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:14.585908    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b50eace0-22ba-490b-9344-0722c9b6cfa0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-466d864d-94qhv\" (UID: \"b50eace0-22ba-490b-9344-0722c9b6cfa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-466d864d-94qhv"
Oct 31 12:40:14.685631 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:14.643689    5060 patch_prober.go:29] interesting pod/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-l8n27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body=
Oct 31 12:40:14.693599 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:14.693558    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-l8n27" podUID="c5c79f52-3049-4611-a935-ef2f4eb5e0bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused"
Oct 31 12:40:14.718938 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:14.686280    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b50eace0-22ba-490b-9344-0722c9b6cfa0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-466d864d-94qhv\" (UID: \"b50eace0-22ba-490b-9344-0722c9b6cfa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-466d864d-94qhv"
Oct 31 12:40:14.808518 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:14.805479    5060 patch_prober.go:29] interesting pod/e2e-worker-w1-85ce08-openshift-default-77d6458845-gxs26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body=
Oct 31 12:40:14.808518 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:14.805571    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-gxs26" podUID="86c84864-127f-4507-8f60-2dbb684c687a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused"
Oct 31 12:40:14.872588 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:14.834982    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-466d864d-94qhv"
Oct 31 12:40:14.909521 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:14.884950    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/premium-w2-e6c79d-kserve-6468bf46d4-h66f8" podUID="cd6a266f-8d12-4089-83ee-c75c716bf26c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Oct 31 12:40:15.053515 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:15.035710    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-rz7mt"]
Oct 31 12:40:15.128578 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:15.068909    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-rz7mt"
Oct 31 12:40:15.246520 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:15.245849    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/978e5cf9-a89e-4fc9-b4ba-fe0ee2de401b-istio-token\") pod \"maas-default-gateway-openshift-default-645cf464d7-rz7mt\" (UID: \"978e5cf9-a89e-4fc9-b4ba-fe0ee2de401b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-rz7mt"
Oct 31 12:40:15.246520 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:15.245919    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/978e5cf9-a89e-4fc9-b4ba-fe0ee2de401b-istio-envoy\") pod \"maas-default-gateway-openshift-default-645cf464d7-rz7mt\" (UID: \"978e5cf9-a89e-4fc9-b4ba-fe0ee2de401b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-rz7mt"
Oct 31 12:40:15.246520 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:15.245950    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/978e5cf9-a89e-4fc9-b4ba-fe0ee2de401b-workload-certs\") pod \"maas-default-gateway-openshift-default-645cf464d7-rz7mt\" (UID: \"978e5cf9-a89e-4fc9-b4ba-fe0ee2de401b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-rz7mt"
Oct 31 12:40:15.246520 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:15.245981    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/978e5cf9-a89e-4fc9-b4ba-fe0ee2de401b-credential-socket\") pod \"maas-default-gateway-openshift-default-645cf464d7-rz7mt\" (UID: \"978e5cf9-a89e-4fc9-b4ba-fe0ee2de401b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-rz7mt"
Oct 31 12:40:15.246520 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:15.246031    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/978e5cf9-a89e-4fc9-b4ba-fe0ee2de401b-istio-data\") pod \"maas-default-gateway-openshift-default-645cf464d7-rz7mt\" (UID: \"978e5cf9-a89e-4fc9-b4ba-fe0ee2de401b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-rz7mt"
Oct 31 12:40:15.246520 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:15.246142    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/978e5cf9-a89e-4fc9-b4ba-fe0ee2de401b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-645cf464d7-rz7mt\" (UID: \"978e5cf9-a89e-4fc9-b4ba-fe0ee2de401b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-rz7mt"
Oct 31 12:40:15.246520 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:15.246190    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/978e5cf9-a89e-4fc9-b4ba-fe0ee2de401b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-645cf464d7-rz7mt\" (UID: \"978e5cf9-a89e-4fc9-b4ba-fe0ee2de401b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-rz7mt"
Oct 31 12:40:15.246520 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:15.246247    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/978e5cf9-a89e-4fc9-b4ba-fe0ee2de401b-workload-socket\") pod \"maas-default-gateway-openshift-default-645cf464d7-rz7mt\" (UID: \"978e5cf9-a89e-4fc9-b4ba-fe0ee2de401b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-rz7mt"
Oct 31 12:40:15.246520 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:15.246339    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztcr5\" (UniqueName: \"kubernetes.io/projected/978e5cf9-a89e-4fc9-b4ba-fe0ee2de401b-kube-api-access-ztcr5\") pod \"maas-default-gateway-openshift-default-645cf464d7-rz7mt\" (UID: \"978e5cf9-a89e-4fc9-b4ba-fe0ee2de401b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-rz7mt"
Oct 31 12:40:15.263526 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:15.260772    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-rz7mt"]
Oct 31 12:40:15.321524 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:15.319933    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x2qzv" podUID="c247ed2a-aa90-424b-9b7e-4f2936d67e3f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:40:15.361548 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:15.360903    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/978e5cf9-a89e-4fc9-b4ba-fe0ee2de401b-istio-data\") pod \"maas-default-gateway-openshift-default-645cf464d7-rz7mt\" (UID: \"978e5cf9-a89e-4fc9-b4ba-fe0ee2de401b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-rz7mt"
Oct 31 12:40:15.361548 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:15.361031    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/978e5cf9-a89e-4fc9-b4ba-fe0ee2de401b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-645cf464d7-rz7mt\" (UID: \"978e5cf9-a89e-4fc9-b4ba-fe0ee2de401b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-rz7mt"
Oct 31 12:40:15.361548 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:15.361081    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/978e5cf9-a89e-4fc9-b4ba-fe0ee2de401b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-645cf464d7-rz7mt\" (UID: \"978e5cf9-a89e-4fc9-b4ba-fe0ee2de401b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-rz7mt"
Oct 31 12:40:15.361548 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:15.361137    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/978e5cf9-a89e-4fc9-b4ba-fe0ee2de401b-workload-socket\") pod \"maas-default-gateway-openshift-default-645cf464d7-rz7mt\" (UID: \"978e5cf9-a89e-4fc9-b4ba-fe0ee2de401b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-rz7mt"
Oct 31 12:40:15.361548 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:15.361227    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztcr5\" (UniqueName: \"kubernetes.io/projected/978e5cf9-a89e-4fc9-b4ba-fe0ee2de401b-kube-api-access-ztcr5\") pod \"maas-default-gateway-openshift-default-645cf464d7-rz7mt\" (UID: \"978e5cf9-a89e-4fc9-b4ba-fe0ee2de401b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-rz7mt"
Oct 31 12:40:15.361548 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:15.361258    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/978e5cf9-a89e-4fc9-b4ba-fe0ee2de401b-istio-token\") pod \"maas-default-gateway-openshift-default-645cf464d7-rz7mt\" (UID: \"978e5cf9-a89e-4fc9-b4ba-fe0ee2de401b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-rz7mt"
Oct 31 12:40:15.361548 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:15.361312    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/978e5cf9-a89e-4fc9-b4ba-fe0ee2de401b-istio-envoy\") pod \"maas-default-gateway-openshift-default-645cf464d7-rz7mt\" (UID: \"978e5cf9-a89e-4fc9-b4ba-fe0ee2de401b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-rz7mt"
Oct 31 12:40:15.361548 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:15.361340    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/978e5cf9-a89e-4fc9-b4ba-fe0ee2de401b-workload-certs\") pod \"maas-default-gateway-openshift-default-645cf464d7-rz7mt\" (UID: \"978e5cf9-a89e-4fc9-b4ba-fe0ee2de401b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-rz7mt"
Oct 31 12:40:15.361548 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:15.361374    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/978e5cf9-a89e-4fc9-b4ba-fe0ee2de401b-credential-socket\") pod \"maas-default-gateway-openshift-default-645cf464d7-rz7mt\" (UID: \"978e5cf9-a89e-4fc9-b4ba-fe0ee2de401b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-rz7mt"
Oct 31 12:40:15.372777 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:15.371063    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/978e5cf9-a89e-4fc9-b4ba-fe0ee2de401b-credential-socket\") pod \"maas-default-gateway-openshift-default-645cf464d7-rz7mt\" (UID: \"978e5cf9-a89e-4fc9-b4ba-fe0ee2de401b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-rz7mt"
Oct 31 12:40:15.372777 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:15.372321    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/978e5cf9-a89e-4fc9-b4ba-fe0ee2de401b-istio-data\") pod \"maas-default-gateway-openshift-default-645cf464d7-rz7mt\" (UID: \"978e5cf9-a89e-4fc9-b4ba-fe0ee2de401b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-rz7mt"
Oct 31 12:40:15.381548 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:15.380917    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/978e5cf9-a89e-4fc9-b4ba-fe0ee2de401b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-645cf464d7-rz7mt\" (UID: \"978e5cf9-a89e-4fc9-b4ba-fe0ee2de401b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-rz7mt"
Oct 31 12:40:15.381548 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:15.381186    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/978e5cf9-a89e-4fc9-b4ba-fe0ee2de401b-workload-socket\") pod \"maas-default-gateway-openshift-default-645cf464d7-rz7mt\" (UID: \"978e5cf9-a89e-4fc9-b4ba-fe0ee2de401b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-rz7mt"
Oct 31 12:40:15.382317 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:15.382037    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/978e5cf9-a89e-4fc9-b4ba-fe0ee2de401b-workload-certs\") pod \"maas-default-gateway-openshift-default-645cf464d7-rz7mt\" (UID: \"978e5cf9-a89e-4fc9-b4ba-fe0ee2de401b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-rz7mt"
Oct 31 12:40:15.397840 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:15.397759    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/978e5cf9-a89e-4fc9-b4ba-fe0ee2de401b-istio-envoy\") pod \"maas-default-gateway-openshift-default-645cf464d7-rz7mt\" (UID: \"978e5cf9-a89e-4fc9-b4ba-fe0ee2de401b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-rz7mt"
Oct 31 12:40:15.429530 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:15.416149    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/978e5cf9-a89e-4fc9-b4ba-fe0ee2de401b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-645cf464d7-rz7mt\" (UID: \"978e5cf9-a89e-4fc9-b4ba-fe0ee2de401b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-rz7mt"
Oct 31 12:40:15.448540 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:15.433056    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztcr5\" (UniqueName: \"kubernetes.io/projected/978e5cf9-a89e-4fc9-b4ba-fe0ee2de401b-kube-api-access-ztcr5\") pod \"maas-default-gateway-openshift-default-645cf464d7-rz7mt\" (UID: \"978e5cf9-a89e-4fc9-b4ba-fe0ee2de401b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-rz7mt"
Oct 31 12:40:15.456541 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:15.448835    5060 patch_prober.go:29] interesting pod/e2e-worker-w3-b218d3-openshift-default-67596dd89f-2fn6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body=
Oct 31 12:40:15.456541 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:15.448895    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-2fn6t" podUID="60d395e9-e4e9-4c68-ab10-9ef5371df1b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused"
Oct 31 12:40:15.488608 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:15.485160    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/978e5cf9-a89e-4fc9-b4ba-fe0ee2de401b-istio-token\") pod \"maas-default-gateway-openshift-default-645cf464d7-rz7mt\" (UID: \"978e5cf9-a89e-4fc9-b4ba-fe0ee2de401b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-rz7mt"
Oct 31 12:40:15.552578 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:15.550723    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-rz7mt"
Oct 31 12:40:15.593440 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:15.591881    5060 patch_prober.go:29] interesting pod/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-l8n27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body=
Oct 31 12:40:15.593440 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:15.591943    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-l8n27" podUID="c5c79f52-3049-4611-a935-ef2f4eb5e0bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused"
Oct 31 12:40:15.731010 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:15.730973    5060 patch_prober.go:29] interesting pod/e2e-worker-w1-85ce08-openshift-default-77d6458845-gxs26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body=
Oct 31 12:40:15.731314 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:15.731269    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-gxs26" podUID="86c84864-127f-4507-8f60-2dbb684c687a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused"
Oct 31 12:40:15.935102 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:15.898613    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/premium-w2-e6c79d-kserve-6468bf46d4-h66f8" podUID="cd6a266f-8d12-4089-83ee-c75c716bf26c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:40:16.376521 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:16.326745    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x2qzv" podUID="c247ed2a-aa90-424b-9b7e-4f2936d67e3f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:40:16.436055 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:16.432695    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/simulator-w4-c67de2-kserve-868679cf9b-gqjbx" podUID="e104d7da-1fb2-4c99-a480-b4ae84c4f273" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:40:16.566112 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:16.560254    5060 patch_prober.go:29] interesting pod/e2e-worker-w3-b218d3-openshift-default-67596dd89f-2fn6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body=
Oct 31 12:40:16.566112 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:16.560321    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-2fn6t" podUID="60d395e9-e4e9-4c68-ab10-9ef5371df1b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused"
Oct 31 12:40:16.587517 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:16.584042    5060 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-c67de2/simulator-w4-c67de2-kserve-868679cf9b-gqjbx" podUID="e104d7da-1fb2-4c99-a480-b4ae84c4f273" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:40:16.605768 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:16.605643    5060 patch_prober.go:29] interesting pod/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-l8n27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body=
Oct 31 12:40:16.605768 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:16.605706    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-l8n27" podUID="c5c79f52-3049-4611-a935-ef2f4eb5e0bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused"
Oct 31 12:40:16.739947 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:16.739533    5060 patch_prober.go:29] interesting pod/e2e-worker-w1-85ce08-openshift-default-77d6458845-gxs26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body=
Oct 31 12:40:16.739947 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:16.739588    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-gxs26" podUID="86c84864-127f-4507-8f60-2dbb684c687a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused"
Oct 31 12:40:16.859554 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:16.855852    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-466d864d-94qhv"]
Oct 31 12:40:16.903991 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:16.902277    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/premium-w2-e6c79d-kserve-6468bf46d4-h66f8" podUID="cd6a266f-8d12-4089-83ee-c75c716bf26c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:40:17.090709 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:17.070924    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:40:17.090709 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:17.071013    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:40:17.090709 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:17.071086    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:40:17.496937 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:17.489082    5060 patch_prober.go:29] interesting pod/e2e-worker-w3-b218d3-openshift-default-67596dd89f-2fn6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body=
Oct 31 12:40:17.496937 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:17.489143    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-2fn6t" podUID="60d395e9-e4e9-4c68-ab10-9ef5371df1b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused"
Oct 31 12:40:17.707650 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:17.707601    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x2qzv" podUID="c247ed2a-aa90-424b-9b7e-4f2936d67e3f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded"
Oct 31 12:40:17.747678 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:17.747588    5060 patch_prober.go:29] interesting pod/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-l8n27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body=
Oct 31 12:40:17.747678 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:17.747652    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-l8n27" podUID="c5c79f52-3049-4611-a935-ef2f4eb5e0bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused"
Oct 31 12:40:17.750266 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:17.750018    5060 patch_prober.go:29] interesting pod/e2e-worker-w1-85ce08-openshift-default-77d6458845-gxs26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body=
Oct 31 12:40:17.774520 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:17.761012    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-gxs26" podUID="86c84864-127f-4507-8f60-2dbb684c687a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused"
Oct 31 12:40:17.998113 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:17.998026    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-rz7mt"]
Oct 31 12:40:17.998411 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:17.998392    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e6c79d/premium-w2-e6c79d-kserve-6468bf46d4-h66f8"
Oct 31 12:40:18.040608 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:18.040551    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-466d864d-94qhv" event={"ID":"b50eace0-22ba-490b-9344-0722c9b6cfa0","Type":"ContainerStarted","Data":"d911a70041d80a96eacd45557bec0953e2f79cc73c5666004d53975c9672620e"}
Oct 31 12:40:18.068917 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:18.053649    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-rz7mt" event={"ID":"978e5cf9-a89e-4fc9-b4ba-fe0ee2de401b","Type":"ContainerStarted","Data":"c8e0af7de7ad8e8c5a1406771e8b7159a4153b876fab1058d2cfd733e1a7e4f9"}
Oct 31 12:40:18.149542 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:18.148780    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x2qzv"
Oct 31 12:40:18.440518 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:18.440230    5060 patch_prober.go:29] interesting pod/e2e-worker-w3-b218d3-openshift-default-67596dd89f-2fn6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body=
Oct 31 12:40:18.440518 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:18.440303    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-2fn6t" podUID="60d395e9-e4e9-4c68-ab10-9ef5371df1b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused"
Oct 31 12:40:18.587450 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:18.586886    5060 patch_prober.go:29] interesting pod/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-l8n27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body=
Oct 31 12:40:18.587450 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:18.586956    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-l8n27" podUID="c5c79f52-3049-4611-a935-ef2f4eb5e0bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused"
Oct 31 12:40:18.723326 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:18.723277    5060 patch_prober.go:29] interesting pod/e2e-worker-w1-85ce08-openshift-default-77d6458845-gxs26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body=
Oct 31 12:40:18.723848 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:18.723345    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-gxs26" podUID="86c84864-127f-4507-8f60-2dbb684c687a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused"
Oct 31 12:40:19.123517 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:19.122614    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-rz7mt" event={"ID":"978e5cf9-a89e-4fc9-b4ba-fe0ee2de401b","Type":"ContainerStarted","Data":"7b6c99a27669b6211e6efa26bf1f2e34370be0f8e174a85a958bd8e42663f1e5"}
Oct 31 12:40:19.127049 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:19.126726    5060 generic.go:356] "Generic (PLEG): container finished" podID="b50eace0-22ba-490b-9344-0722c9b6cfa0" containerID="82822d00310366795621427ef47d398710cb2fbd3f89567ac4ae2abd0940060e" exitCode=0
Oct 31 12:40:19.127049 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:19.126805    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-466d864d-94qhv" event={"ID":"b50eace0-22ba-490b-9344-0722c9b6cfa0","Type":"ContainerDied","Data":"82822d00310366795621427ef47d398710cb2fbd3f89567ac4ae2abd0940060e"}
Oct 31 12:40:19.190516 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:19.187115    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-rz7mt" podStartSLOduration=5.187090985 podStartE2EDuration="5.187090985s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:40:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:40:19.153665775 +0000 UTC m=+782.256733479" watchObservedRunningTime="2026-10-31 12:40:19.187090985 +0000 UTC m=+782.290158698"
Oct 31 12:40:19.553847 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:19.553726    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-rz7mt"
Oct 31 12:40:19.554374 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:19.554339    5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-645cf464d7-rz7mt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Oct 31 12:40:19.554558 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:19.554414    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-rz7mt" podUID="978e5cf9-a89e-4fc9-b4ba-fe0ee2de401b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Oct 31 12:40:19.862569 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:19.861889    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-l8n27"
Oct 31 12:40:19.866513 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:19.863206    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-l8n27"
Oct 31 12:40:19.866513 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:19.866463    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-l8n27"
Oct 31 12:40:20.452130 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:20.452033    5060 patch_prober.go:29] interesting pod/e2e-worker-w3-b218d3-openshift-default-67596dd89f-2fn6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 12:40:20.452368 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:20.452341    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-2fn6t" podUID="60d395e9-e4e9-4c68-ab10-9ef5371df1b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:40:20.563807 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:20.562812    5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-645cf464d7-rz7mt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Oct 31 12:40:20.563807 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:20.562879    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-rz7mt" podUID="978e5cf9-a89e-4fc9-b4ba-fe0ee2de401b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Oct 31 12:40:20.835618 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:20.831734    5060 patch_prober.go:29] interesting pod/e2e-worker-w1-85ce08-openshift-default-77d6458845-gxs26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 12:40:20.835618 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:20.831804    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-gxs26" podUID="86c84864-127f-4507-8f60-2dbb684c687a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:40:20.986458 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:20.981799    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-gxs26"
Oct 31 12:40:20.986458 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:20.983474    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-gxs26"
Oct 31 12:40:21.032029 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:21.031596    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-gxs26"
Oct 31 12:40:21.059510 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:21.059123    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-466d864d-94qhv"
Oct 31 12:40:21.094364 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:21.094149    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b50eace0-22ba-490b-9344-0722c9b6cfa0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b50eace0-22ba-490b-9344-0722c9b6cfa0-maas-api-xks-ca\") pod \"b50eace0-22ba-490b-9344-0722c9b6cfa0\" (UID: \"b50eace0-22ba-490b-9344-0722c9b6cfa0\") "
Oct 31 12:40:21.094998 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:21.094935    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b50eace0-22ba-490b-9344-0722c9b6cfa0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b50eace0-22ba-490b-9344-0722c9b6cfa0-maas-api-openshift-ca\") pod \"b50eace0-22ba-490b-9344-0722c9b6cfa0\" (UID: \"b50eace0-22ba-490b-9344-0722c9b6cfa0\") "
Oct 31 12:40:21.126592 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:21.101205    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b50eace0-22ba-490b-9344-0722c9b6cfa0-kube-api-access-9px66\" (UniqueName: \"kubernetes.io/projected/b50eace0-22ba-490b-9344-0722c9b6cfa0-kube-api-access-9px66\") pod \"b50eace0-22ba-490b-9344-0722c9b6cfa0\" (UID: \"b50eace0-22ba-490b-9344-0722c9b6cfa0\") "
Oct 31 12:40:21.126592 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:21.095609    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b50eace0-22ba-490b-9344-0722c9b6cfa0-maas-api-openshift-ca" pod "b50eace0-22ba-490b-9344-0722c9b6cfa0" (UID: "b50eace0-22ba-490b-9344-0722c9b6cfa0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:40:21.126592 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:21.101072    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b50eace0-22ba-490b-9344-0722c9b6cfa0-maas-api-xks-ca" pod "b50eace0-22ba-490b-9344-0722c9b6cfa0" (UID: "b50eace0-22ba-490b-9344-0722c9b6cfa0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:40:21.126592 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:21.111155    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b50eace0-22ba-490b-9344-0722c9b6cfa0-kube-api-access-9px66" pod "b50eace0-22ba-490b-9344-0722c9b6cfa0" (UID: "b50eace0-22ba-490b-9344-0722c9b6cfa0"). InnerVolumeSpecName "kube-api-access-9px66". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:40:21.234130 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:21.233965    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b50eace0-22ba-490b-9344-0722c9b6cfa0-maas-api-xks-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:40:21.234130 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:21.234004    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b50eace0-22ba-490b-9344-0722c9b6cfa0-maas-api-openshift-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:40:21.234130 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:21.234021    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9px66\" (UniqueName: \"kubernetes.io/projected/b50eace0-22ba-490b-9344-0722c9b6cfa0-kube-api-access-9px66\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:40:21.279515 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:21.278736    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-466d864d-94qhv"
Oct 31 12:40:21.305869 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:21.305808    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-466d864d-94qhv" event={"ID":"b50eace0-22ba-490b-9344-0722c9b6cfa0","Type":"ContainerDied","Data":"d911a70041d80a96eacd45557bec0953e2f79cc73c5666004d53975c9672620e"}
Oct 31 12:40:21.306133 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:21.306097    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d911a70041d80a96eacd45557bec0953e2f79cc73c5666004d53975c9672620e"
Oct 31 12:40:21.497571 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:21.494591    5060 patch_prober.go:29] interesting pod/e2e-worker-w3-b218d3-openshift-default-67596dd89f-2fn6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 12:40:21.497571 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:21.494664    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-2fn6t" podUID="60d395e9-e4e9-4c68-ab10-9ef5371df1b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:40:21.552178 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:21.552136    5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-645cf464d7-rz7mt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Oct 31 12:40:21.552435 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:21.552409    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-rz7mt" podUID="978e5cf9-a89e-4fc9-b4ba-fe0ee2de401b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Oct 31 12:40:21.926239 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:21.926202    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3e89004-fbpg7"]
Oct 31 12:40:21.926994 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:21.926969    5060 cpu_manager.go:414] "containerMap: removing container" podUID="b50eace0-22ba-490b-9344-0722c9b6cfa0" containerName="revoke-keys"
Oct 31 12:40:21.926994 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:21.926994    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b50eace0-22ba-490b-9344-0722c9b6cfa0" containerName="revoke-keys"
Oct 31 12:40:21.927207 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:21.927186    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="b50eace0-22ba-490b-9344-0722c9b6cfa0" containerName="revoke-keys"
Oct 31 12:40:21.933374 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:21.933346    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3e89004-fbpg7"
Oct 31 12:40:21.937808 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:21.937780    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3e89004-fbpg7"]
Oct 31 12:40:21.938357 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:21.938337    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 12:40:21.939191 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:21.938764    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7wsmb\""
Oct 31 12:40:21.939191 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:21.938855    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 12:40:21.939191 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:21.938973    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 12:40:22.017517 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:21.999827    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-2fn6t"
Oct 31 12:40:22.018946 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:22.018615    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-2fn6t"
Oct 31 12:40:22.021318 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:22.021262    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-2fn6t"
Oct 31 12:40:22.072538 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:22.060936    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19589aa0-238f-4078-9052-b27af7131832-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a3e89004-fbpg7\" (UID: \"19589aa0-238f-4078-9052-b27af7131832\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3e89004-fbpg7"
Oct 31 12:40:22.072538 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:22.061026    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19589aa0-238f-4078-9052-b27af7131832-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a3e89004-fbpg7\" (UID: \"19589aa0-238f-4078-9052-b27af7131832\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3e89004-fbpg7"
Oct 31 12:40:22.072538 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:22.061109    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhn78\" (UniqueName: \"kubernetes.io/projected/19589aa0-238f-4078-9052-b27af7131832-kube-api-access-zhn78\") pod \"maas-api-revoke-sub-a3e89004-fbpg7\" (UID: \"19589aa0-238f-4078-9052-b27af7131832\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3e89004-fbpg7"
Oct 31 12:40:22.174123 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:22.174092    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19589aa0-238f-4078-9052-b27af7131832-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a3e89004-fbpg7\" (UID: \"19589aa0-238f-4078-9052-b27af7131832\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3e89004-fbpg7"
Oct 31 12:40:22.174866 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:22.174843    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19589aa0-238f-4078-9052-b27af7131832-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a3e89004-fbpg7\" (UID: \"19589aa0-238f-4078-9052-b27af7131832\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3e89004-fbpg7"
Oct 31 12:40:22.175780 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:22.175665    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19589aa0-238f-4078-9052-b27af7131832-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a3e89004-fbpg7\" (UID: \"19589aa0-238f-4078-9052-b27af7131832\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3e89004-fbpg7"
Oct 31 12:40:22.175951 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:22.175936    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhn78\" (UniqueName: \"kubernetes.io/projected/19589aa0-238f-4078-9052-b27af7131832-kube-api-access-zhn78\") pod \"maas-api-revoke-sub-a3e89004-fbpg7\" (UID: \"19589aa0-238f-4078-9052-b27af7131832\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3e89004-fbpg7"
Oct 31 12:40:22.190733 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:22.188911    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19589aa0-238f-4078-9052-b27af7131832-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a3e89004-fbpg7\" (UID: \"19589aa0-238f-4078-9052-b27af7131832\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3e89004-fbpg7"
Oct 31 12:40:22.190733 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:22.189670    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhn78\" (UniqueName: \"kubernetes.io/projected/19589aa0-238f-4078-9052-b27af7131832-kube-api-access-zhn78\") pod \"maas-api-revoke-sub-a3e89004-fbpg7\" (UID: \"19589aa0-238f-4078-9052-b27af7131832\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3e89004-fbpg7"
Oct 31 12:40:22.274526 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:22.271678    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3e89004-fbpg7"
Oct 31 12:40:22.552139 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:22.552095    5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-645cf464d7-rz7mt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Oct 31 12:40:22.552336 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:22.552175    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-rz7mt" podUID="978e5cf9-a89e-4fc9-b4ba-fe0ee2de401b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Oct 31 12:40:22.604339 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:40:22.604306    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19589aa0_238f_4078_9052_b27af7131832.slice/crio-a9354150f004072bc973df1ef0ad0c3d291a468020182c2f14c8a7ba8b58f2e8 WatchSource:0}: Error finding container a9354150f004072bc973df1ef0ad0c3d291a468020182c2f14c8a7ba8b58f2e8: Status 404 returned error can't find the container with id a9354150f004072bc973df1ef0ad0c3d291a468020182c2f14c8a7ba8b58f2e8
Oct 31 12:40:22.604990 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:22.604962    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3e89004-fbpg7"]
Oct 31 12:40:23.309538 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:23.303691    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3e89004-fbpg7" event={"ID":"19589aa0-238f-4078-9052-b27af7131832","Type":"ContainerStarted","Data":"27a99613806fcc2aac38ad5e01181eab7b713f52dd347608cf7fc78045b441ff"}
Oct 31 12:40:23.309538 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:23.303771    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3e89004-fbpg7" event={"ID":"19589aa0-238f-4078-9052-b27af7131832","Type":"ContainerStarted","Data":"a9354150f004072bc973df1ef0ad0c3d291a468020182c2f14c8a7ba8b58f2e8"}
Oct 31 12:40:23.325870 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:23.325820    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3e89004-fbpg7" podStartSLOduration=2.325805525 podStartE2EDuration="2.325805525s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:40:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:40:23.325084236 +0000 UTC m=+786.428151948" watchObservedRunningTime="2026-10-31 12:40:23.325805525 +0000 UTC m=+786.428873285"
Oct 31 12:40:23.552515 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:23.552358    5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-645cf464d7-rz7mt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Oct 31 12:40:23.552515 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:23.552423    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-rz7mt" podUID="978e5cf9-a89e-4fc9-b4ba-fe0ee2de401b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Oct 31 12:40:24.315738 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:24.315697    5060 generic.go:356] "Generic (PLEG): container finished" podID="19589aa0-238f-4078-9052-b27af7131832" containerID="27a99613806fcc2aac38ad5e01181eab7b713f52dd347608cf7fc78045b441ff" exitCode=0
Oct 31 12:40:24.316238 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:24.315782    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3e89004-fbpg7" event={"ID":"19589aa0-238f-4078-9052-b27af7131832","Type":"ContainerDied","Data":"27a99613806fcc2aac38ad5e01181eab7b713f52dd347608cf7fc78045b441ff"}
Oct 31 12:40:24.552145 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:24.552102    5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-645cf464d7-rz7mt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Oct 31 12:40:24.552350 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:24.552174    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-rz7mt" podUID="978e5cf9-a89e-4fc9-b4ba-fe0ee2de401b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Oct 31 12:40:25.174056 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:25.174011    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-b25d3b/distinct-2-w5-b25d3b-kserve-76f798d59f-vvzhg"]
Oct 31 12:40:25.178442 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:25.178417    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-2-w5-b25d3b-kserve-76f798d59f-vvzhg"
Oct 31 12:40:25.198134 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:25.198111    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-b25d3b\"/\"distinct-2-w5-b25d3b-kserve-self-signed-certs\""
Oct 31 12:40:25.202485 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:25.202433    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-b25d3b\"/\"kube-root-ca.crt\""
Oct 31 12:40:25.202624 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:25.202521    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-b25d3b\"/\"openshift-service-ca.crt\""
Oct 31 12:40:25.210900 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:25.210872    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-b25d3b/distinct-2-w5-b25d3b-kserve-76f798d59f-vvzhg"]
Oct 31 12:40:25.228457 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:25.228420    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qhrx\" (UniqueName: \"kubernetes.io/projected/394ed76c-84ac-4612-9acb-a7e36cb68d89-kube-api-access-6qhrx\") pod \"distinct-2-w5-b25d3b-kserve-76f798d59f-vvzhg\" (UID: \"394ed76c-84ac-4612-9acb-a7e36cb68d89\") " pod="e2e-models-e2e-worker-w5-b25d3b/distinct-2-w5-b25d3b-kserve-76f798d59f-vvzhg"
Oct 31 12:40:25.228667 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:25.228481    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/394ed76c-84ac-4612-9acb-a7e36cb68d89-home\") pod \"distinct-2-w5-b25d3b-kserve-76f798d59f-vvzhg\" (UID: \"394ed76c-84ac-4612-9acb-a7e36cb68d89\") " pod="e2e-models-e2e-worker-w5-b25d3b/distinct-2-w5-b25d3b-kserve-76f798d59f-vvzhg"
Oct 31 12:40:25.228667 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:25.228538    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/394ed76c-84ac-4612-9acb-a7e36cb68d89-tls-certs\") pod \"distinct-2-w5-b25d3b-kserve-76f798d59f-vvzhg\" (UID: \"394ed76c-84ac-4612-9acb-a7e36cb68d89\") " pod="e2e-models-e2e-worker-w5-b25d3b/distinct-2-w5-b25d3b-kserve-76f798d59f-vvzhg"
Oct 31 12:40:25.228828 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:25.228686    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/394ed76c-84ac-4612-9acb-a7e36cb68d89-model-cache\") pod \"distinct-2-w5-b25d3b-kserve-76f798d59f-vvzhg\" (UID: \"394ed76c-84ac-4612-9acb-a7e36cb68d89\") " pod="e2e-models-e2e-worker-w5-b25d3b/distinct-2-w5-b25d3b-kserve-76f798d59f-vvzhg"
Oct 31 12:40:25.228828 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:25.228725    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/394ed76c-84ac-4612-9acb-a7e36cb68d89-tmp-dir\") pod \"distinct-2-w5-b25d3b-kserve-76f798d59f-vvzhg\" (UID: \"394ed76c-84ac-4612-9acb-a7e36cb68d89\") " pod="e2e-models-e2e-worker-w5-b25d3b/distinct-2-w5-b25d3b-kserve-76f798d59f-vvzhg"
Oct 31 12:40:25.228828 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:25.228788    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/394ed76c-84ac-4612-9acb-a7e36cb68d89-dshm\") pod \"distinct-2-w5-b25d3b-kserve-76f798d59f-vvzhg\" (UID: \"394ed76c-84ac-4612-9acb-a7e36cb68d89\") " pod="e2e-models-e2e-worker-w5-b25d3b/distinct-2-w5-b25d3b-kserve-76f798d59f-vvzhg"
Oct 31 12:40:25.298738 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:25.298706    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-b25d3b\"/\"default-dockercfg-rt29c\""
Oct 31 12:40:25.339517 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:25.337007    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qhrx\" (UniqueName: \"kubernetes.io/projected/394ed76c-84ac-4612-9acb-a7e36cb68d89-kube-api-access-6qhrx\") pod \"distinct-2-w5-b25d3b-kserve-76f798d59f-vvzhg\" (UID: \"394ed76c-84ac-4612-9acb-a7e36cb68d89\") " pod="e2e-models-e2e-worker-w5-b25d3b/distinct-2-w5-b25d3b-kserve-76f798d59f-vvzhg"
Oct 31 12:40:25.339517 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:25.337061    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/394ed76c-84ac-4612-9acb-a7e36cb68d89-home\") pod \"distinct-2-w5-b25d3b-kserve-76f798d59f-vvzhg\" (UID: \"394ed76c-84ac-4612-9acb-a7e36cb68d89\") " pod="e2e-models-e2e-worker-w5-b25d3b/distinct-2-w5-b25d3b-kserve-76f798d59f-vvzhg"
Oct 31 12:40:25.339517 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:25.337094    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/394ed76c-84ac-4612-9acb-a7e36cb68d89-tls-certs\") pod \"distinct-2-w5-b25d3b-kserve-76f798d59f-vvzhg\" (UID: \"394ed76c-84ac-4612-9acb-a7e36cb68d89\") " pod="e2e-models-e2e-worker-w5-b25d3b/distinct-2-w5-b25d3b-kserve-76f798d59f-vvzhg"
Oct 31 12:40:25.339517 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:25.337223    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/394ed76c-84ac-4612-9acb-a7e36cb68d89-model-cache\") pod \"distinct-2-w5-b25d3b-kserve-76f798d59f-vvzhg\" (UID: \"394ed76c-84ac-4612-9acb-a7e36cb68d89\") " pod="e2e-models-e2e-worker-w5-b25d3b/distinct-2-w5-b25d3b-kserve-76f798d59f-vvzhg"
Oct 31 12:40:25.339517 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:25.337258    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/394ed76c-84ac-4612-9acb-a7e36cb68d89-tmp-dir\") pod \"distinct-2-w5-b25d3b-kserve-76f798d59f-vvzhg\" (UID: \"394ed76c-84ac-4612-9acb-a7e36cb68d89\") " pod="e2e-models-e2e-worker-w5-b25d3b/distinct-2-w5-b25d3b-kserve-76f798d59f-vvzhg"
Oct 31 12:40:25.339517 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:25.337314    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/394ed76c-84ac-4612-9acb-a7e36cb68d89-dshm\") pod \"distinct-2-w5-b25d3b-kserve-76f798d59f-vvzhg\" (UID: \"394ed76c-84ac-4612-9acb-a7e36cb68d89\") " pod="e2e-models-e2e-worker-w5-b25d3b/distinct-2-w5-b25d3b-kserve-76f798d59f-vvzhg"
Oct 31 12:40:25.348519 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:25.346027    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/394ed76c-84ac-4612-9acb-a7e36cb68d89-home\") pod \"distinct-2-w5-b25d3b-kserve-76f798d59f-vvzhg\" (UID: \"394ed76c-84ac-4612-9acb-a7e36cb68d89\") " pod="e2e-models-e2e-worker-w5-b25d3b/distinct-2-w5-b25d3b-kserve-76f798d59f-vvzhg"
Oct 31 12:40:25.348519 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:25.347594    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/394ed76c-84ac-4612-9acb-a7e36cb68d89-model-cache\") pod \"distinct-2-w5-b25d3b-kserve-76f798d59f-vvzhg\" (UID: \"394ed76c-84ac-4612-9acb-a7e36cb68d89\") " pod="e2e-models-e2e-worker-w5-b25d3b/distinct-2-w5-b25d3b-kserve-76f798d59f-vvzhg"
Oct 31 12:40:25.363525 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:25.349465    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/394ed76c-84ac-4612-9acb-a7e36cb68d89-tmp-dir\") pod \"distinct-2-w5-b25d3b-kserve-76f798d59f-vvzhg\" (UID: \"394ed76c-84ac-4612-9acb-a7e36cb68d89\") " pod="e2e-models-e2e-worker-w5-b25d3b/distinct-2-w5-b25d3b-kserve-76f798d59f-vvzhg"
Oct 31 12:40:25.363525 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:25.349771    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/394ed76c-84ac-4612-9acb-a7e36cb68d89-dshm\") pod \"distinct-2-w5-b25d3b-kserve-76f798d59f-vvzhg\" (UID: \"394ed76c-84ac-4612-9acb-a7e36cb68d89\") " pod="e2e-models-e2e-worker-w5-b25d3b/distinct-2-w5-b25d3b-kserve-76f798d59f-vvzhg"
Oct 31 12:40:25.372295 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:25.372264    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/394ed76c-84ac-4612-9acb-a7e36cb68d89-tls-certs\") pod \"distinct-2-w5-b25d3b-kserve-76f798d59f-vvzhg\" (UID: \"394ed76c-84ac-4612-9acb-a7e36cb68d89\") " pod="e2e-models-e2e-worker-w5-b25d3b/distinct-2-w5-b25d3b-kserve-76f798d59f-vvzhg"
Oct 31 12:40:25.375515 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:25.375441    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qhrx\" (UniqueName: \"kubernetes.io/projected/394ed76c-84ac-4612-9acb-a7e36cb68d89-kube-api-access-6qhrx\") pod \"distinct-2-w5-b25d3b-kserve-76f798d59f-vvzhg\" (UID: \"394ed76c-84ac-4612-9acb-a7e36cb68d89\") " pod="e2e-models-e2e-worker-w5-b25d3b/distinct-2-w5-b25d3b-kserve-76f798d59f-vvzhg"
Oct 31 12:40:25.492975 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:25.492022    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-2-w5-b25d3b-kserve-76f798d59f-vvzhg"
Oct 31 12:40:25.552796 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:25.552759    5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-645cf464d7-rz7mt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Oct 31 12:40:25.552982 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:25.552815    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-rz7mt" podUID="978e5cf9-a89e-4fc9-b4ba-fe0ee2de401b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Oct 31 12:40:25.593003 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:25.591091    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3e89004-fbpg7"
Oct 31 12:40:25.640699 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:25.640659    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19589aa0-238f-4078-9052-b27af7131832-kube-api-access-zhn78\" (UniqueName: \"kubernetes.io/projected/19589aa0-238f-4078-9052-b27af7131832-kube-api-access-zhn78\") pod \"19589aa0-238f-4078-9052-b27af7131832\" (UID: \"19589aa0-238f-4078-9052-b27af7131832\") "
Oct 31 12:40:25.640836 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:25.640754    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19589aa0-238f-4078-9052-b27af7131832-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19589aa0-238f-4078-9052-b27af7131832-maas-api-openshift-ca\") pod \"19589aa0-238f-4078-9052-b27af7131832\" (UID: \"19589aa0-238f-4078-9052-b27af7131832\") "
Oct 31 12:40:25.640931 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:25.640910    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/19589aa0-238f-4078-9052-b27af7131832-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19589aa0-238f-4078-9052-b27af7131832-maas-api-xks-ca\") pod \"19589aa0-238f-4078-9052-b27af7131832\" (UID: \"19589aa0-238f-4078-9052-b27af7131832\") "
Oct 31 12:40:25.643712 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:25.643677    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19589aa0-238f-4078-9052-b27af7131832-kube-api-access-zhn78" pod "19589aa0-238f-4078-9052-b27af7131832" (UID: "19589aa0-238f-4078-9052-b27af7131832"). InnerVolumeSpecName "kube-api-access-zhn78". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:40:25.643886 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:25.643864    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19589aa0-238f-4078-9052-b27af7131832-maas-api-xks-ca" pod "19589aa0-238f-4078-9052-b27af7131832" (UID: "19589aa0-238f-4078-9052-b27af7131832"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:40:25.644028 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:25.644006    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19589aa0-238f-4078-9052-b27af7131832-maas-api-openshift-ca" pod "19589aa0-238f-4078-9052-b27af7131832" (UID: "19589aa0-238f-4078-9052-b27af7131832"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:40:25.742298 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:25.742211    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19589aa0-238f-4078-9052-b27af7131832-maas-api-xks-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:40:25.742298 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:25.742240    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhn78\" (UniqueName: \"kubernetes.io/projected/19589aa0-238f-4078-9052-b27af7131832-kube-api-access-zhn78\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:40:25.742298 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:25.742252    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19589aa0-238f-4078-9052-b27af7131832-maas-api-openshift-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:40:25.810597 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:40:25.810560    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod394ed76c_84ac_4612_9acb_a7e36cb68d89.slice/crio-6cf96c6ede1f75a8b30adcf4b90c62aae92014994fb1bbd968771a672caba75a WatchSource:0}: Error finding container 6cf96c6ede1f75a8b30adcf4b90c62aae92014994fb1bbd968771a672caba75a: Status 404 returned error can't find the container with id 6cf96c6ede1f75a8b30adcf4b90c62aae92014994fb1bbd968771a672caba75a
Oct 31 12:40:25.811431 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:25.811398    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-b25d3b/distinct-2-w5-b25d3b-kserve-76f798d59f-vvzhg"]
Oct 31 12:40:26.119483 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:26.119450    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98581cd6-p78l7"]
Oct 31 12:40:26.120066 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:26.120051    5060 cpu_manager.go:414] "containerMap: removing container" podUID="19589aa0-238f-4078-9052-b27af7131832" containerName="revoke-keys"
Oct 31 12:40:26.120136 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:26.120067    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19589aa0-238f-4078-9052-b27af7131832" containerName="revoke-keys"
Oct 31 12:40:26.120199 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:26.120190    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="19589aa0-238f-4078-9052-b27af7131832" containerName="revoke-keys"
Oct 31 12:40:26.124073 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:26.124047    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98581cd6-p78l7"
Oct 31 12:40:26.146634 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:26.146595    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ffks\" (UniqueName: \"kubernetes.io/projected/d547045f-f0f5-4244-a709-dc84a1fd5de9-kube-api-access-7ffks\") pod \"maas-api-revoke-sub-98581cd6-p78l7\" (UID: \"d547045f-f0f5-4244-a709-dc84a1fd5de9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98581cd6-p78l7"
Oct 31 12:40:26.146990 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:26.146964    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d547045f-f0f5-4244-a709-dc84a1fd5de9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-98581cd6-p78l7\" (UID: \"d547045f-f0f5-4244-a709-dc84a1fd5de9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98581cd6-p78l7"
Oct 31 12:40:26.147232 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:26.147207    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d547045f-f0f5-4244-a709-dc84a1fd5de9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-98581cd6-p78l7\" (UID: \"d547045f-f0f5-4244-a709-dc84a1fd5de9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98581cd6-p78l7"
Oct 31 12:40:26.177830 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:26.177792    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98581cd6-p78l7"]
Oct 31 12:40:26.248583 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:26.248548    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ffks\" (UniqueName: \"kubernetes.io/projected/d547045f-f0f5-4244-a709-dc84a1fd5de9-kube-api-access-7ffks\") pod \"maas-api-revoke-sub-98581cd6-p78l7\" (UID: \"d547045f-f0f5-4244-a709-dc84a1fd5de9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98581cd6-p78l7"
Oct 31 12:40:26.248751 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:26.248647    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d547045f-f0f5-4244-a709-dc84a1fd5de9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-98581cd6-p78l7\" (UID: \"d547045f-f0f5-4244-a709-dc84a1fd5de9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98581cd6-p78l7"
Oct 31 12:40:26.248751 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:26.248729    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d547045f-f0f5-4244-a709-dc84a1fd5de9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-98581cd6-p78l7\" (UID: \"d547045f-f0f5-4244-a709-dc84a1fd5de9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98581cd6-p78l7"
Oct 31 12:40:26.249683 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:26.249654    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d547045f-f0f5-4244-a709-dc84a1fd5de9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-98581cd6-p78l7\" (UID: \"d547045f-f0f5-4244-a709-dc84a1fd5de9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98581cd6-p78l7"
Oct 31 12:40:26.251015 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:26.250995    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d547045f-f0f5-4244-a709-dc84a1fd5de9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-98581cd6-p78l7\" (UID: \"d547045f-f0f5-4244-a709-dc84a1fd5de9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98581cd6-p78l7"
Oct 31 12:40:26.260719 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:26.260693    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ffks\" (UniqueName: \"kubernetes.io/projected/d547045f-f0f5-4244-a709-dc84a1fd5de9-kube-api-access-7ffks\") pod \"maas-api-revoke-sub-98581cd6-p78l7\" (UID: \"d547045f-f0f5-4244-a709-dc84a1fd5de9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98581cd6-p78l7"
Oct 31 12:40:26.386118 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:26.386014    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3e89004-fbpg7" event={"ID":"19589aa0-238f-4078-9052-b27af7131832","Type":"ContainerDied","Data":"a9354150f004072bc973df1ef0ad0c3d291a468020182c2f14c8a7ba8b58f2e8"}
Oct 31 12:40:26.386118 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:26.386085    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a9354150f004072bc973df1ef0ad0c3d291a468020182c2f14c8a7ba8b58f2e8"
Oct 31 12:40:26.386118 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:26.386030    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3e89004-fbpg7"
Oct 31 12:40:26.388472 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:26.388439    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-2-w5-b25d3b-kserve-76f798d59f-vvzhg" event={"ID":"394ed76c-84ac-4612-9acb-a7e36cb68d89","Type":"ContainerStarted","Data":"1270a228fe9e90c1c467bed01ee44027f4ab4b23919044785878fe418213c5bd"}
Oct 31 12:40:26.388629 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:26.388527    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-2-w5-b25d3b-kserve-76f798d59f-vvzhg" event={"ID":"394ed76c-84ac-4612-9acb-a7e36cb68d89","Type":"ContainerStarted","Data":"6cf96c6ede1f75a8b30adcf4b90c62aae92014994fb1bbd968771a672caba75a"}
Oct 31 12:40:26.413466 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:26.413414    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-2-w5-b25d3b-kserve-76f798d59f-vvzhg" podStartSLOduration=1.413400722 podStartE2EDuration="1.413400722s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:40:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:40:26.409537938 +0000 UTC m=+789.512605651" watchObservedRunningTime="2026-10-31 12:40:26.413400722 +0000 UTC m=+789.516468434"
Oct 31 12:40:26.491656 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:26.491623    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98581cd6-p78l7"
Oct 31 12:40:26.492671 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:26.492639    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-2-w5-b25d3b-kserve-76f798d59f-vvzhg"
Oct 31 12:40:26.810114 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:40:26.810083    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd547045f_f0f5_4244_a709_dc84a1fd5de9.slice/crio-b1b03a93ab6b517b2680f3172b84ced626ad87549ed663371ccfc3b6ecdb887a WatchSource:0}: Error finding container b1b03a93ab6b517b2680f3172b84ced626ad87549ed663371ccfc3b6ecdb887a: Status 404 returned error can't find the container with id b1b03a93ab6b517b2680f3172b84ced626ad87549ed663371ccfc3b6ecdb887a
Oct 31 12:40:26.813660 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:26.813630    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98581cd6-p78l7"]
Oct 31 12:40:27.243412 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:27.243368    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-rz7mt"
Oct 31 12:40:27.243887 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:27.243871    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-rz7mt"
Oct 31 12:40:27.244741 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:27.244718    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-rz7mt"
Oct 31 12:40:27.403113 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:27.403075    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98581cd6-p78l7" event={"ID":"d547045f-f0f5-4244-a709-dc84a1fd5de9","Type":"ContainerStarted","Data":"b615ac3dcc628fa62bac5a336ad513972f03503c5b23495f22297bc1efd6eccb"}
Oct 31 12:40:27.403602 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:27.403132    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98581cd6-p78l7" event={"ID":"d547045f-f0f5-4244-a709-dc84a1fd5de9","Type":"ContainerStarted","Data":"b1b03a93ab6b517b2680f3172b84ced626ad87549ed663371ccfc3b6ecdb887a"}
Oct 31 12:40:27.427178 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:27.427082    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98581cd6-p78l7" podStartSLOduration=1.42706837 podStartE2EDuration="1.42706837s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:40:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:40:27.422748339 +0000 UTC m=+790.525816067" watchObservedRunningTime="2026-10-31 12:40:27.42706837 +0000 UTC m=+790.530136104"
Oct 31 12:40:28.412533 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:28.412475    5060 generic.go:356] "Generic (PLEG): container finished" podID="d547045f-f0f5-4244-a709-dc84a1fd5de9" containerID="b615ac3dcc628fa62bac5a336ad513972f03503c5b23495f22297bc1efd6eccb" exitCode=0
Oct 31 12:40:28.412976 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:28.412588    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98581cd6-p78l7" event={"ID":"d547045f-f0f5-4244-a709-dc84a1fd5de9","Type":"ContainerDied","Data":"b615ac3dcc628fa62bac5a336ad513972f03503c5b23495f22297bc1efd6eccb"}
Oct 31 12:40:29.587673 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:29.587636    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98581cd6-p78l7"
Oct 31 12:40:29.705581 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:29.705407    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d547045f-f0f5-4244-a709-dc84a1fd5de9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d547045f-f0f5-4244-a709-dc84a1fd5de9-maas-api-xks-ca\") pod \"d547045f-f0f5-4244-a709-dc84a1fd5de9\" (UID: \"d547045f-f0f5-4244-a709-dc84a1fd5de9\") "
Oct 31 12:40:29.705791 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:29.705652    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d547045f-f0f5-4244-a709-dc84a1fd5de9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d547045f-f0f5-4244-a709-dc84a1fd5de9-maas-api-openshift-ca\") pod \"d547045f-f0f5-4244-a709-dc84a1fd5de9\" (UID: \"d547045f-f0f5-4244-a709-dc84a1fd5de9\") "
Oct 31 12:40:29.705791 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:29.705690    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d547045f-f0f5-4244-a709-dc84a1fd5de9-kube-api-access-7ffks\" (UniqueName: \"kubernetes.io/projected/d547045f-f0f5-4244-a709-dc84a1fd5de9-kube-api-access-7ffks\") pod \"d547045f-f0f5-4244-a709-dc84a1fd5de9\" (UID: \"d547045f-f0f5-4244-a709-dc84a1fd5de9\") "
Oct 31 12:40:29.706897 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:29.706863    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d547045f-f0f5-4244-a709-dc84a1fd5de9-maas-api-openshift-ca" pod "d547045f-f0f5-4244-a709-dc84a1fd5de9" (UID: "d547045f-f0f5-4244-a709-dc84a1fd5de9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:40:29.708793 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:29.708765    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d547045f-f0f5-4244-a709-dc84a1fd5de9-kube-api-access-7ffks" pod "d547045f-f0f5-4244-a709-dc84a1fd5de9" (UID: "d547045f-f0f5-4244-a709-dc84a1fd5de9"). InnerVolumeSpecName "kube-api-access-7ffks". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:40:29.710620 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:29.710592    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d547045f-f0f5-4244-a709-dc84a1fd5de9-maas-api-xks-ca" pod "d547045f-f0f5-4244-a709-dc84a1fd5de9" (UID: "d547045f-f0f5-4244-a709-dc84a1fd5de9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:40:29.807649 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:29.807609    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d547045f-f0f5-4244-a709-dc84a1fd5de9-maas-api-xks-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:40:29.807649 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:29.807641    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d547045f-f0f5-4244-a709-dc84a1fd5de9-maas-api-openshift-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:40:29.807649 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:29.807655    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ffks\" (UniqueName: \"kubernetes.io/projected/d547045f-f0f5-4244-a709-dc84a1fd5de9-kube-api-access-7ffks\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:40:29.923457 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:40:29.923422    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7phb" containerName="extractor"
Oct 31 12:40:30.432538 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:30.432476    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98581cd6-p78l7"
Oct 31 12:40:30.432538 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:30.432526    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98581cd6-p78l7" event={"ID":"d547045f-f0f5-4244-a709-dc84a1fd5de9","Type":"ContainerDied","Data":"b1b03a93ab6b517b2680f3172b84ced626ad87549ed663371ccfc3b6ecdb887a"}
Oct 31 12:40:30.432784 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:30.432576    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b1b03a93ab6b517b2680f3172b84ced626ad87549ed663371ccfc3b6ecdb887a"
Oct 31 12:40:32.844520 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:32.843245    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/premium-w2-e6c79d-kserve-6468bf46d4-h66f8" podUID="cd6a266f-8d12-4089-83ee-c75c716bf26c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:40:35.501556 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:35.500906    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-2-w5-b25d3b-kserve-76f798d59f-vvzhg"
Oct 31 12:40:35.530735 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:35.530702    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-2-w5-b25d3b-kserve-76f798d59f-vvzhg"
Oct 31 12:40:35.550745 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:35.550717    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-2-w5-b25d3b-kserve-76f798d59f-vvzhg"
Oct 31 12:40:36.825701 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:36.825666    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-sd68f"]
Oct 31 12:40:36.826461 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:36.826439    5060 cpu_manager.go:414] "containerMap: removing container" podUID="d547045f-f0f5-4244-a709-dc84a1fd5de9" containerName="revoke-keys"
Oct 31 12:40:36.826608 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:36.826463    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d547045f-f0f5-4244-a709-dc84a1fd5de9" containerName="revoke-keys"
Oct 31 12:40:36.826729 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:36.826714    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="d547045f-f0f5-4244-a709-dc84a1fd5de9" containerName="revoke-keys"
Oct 31 12:40:36.832000 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:36.831979    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-sd68f"
Oct 31 12:40:36.849835 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:36.844816    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-sd68f"]
Oct 31 12:40:36.887025 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:36.886979    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4129ab4f-7756-4a6d-8d7f-9dff5d0f1c57-istiod-ca-cert\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-sd68f\" (UID: \"4129ab4f-7756-4a6d-8d7f-9dff5d0f1c57\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-sd68f"
Oct 31 12:40:36.887209 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:36.887037    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4129ab4f-7756-4a6d-8d7f-9dff5d0f1c57-istio-podinfo\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-sd68f\" (UID: \"4129ab4f-7756-4a6d-8d7f-9dff5d0f1c57\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-sd68f"
Oct 31 12:40:36.887209 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:36.887155    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h546b\" (UniqueName: \"kubernetes.io/projected/4129ab4f-7756-4a6d-8d7f-9dff5d0f1c57-kube-api-access-h546b\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-sd68f\" (UID: \"4129ab4f-7756-4a6d-8d7f-9dff5d0f1c57\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-sd68f"
Oct 31 12:40:36.887297 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:36.887209    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4129ab4f-7756-4a6d-8d7f-9dff5d0f1c57-istio-token\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-sd68f\" (UID: \"4129ab4f-7756-4a6d-8d7f-9dff5d0f1c57\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-sd68f"
Oct 31 12:40:36.887297 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:36.887252    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4129ab4f-7756-4a6d-8d7f-9dff5d0f1c57-istio-data\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-sd68f\" (UID: \"4129ab4f-7756-4a6d-8d7f-9dff5d0f1c57\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-sd68f"
Oct 31 12:40:36.887297 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:36.887280    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4129ab4f-7756-4a6d-8d7f-9dff5d0f1c57-istio-envoy\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-sd68f\" (UID: \"4129ab4f-7756-4a6d-8d7f-9dff5d0f1c57\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-sd68f"
Oct 31 12:40:36.887387 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:36.887329    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4129ab4f-7756-4a6d-8d7f-9dff5d0f1c57-workload-socket\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-sd68f\" (UID: \"4129ab4f-7756-4a6d-8d7f-9dff5d0f1c57\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-sd68f"
Oct 31 12:40:36.887417 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:36.887383    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4129ab4f-7756-4a6d-8d7f-9dff5d0f1c57-workload-certs\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-sd68f\" (UID: \"4129ab4f-7756-4a6d-8d7f-9dff5d0f1c57\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-sd68f"
Oct 31 12:40:36.887448 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:36.887416    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4129ab4f-7756-4a6d-8d7f-9dff5d0f1c57-credential-socket\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-sd68f\" (UID: \"4129ab4f-7756-4a6d-8d7f-9dff5d0f1c57\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-sd68f"
Oct 31 12:40:36.988757 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:36.988725    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h546b\" (UniqueName: \"kubernetes.io/projected/4129ab4f-7756-4a6d-8d7f-9dff5d0f1c57-kube-api-access-h546b\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-sd68f\" (UID: \"4129ab4f-7756-4a6d-8d7f-9dff5d0f1c57\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-sd68f"
Oct 31 12:40:36.988953 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:36.988774    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4129ab4f-7756-4a6d-8d7f-9dff5d0f1c57-istio-token\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-sd68f\" (UID: \"4129ab4f-7756-4a6d-8d7f-9dff5d0f1c57\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-sd68f"
Oct 31 12:40:36.988953 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:36.988900    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4129ab4f-7756-4a6d-8d7f-9dff5d0f1c57-istio-data\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-sd68f\" (UID: \"4129ab4f-7756-4a6d-8d7f-9dff5d0f1c57\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-sd68f"
Oct 31 12:40:36.988953 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:36.988951    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4129ab4f-7756-4a6d-8d7f-9dff5d0f1c57-istio-envoy\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-sd68f\" (UID: \"4129ab4f-7756-4a6d-8d7f-9dff5d0f1c57\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-sd68f"
Oct 31 12:40:36.989129 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:36.989013    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4129ab4f-7756-4a6d-8d7f-9dff5d0f1c57-workload-socket\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-sd68f\" (UID: \"4129ab4f-7756-4a6d-8d7f-9dff5d0f1c57\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-sd68f"
Oct 31 12:40:36.989129 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:36.989074    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4129ab4f-7756-4a6d-8d7f-9dff5d0f1c57-workload-certs\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-sd68f\" (UID: \"4129ab4f-7756-4a6d-8d7f-9dff5d0f1c57\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-sd68f"
Oct 31 12:40:36.989129 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:36.989109    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4129ab4f-7756-4a6d-8d7f-9dff5d0f1c57-credential-socket\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-sd68f\" (UID: \"4129ab4f-7756-4a6d-8d7f-9dff5d0f1c57\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-sd68f"
Oct 31 12:40:36.989324 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:36.989300    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4129ab4f-7756-4a6d-8d7f-9dff5d0f1c57-istiod-ca-cert\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-sd68f\" (UID: \"4129ab4f-7756-4a6d-8d7f-9dff5d0f1c57\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-sd68f"
Oct 31 12:40:36.989412 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:36.989351    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4129ab4f-7756-4a6d-8d7f-9dff5d0f1c57-istio-podinfo\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-sd68f\" (UID: \"4129ab4f-7756-4a6d-8d7f-9dff5d0f1c57\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-sd68f"
Oct 31 12:40:36.989467 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:36.989423    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4129ab4f-7756-4a6d-8d7f-9dff5d0f1c57-istio-data\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-sd68f\" (UID: \"4129ab4f-7756-4a6d-8d7f-9dff5d0f1c57\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-sd68f"
Oct 31 12:40:36.989542 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:36.989517    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4129ab4f-7756-4a6d-8d7f-9dff5d0f1c57-credential-socket\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-sd68f\" (UID: \"4129ab4f-7756-4a6d-8d7f-9dff5d0f1c57\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-sd68f"
Oct 31 12:40:36.989635 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:36.989610    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4129ab4f-7756-4a6d-8d7f-9dff5d0f1c57-workload-certs\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-sd68f\" (UID: \"4129ab4f-7756-4a6d-8d7f-9dff5d0f1c57\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-sd68f"
Oct 31 12:40:36.989635 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:36.989628    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4129ab4f-7756-4a6d-8d7f-9dff5d0f1c57-workload-socket\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-sd68f\" (UID: \"4129ab4f-7756-4a6d-8d7f-9dff5d0f1c57\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-sd68f"
Oct 31 12:40:36.990142 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:36.990116    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4129ab4f-7756-4a6d-8d7f-9dff5d0f1c57-istiod-ca-cert\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-sd68f\" (UID: \"4129ab4f-7756-4a6d-8d7f-9dff5d0f1c57\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-sd68f"
Oct 31 12:40:36.991807 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:36.991779    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4129ab4f-7756-4a6d-8d7f-9dff5d0f1c57-istio-envoy\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-sd68f\" (UID: \"4129ab4f-7756-4a6d-8d7f-9dff5d0f1c57\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-sd68f"
Oct 31 12:40:36.991910 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:36.991866    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4129ab4f-7756-4a6d-8d7f-9dff5d0f1c57-istio-podinfo\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-sd68f\" (UID: \"4129ab4f-7756-4a6d-8d7f-9dff5d0f1c57\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-sd68f"
Oct 31 12:40:37.001243 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:37.001219    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h546b\" (UniqueName: \"kubernetes.io/projected/4129ab4f-7756-4a6d-8d7f-9dff5d0f1c57-kube-api-access-h546b\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-sd68f\" (UID: \"4129ab4f-7756-4a6d-8d7f-9dff5d0f1c57\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-sd68f"
Oct 31 12:40:37.001344 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:37.001254    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4129ab4f-7756-4a6d-8d7f-9dff5d0f1c57-istio-token\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-sd68f\" (UID: \"4129ab4f-7756-4a6d-8d7f-9dff5d0f1c57\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-sd68f"
Oct 31 12:40:37.153642 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:37.153559    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-sd68f"
Oct 31 12:40:37.445639 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:37.445606    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-sd68f"]
Oct 31 12:40:37.446046 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:40:37.446018    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4129ab4f_7756_4a6d_8d7f_9dff5d0f1c57.slice/crio-977487d36393140b45690701ef80d87813f658a59a4c69355a10d0a42803f1c3 WatchSource:0}: Error finding container 977487d36393140b45690701ef80d87813f658a59a4c69355a10d0a42803f1c3: Status 404 returned error can't find the container with id 977487d36393140b45690701ef80d87813f658a59a4c69355a10d0a42803f1c3
Oct 31 12:40:37.455236 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:37.454128    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:40:37.455236 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:37.454195    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:40:37.455236 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:37.454229    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:40:37.509479 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:37.509446    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-8s2sm"]
Oct 31 12:40:37.517059 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:37.517035    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-8s2sm"
Oct 31 12:40:37.526206 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:37.526180    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-8s2sm"]
Oct 31 12:40:37.571257 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:37.571219    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-sd68f" event={"ID":"4129ab4f-7756-4a6d-8d7f-9dff5d0f1c57","Type":"ContainerStarted","Data":"977487d36393140b45690701ef80d87813f658a59a4c69355a10d0a42803f1c3"}
Oct 31 12:40:37.597542 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:37.597432    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae6f8c0f-87f7-4ad7-85e4-4d69e29ded3b-istio-envoy\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-8s2sm\" (UID: \"ae6f8c0f-87f7-4ad7-85e4-4d69e29ded3b\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-8s2sm"
Oct 31 12:40:37.597779 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:37.597747    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae6f8c0f-87f7-4ad7-85e4-4d69e29ded3b-istio-token\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-8s2sm\" (UID: \"ae6f8c0f-87f7-4ad7-85e4-4d69e29ded3b\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-8s2sm"
Oct 31 12:40:37.597934 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:37.597904    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae6f8c0f-87f7-4ad7-85e4-4d69e29ded3b-istio-data\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-8s2sm\" (UID: \"ae6f8c0f-87f7-4ad7-85e4-4d69e29ded3b\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-8s2sm"
Oct 31 12:40:37.598096 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:37.598064    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhhd8\" (UniqueName: \"kubernetes.io/projected/ae6f8c0f-87f7-4ad7-85e4-4d69e29ded3b-kube-api-access-zhhd8\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-8s2sm\" (UID: \"ae6f8c0f-87f7-4ad7-85e4-4d69e29ded3b\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-8s2sm"
Oct 31 12:40:37.598341 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:37.598310    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae6f8c0f-87f7-4ad7-85e4-4d69e29ded3b-workload-certs\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-8s2sm\" (UID: \"ae6f8c0f-87f7-4ad7-85e4-4d69e29ded3b\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-8s2sm"
Oct 31 12:40:37.598502 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:37.598462    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae6f8c0f-87f7-4ad7-85e4-4d69e29ded3b-workload-socket\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-8s2sm\" (UID: \"ae6f8c0f-87f7-4ad7-85e4-4d69e29ded3b\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-8s2sm"
Oct 31 12:40:37.598646 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:37.598614    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae6f8c0f-87f7-4ad7-85e4-4d69e29ded3b-istiod-ca-cert\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-8s2sm\" (UID: \"ae6f8c0f-87f7-4ad7-85e4-4d69e29ded3b\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-8s2sm"
Oct 31 12:40:37.598793 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:37.598778    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae6f8c0f-87f7-4ad7-85e4-4d69e29ded3b-credential-socket\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-8s2sm\" (UID: \"ae6f8c0f-87f7-4ad7-85e4-4d69e29ded3b\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-8s2sm"
Oct 31 12:40:37.598972 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:37.598939    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae6f8c0f-87f7-4ad7-85e4-4d69e29ded3b-istio-podinfo\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-8s2sm\" (UID: \"ae6f8c0f-87f7-4ad7-85e4-4d69e29ded3b\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-8s2sm"
Oct 31 12:40:37.690936 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:37.688848    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f518e29f-nt5kd"]
Oct 31 12:40:37.694689 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:37.694619    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f518e29f-nt5kd"
Oct 31 12:40:37.699533 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:37.698934    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 12:40:37.699533 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:37.699053    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 12:40:37.699533 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:37.699169    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7wsmb\""
Oct 31 12:40:37.702512 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:37.699904    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzjtx\" (UniqueName: \"kubernetes.io/projected/ef3b98e5-482d-4f09-8c3d-ee886685f11f-kube-api-access-dzjtx\") pod \"maas-api-revoke-sub-f518e29f-nt5kd\" (UID: \"ef3b98e5-482d-4f09-8c3d-ee886685f11f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f518e29f-nt5kd"
Oct 31 12:40:37.702512 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:37.699948    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae6f8c0f-87f7-4ad7-85e4-4d69e29ded3b-istio-podinfo\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-8s2sm\" (UID: \"ae6f8c0f-87f7-4ad7-85e4-4d69e29ded3b\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-8s2sm"
Oct 31 12:40:37.702512 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:37.700047    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef3b98e5-482d-4f09-8c3d-ee886685f11f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f518e29f-nt5kd\" (UID: \"ef3b98e5-482d-4f09-8c3d-ee886685f11f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f518e29f-nt5kd"
Oct 31 12:40:37.702512 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:37.700088    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae6f8c0f-87f7-4ad7-85e4-4d69e29ded3b-istio-envoy\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-8s2sm\" (UID: \"ae6f8c0f-87f7-4ad7-85e4-4d69e29ded3b\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-8s2sm"
Oct 31 12:40:37.702512 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:37.700130    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae6f8c0f-87f7-4ad7-85e4-4d69e29ded3b-istio-token\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-8s2sm\" (UID: \"ae6f8c0f-87f7-4ad7-85e4-4d69e29ded3b\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-8s2sm"
Oct 31 12:40:37.702512 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:37.700170    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae6f8c0f-87f7-4ad7-85e4-4d69e29ded3b-istio-data\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-8s2sm\" (UID: \"ae6f8c0f-87f7-4ad7-85e4-4d69e29ded3b\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-8s2sm"
Oct 31 12:40:37.702512 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:37.700205    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhhd8\" (UniqueName: \"kubernetes.io/projected/ae6f8c0f-87f7-4ad7-85e4-4d69e29ded3b-kube-api-access-zhhd8\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-8s2sm\" (UID: \"ae6f8c0f-87f7-4ad7-85e4-4d69e29ded3b\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-8s2sm"
Oct 31 12:40:37.702512 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:37.700307    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae6f8c0f-87f7-4ad7-85e4-4d69e29ded3b-workload-certs\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-8s2sm\" (UID: \"ae6f8c0f-87f7-4ad7-85e4-4d69e29ded3b\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-8s2sm"
Oct 31 12:40:37.702512 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:37.700343    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef3b98e5-482d-4f09-8c3d-ee886685f11f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f518e29f-nt5kd\" (UID: \"ef3b98e5-482d-4f09-8c3d-ee886685f11f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f518e29f-nt5kd"
Oct 31 12:40:37.702512 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:37.700367    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae6f8c0f-87f7-4ad7-85e4-4d69e29ded3b-workload-socket\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-8s2sm\" (UID: \"ae6f8c0f-87f7-4ad7-85e4-4d69e29ded3b\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-8s2sm"
Oct 31 12:40:37.702512 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:37.700392    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae6f8c0f-87f7-4ad7-85e4-4d69e29ded3b-istiod-ca-cert\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-8s2sm\" (UID: \"ae6f8c0f-87f7-4ad7-85e4-4d69e29ded3b\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-8s2sm"
Oct 31 12:40:37.702512 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:37.700427    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae6f8c0f-87f7-4ad7-85e4-4d69e29ded3b-credential-socket\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-8s2sm\" (UID: \"ae6f8c0f-87f7-4ad7-85e4-4d69e29ded3b\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-8s2sm"
Oct 31 12:40:37.702512 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:37.700869    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae6f8c0f-87f7-4ad7-85e4-4d69e29ded3b-credential-socket\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-8s2sm\" (UID: \"ae6f8c0f-87f7-4ad7-85e4-4d69e29ded3b\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-8s2sm"
Oct 31 12:40:37.702512 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:37.700944    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 12:40:37.708772 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:37.704657    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae6f8c0f-87f7-4ad7-85e4-4d69e29ded3b-workload-certs\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-8s2sm\" (UID: \"ae6f8c0f-87f7-4ad7-85e4-4d69e29ded3b\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-8s2sm"
Oct 31 12:40:37.708772 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:37.705332    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae6f8c0f-87f7-4ad7-85e4-4d69e29ded3b-istiod-ca-cert\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-8s2sm\" (UID: \"ae6f8c0f-87f7-4ad7-85e4-4d69e29ded3b\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-8s2sm"
Oct 31 12:40:37.708772 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:37.705997    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f518e29f-nt5kd"]
Oct 31 12:40:37.708772 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:37.707694    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae6f8c0f-87f7-4ad7-85e4-4d69e29ded3b-istio-data\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-8s2sm\" (UID: \"ae6f8c0f-87f7-4ad7-85e4-4d69e29ded3b\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-8s2sm"
Oct 31 12:40:37.714511 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:37.709261    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae6f8c0f-87f7-4ad7-85e4-4d69e29ded3b-workload-socket\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-8s2sm\" (UID: \"ae6f8c0f-87f7-4ad7-85e4-4d69e29ded3b\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-8s2sm"
Oct 31 12:40:37.714511 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:37.710286    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae6f8c0f-87f7-4ad7-85e4-4d69e29ded3b-istio-podinfo\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-8s2sm\" (UID: \"ae6f8c0f-87f7-4ad7-85e4-4d69e29ded3b\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-8s2sm"
Oct 31 12:40:37.714511 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:37.710812    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae6f8c0f-87f7-4ad7-85e4-4d69e29ded3b-istio-envoy\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-8s2sm\" (UID: \"ae6f8c0f-87f7-4ad7-85e4-4d69e29ded3b\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-8s2sm"
Oct 31 12:40:37.724315 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:37.724284    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhhd8\" (UniqueName: \"kubernetes.io/projected/ae6f8c0f-87f7-4ad7-85e4-4d69e29ded3b-kube-api-access-zhhd8\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-8s2sm\" (UID: \"ae6f8c0f-87f7-4ad7-85e4-4d69e29ded3b\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-8s2sm"
Oct 31 12:40:37.732341 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:37.732317    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae6f8c0f-87f7-4ad7-85e4-4d69e29ded3b-istio-token\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-8s2sm\" (UID: \"ae6f8c0f-87f7-4ad7-85e4-4d69e29ded3b\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-8s2sm"
Oct 31 12:40:37.802197 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:37.802160    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef3b98e5-482d-4f09-8c3d-ee886685f11f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f518e29f-nt5kd\" (UID: \"ef3b98e5-482d-4f09-8c3d-ee886685f11f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f518e29f-nt5kd"
Oct 31 12:40:37.802417 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:37.802257    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzjtx\" (UniqueName: \"kubernetes.io/projected/ef3b98e5-482d-4f09-8c3d-ee886685f11f-kube-api-access-dzjtx\") pod \"maas-api-revoke-sub-f518e29f-nt5kd\" (UID: \"ef3b98e5-482d-4f09-8c3d-ee886685f11f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f518e29f-nt5kd"
Oct 31 12:40:37.802417 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:37.802354    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef3b98e5-482d-4f09-8c3d-ee886685f11f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f518e29f-nt5kd\" (UID: \"ef3b98e5-482d-4f09-8c3d-ee886685f11f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f518e29f-nt5kd"
Oct 31 12:40:37.803080 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:37.803049    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef3b98e5-482d-4f09-8c3d-ee886685f11f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f518e29f-nt5kd\" (UID: \"ef3b98e5-482d-4f09-8c3d-ee886685f11f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f518e29f-nt5kd"
Oct 31 12:40:37.805010 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:37.804983    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef3b98e5-482d-4f09-8c3d-ee886685f11f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f518e29f-nt5kd\" (UID: \"ef3b98e5-482d-4f09-8c3d-ee886685f11f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f518e29f-nt5kd"
Oct 31 12:40:37.814351 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:37.814321    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzjtx\" (UniqueName: \"kubernetes.io/projected/ef3b98e5-482d-4f09-8c3d-ee886685f11f-kube-api-access-dzjtx\") pod \"maas-api-revoke-sub-f518e29f-nt5kd\" (UID: \"ef3b98e5-482d-4f09-8c3d-ee886685f11f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f518e29f-nt5kd"
Oct 31 12:40:37.892296 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:37.892261    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-8s2sm"
Oct 31 12:40:38.028207 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:38.028166    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f518e29f-nt5kd"
Oct 31 12:40:38.208227 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:38.208134    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-8s2sm"]
Oct 31 12:40:38.219584 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:38.216990    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:40:38.219584 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:38.217077    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:40:38.219584 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:38.217117    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:40:38.393407 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:38.393364    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f518e29f-nt5kd"]
Oct 31 12:40:38.400954 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:40:38.400922    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef3b98e5_482d_4f09_8c3d_ee886685f11f.slice/crio-ef26911bfbfdf55be5a2b3d191245020b7952bac47a1e805259389d8a8f65673 WatchSource:0}: Error finding container ef26911bfbfdf55be5a2b3d191245020b7952bac47a1e805259389d8a8f65673: Status 404 returned error can't find the container with id ef26911bfbfdf55be5a2b3d191245020b7952bac47a1e805259389d8a8f65673
Oct 31 12:40:38.597427 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:38.597100    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-8s2sm" event={"ID":"ae6f8c0f-87f7-4ad7-85e4-4d69e29ded3b","Type":"ContainerStarted","Data":"2354abb3c178a09bbcf0d1781171038f45e135388fb4eaf3ae392c7cd677ef7a"}
Oct 31 12:40:38.597427 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:38.597155    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-8s2sm" event={"ID":"ae6f8c0f-87f7-4ad7-85e4-4d69e29ded3b","Type":"ContainerStarted","Data":"ba533a83cdc7eaff6671dd3b3e7aa8b2085c529d22d56f5aa016db60c45d7e93"}
Oct 31 12:40:38.599074 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:38.598787    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f518e29f-nt5kd" event={"ID":"ef3b98e5-482d-4f09-8c3d-ee886685f11f","Type":"ContainerStarted","Data":"ef26911bfbfdf55be5a2b3d191245020b7952bac47a1e805259389d8a8f65673"}
Oct 31 12:40:38.600659 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:38.600632    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-sd68f" event={"ID":"4129ab4f-7756-4a6d-8d7f-9dff5d0f1c57","Type":"ContainerStarted","Data":"b93a12d0d99a0df2c3d5d6cf01e704001b745c2d41bb5bb35ff426730500162d"}
Oct 31 12:40:38.627230 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:38.627184    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-8s2sm" podStartSLOduration=1.627165541 podStartE2EDuration="1.627165541s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:40:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:40:38.624871302 +0000 UTC m=+801.727939016" watchObservedRunningTime="2026-10-31 12:40:38.627165541 +0000 UTC m=+801.730233253"
Oct 31 12:40:38.649862 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:38.649801    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-sd68f" podStartSLOduration=2.649782967 podStartE2EDuration="2.649782967s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:40:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:40:38.646944604 +0000 UTC m=+801.750012318" watchObservedRunningTime="2026-10-31 12:40:38.649782967 +0000 UTC m=+801.752850691"
Oct 31 12:40:38.895470 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:38.895437    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-8s2sm"
Oct 31 12:40:39.154200 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:39.154110    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-sd68f"
Oct 31 12:40:39.190515 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:39.186873    5060 patch_prober.go:29] interesting pod/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-sd68f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body=
Oct 31 12:40:39.190515 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:39.186937    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-sd68f" podUID="4129ab4f-7756-4a6d-8d7f-9dff5d0f1c57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused"
Oct 31 12:40:39.630809 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:39.630775    5060 generic.go:356] "Generic (PLEG): container finished" podID="ef3b98e5-482d-4f09-8c3d-ee886685f11f" containerID="6dc01d41f18979e1ae088ac419af21a6e1c2691a0660b25e1a4f2ee98a2496b8" exitCode=0
Oct 31 12:40:39.631082 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:39.631042    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f518e29f-nt5kd" event={"ID":"ef3b98e5-482d-4f09-8c3d-ee886685f11f","Type":"ContainerDied","Data":"6dc01d41f18979e1ae088ac419af21a6e1c2691a0660b25e1a4f2ee98a2496b8"}
Oct 31 12:40:39.894680 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:39.894626    5060 patch_prober.go:29] interesting pod/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-8s2sm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body=
Oct 31 12:40:39.894990 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:39.894949    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-8s2sm" podUID="ae6f8c0f-87f7-4ad7-85e4-4d69e29ded3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused"
Oct 31 12:40:40.153998 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:40.153920    5060 patch_prober.go:29] interesting pod/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-sd68f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body=
Oct 31 12:40:40.153998 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:40.153974    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-sd68f" podUID="4129ab4f-7756-4a6d-8d7f-9dff5d0f1c57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused"
Oct 31 12:40:40.902569 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:40.901887    5060 patch_prober.go:29] interesting pod/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-8s2sm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body=
Oct 31 12:40:40.902569 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:40.901961    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-8s2sm" podUID="ae6f8c0f-87f7-4ad7-85e4-4d69e29ded3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused"
Oct 31 12:40:41.026776 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:41.026470    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f518e29f-nt5kd"
Oct 31 12:40:41.153988 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:41.153895    5060 patch_prober.go:29] interesting pod/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-sd68f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body=
Oct 31 12:40:41.154248 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:41.154214    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-sd68f" podUID="4129ab4f-7756-4a6d-8d7f-9dff5d0f1c57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused"
Oct 31 12:40:41.223513 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:41.198885    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ef3b98e5-482d-4f09-8c3d-ee886685f11f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef3b98e5-482d-4f09-8c3d-ee886685f11f-maas-api-xks-ca\") pod \"ef3b98e5-482d-4f09-8c3d-ee886685f11f\" (UID: \"ef3b98e5-482d-4f09-8c3d-ee886685f11f\") "
Oct 31 12:40:41.228433 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:41.228381    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef3b98e5-482d-4f09-8c3d-ee886685f11f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef3b98e5-482d-4f09-8c3d-ee886685f11f-maas-api-openshift-ca\") pod \"ef3b98e5-482d-4f09-8c3d-ee886685f11f\" (UID: \"ef3b98e5-482d-4f09-8c3d-ee886685f11f\") "
Oct 31 12:40:41.228814 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:41.228791    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef3b98e5-482d-4f09-8c3d-ee886685f11f-kube-api-access-dzjtx\" (UniqueName: \"kubernetes.io/projected/ef3b98e5-482d-4f09-8c3d-ee886685f11f-kube-api-access-dzjtx\") pod \"ef3b98e5-482d-4f09-8c3d-ee886685f11f\" (UID: \"ef3b98e5-482d-4f09-8c3d-ee886685f11f\") "
Oct 31 12:40:41.230616 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:41.230591    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef3b98e5-482d-4f09-8c3d-ee886685f11f-maas-api-openshift-ca" pod "ef3b98e5-482d-4f09-8c3d-ee886685f11f" (UID: "ef3b98e5-482d-4f09-8c3d-ee886685f11f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:40:41.234099 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:41.234062    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef3b98e5-482d-4f09-8c3d-ee886685f11f-kube-api-access-dzjtx" pod "ef3b98e5-482d-4f09-8c3d-ee886685f11f" (UID: "ef3b98e5-482d-4f09-8c3d-ee886685f11f"). InnerVolumeSpecName "kube-api-access-dzjtx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:40:41.235156 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:41.235131    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef3b98e5-482d-4f09-8c3d-ee886685f11f-maas-api-xks-ca" pod "ef3b98e5-482d-4f09-8c3d-ee886685f11f" (UID: "ef3b98e5-482d-4f09-8c3d-ee886685f11f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:40:41.331739 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:41.331690    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef3b98e5-482d-4f09-8c3d-ee886685f11f-maas-api-xks-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:40:41.331739 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:41.331733    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef3b98e5-482d-4f09-8c3d-ee886685f11f-maas-api-openshift-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:40:41.331940 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:41.331749    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dzjtx\" (UniqueName: \"kubernetes.io/projected/ef3b98e5-482d-4f09-8c3d-ee886685f11f-kube-api-access-dzjtx\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:40:41.653740 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:41.653706    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f518e29f-nt5kd"
Oct 31 12:40:41.653905 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:41.653744    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f518e29f-nt5kd" event={"ID":"ef3b98e5-482d-4f09-8c3d-ee886685f11f","Type":"ContainerDied","Data":"ef26911bfbfdf55be5a2b3d191245020b7952bac47a1e805259389d8a8f65673"}
Oct 31 12:40:41.653905 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:41.653796    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ef26911bfbfdf55be5a2b3d191245020b7952bac47a1e805259389d8a8f65673"
Oct 31 12:40:41.899516 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:41.898929    5060 patch_prober.go:29] interesting pod/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-8s2sm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body=
Oct 31 12:40:41.899516 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:41.899000    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-8s2sm" podUID="ae6f8c0f-87f7-4ad7-85e4-4d69e29ded3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused"
Oct 31 12:40:42.154108 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:42.154073    5060 patch_prober.go:29] interesting pod/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-sd68f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body=
Oct 31 12:40:42.154313 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:42.154135    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-sd68f" podUID="4129ab4f-7756-4a6d-8d7f-9dff5d0f1c57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused"
Oct 31 12:40:42.892936 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:42.892894    5060 patch_prober.go:29] interesting pod/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-8s2sm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body=
Oct 31 12:40:42.893226 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:42.893198    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-8s2sm" podUID="ae6f8c0f-87f7-4ad7-85e4-4d69e29ded3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused"
Oct 31 12:40:43.154463 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:43.154383    5060 patch_prober.go:29] interesting pod/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-sd68f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body=
Oct 31 12:40:43.154463 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:43.154444    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-sd68f" podUID="4129ab4f-7756-4a6d-8d7f-9dff5d0f1c57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused"
Oct 31 12:40:43.892887 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:43.892853    5060 patch_prober.go:29] interesting pod/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-8s2sm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body=
Oct 31 12:40:43.893076 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:43.892906    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-8s2sm" podUID="ae6f8c0f-87f7-4ad7-85e4-4d69e29ded3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused"
Oct 31 12:40:44.154297 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:44.154198    5060 patch_prober.go:29] interesting pod/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-sd68f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body=
Oct 31 12:40:44.154297 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:44.154284    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-sd68f" podUID="4129ab4f-7756-4a6d-8d7f-9dff5d0f1c57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused"
Oct 31 12:40:44.893692 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:44.893656    5060 patch_prober.go:29] interesting pod/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-8s2sm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body=
Oct 31 12:40:44.894098 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:44.893716    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-8s2sm" podUID="ae6f8c0f-87f7-4ad7-85e4-4d69e29ded3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused"
Oct 31 12:40:45.158690 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:45.158606    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-sd68f"
Oct 31 12:40:45.159179 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:45.159084    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-sd68f"
Oct 31 12:40:45.160028 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:45.160004    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-sd68f"
Oct 31 12:40:45.897785 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:45.897751    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-8s2sm"
Oct 31 12:40:45.898269 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:45.898115    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-8s2sm"
Oct 31 12:40:45.898860 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:45.898840    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-8s2sm"
Oct 31 12:40:49.120556 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:49.120522    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a4565dd-gcdht"]
Oct 31 12:40:49.121179 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:49.121163    5060 cpu_manager.go:414] "containerMap: removing container" podUID="ef3b98e5-482d-4f09-8c3d-ee886685f11f" containerName="revoke-keys"
Oct 31 12:40:49.121225 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:49.121180    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef3b98e5-482d-4f09-8c3d-ee886685f11f" containerName="revoke-keys"
Oct 31 12:40:49.121297 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:49.121289    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef3b98e5-482d-4f09-8c3d-ee886685f11f" containerName="revoke-keys"
Oct 31 12:40:49.125445 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:49.125422    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a4565dd-gcdht"
Oct 31 12:40:49.143912 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:49.143887    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 12:40:49.144024 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:49.143946    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 12:40:49.144273 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:49.144253    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 12:40:49.144386 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:49.144332    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7wsmb\""
Oct 31 12:40:49.148898 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:49.148873    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a4565dd-gcdht"]
Oct 31 12:40:49.281164 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:49.281126    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f2bbf07-a93d-4238-a69a-3530a1cbe1e2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0a4565dd-gcdht\" (UID: \"0f2bbf07-a93d-4238-a69a-3530a1cbe1e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a4565dd-gcdht"
Oct 31 12:40:49.281164 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:49.281172    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f2bbf07-a93d-4238-a69a-3530a1cbe1e2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0a4565dd-gcdht\" (UID: \"0f2bbf07-a93d-4238-a69a-3530a1cbe1e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a4565dd-gcdht"
Oct 31 12:40:49.281400 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:49.281339    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cl6vm\" (UniqueName: \"kubernetes.io/projected/0f2bbf07-a93d-4238-a69a-3530a1cbe1e2-kube-api-access-cl6vm\") pod \"maas-api-revoke-sub-0a4565dd-gcdht\" (UID: \"0f2bbf07-a93d-4238-a69a-3530a1cbe1e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a4565dd-gcdht"
Oct 31 12:40:49.383074 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:49.382978    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f2bbf07-a93d-4238-a69a-3530a1cbe1e2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0a4565dd-gcdht\" (UID: \"0f2bbf07-a93d-4238-a69a-3530a1cbe1e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a4565dd-gcdht"
Oct 31 12:40:49.383074 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:49.383026    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f2bbf07-a93d-4238-a69a-3530a1cbe1e2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0a4565dd-gcdht\" (UID: \"0f2bbf07-a93d-4238-a69a-3530a1cbe1e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a4565dd-gcdht"
Oct 31 12:40:49.383314 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:49.383159    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cl6vm\" (UniqueName: \"kubernetes.io/projected/0f2bbf07-a93d-4238-a69a-3530a1cbe1e2-kube-api-access-cl6vm\") pod \"maas-api-revoke-sub-0a4565dd-gcdht\" (UID: \"0f2bbf07-a93d-4238-a69a-3530a1cbe1e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a4565dd-gcdht"
Oct 31 12:40:49.383826 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:49.383795    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f2bbf07-a93d-4238-a69a-3530a1cbe1e2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0a4565dd-gcdht\" (UID: \"0f2bbf07-a93d-4238-a69a-3530a1cbe1e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a4565dd-gcdht"
Oct 31 12:40:49.385623 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:49.385593    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f2bbf07-a93d-4238-a69a-3530a1cbe1e2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0a4565dd-gcdht\" (UID: \"0f2bbf07-a93d-4238-a69a-3530a1cbe1e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a4565dd-gcdht"
Oct 31 12:40:49.396331 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:49.396292    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cl6vm\" (UniqueName: \"kubernetes.io/projected/0f2bbf07-a93d-4238-a69a-3530a1cbe1e2-kube-api-access-cl6vm\") pod \"maas-api-revoke-sub-0a4565dd-gcdht\" (UID: \"0f2bbf07-a93d-4238-a69a-3530a1cbe1e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a4565dd-gcdht"
Oct 31 12:40:49.439053 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:49.439018    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a4565dd-gcdht"
Oct 31 12:40:49.725140 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:49.725105    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a4565dd-gcdht"]
Oct 31 12:40:49.725676 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:40:49.725649    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f2bbf07_a93d_4238_a69a_3530a1cbe1e2.slice/crio-119c824a9be624b7fe3c47db72f9d2a6b163051ec355a8d479e5ff088b58959b WatchSource:0}: Error finding container 119c824a9be624b7fe3c47db72f9d2a6b163051ec355a8d479e5ff088b58959b: Status 404 returned error can't find the container with id 119c824a9be624b7fe3c47db72f9d2a6b163051ec355a8d479e5ff088b58959b
Oct 31 12:40:49.795707 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:49.795670    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a4565dd-gcdht" event={"ID":"0f2bbf07-a93d-4238-a69a-3530a1cbe1e2","Type":"ContainerStarted","Data":"119c824a9be624b7fe3c47db72f9d2a6b163051ec355a8d479e5ff088b58959b"}
Oct 31 12:40:50.806390 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:50.806354    5060 generic.go:356] "Generic (PLEG): container finished" podID="0f2bbf07-a93d-4238-a69a-3530a1cbe1e2" containerID="d4df205b92185a7af1fe7d1d0aa370404833b9ec7b2632649dab183491e6d297" exitCode=0
Oct 31 12:40:50.806869 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:50.806443    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a4565dd-gcdht" event={"ID":"0f2bbf07-a93d-4238-a69a-3530a1cbe1e2","Type":"ContainerDied","Data":"d4df205b92185a7af1fe7d1d0aa370404833b9ec7b2632649dab183491e6d297"}
Oct 31 12:40:51.956693 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:51.956661    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-b9brm"]
Oct 31 12:40:51.988322 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:51.988293    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-b9brm"
Oct 31 12:40:52.040342 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:52.040302    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-b9brm"]
Oct 31 12:40:52.057611 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:52.044064    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83855e68-24ce-41e1-97be-79df68bc9505-credential-socket\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-b9brm\" (UID: \"83855e68-24ce-41e1-97be-79df68bc9505\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-b9brm"
Oct 31 12:40:52.057764 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:52.057658    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83855e68-24ce-41e1-97be-79df68bc9505-istio-token\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-b9brm\" (UID: \"83855e68-24ce-41e1-97be-79df68bc9505\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-b9brm"
Oct 31 12:40:52.057764 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:52.057716    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83855e68-24ce-41e1-97be-79df68bc9505-workload-certs\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-b9brm\" (UID: \"83855e68-24ce-41e1-97be-79df68bc9505\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-b9brm"
Oct 31 12:40:52.057890 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:52.057789    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83855e68-24ce-41e1-97be-79df68bc9505-workload-socket\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-b9brm\" (UID: \"83855e68-24ce-41e1-97be-79df68bc9505\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-b9brm"
Oct 31 12:40:52.057890 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:52.057817    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83855e68-24ce-41e1-97be-79df68bc9505-istiod-ca-cert\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-b9brm\" (UID: \"83855e68-24ce-41e1-97be-79df68bc9505\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-b9brm"
Oct 31 12:40:52.057890 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:52.057871    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83855e68-24ce-41e1-97be-79df68bc9505-istio-envoy\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-b9brm\" (UID: \"83855e68-24ce-41e1-97be-79df68bc9505\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-b9brm"
Oct 31 12:40:52.058043 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:52.057914    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83855e68-24ce-41e1-97be-79df68bc9505-istio-data\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-b9brm\" (UID: \"83855e68-24ce-41e1-97be-79df68bc9505\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-b9brm"
Oct 31 12:40:52.058043 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:52.057954    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83855e68-24ce-41e1-97be-79df68bc9505-istio-podinfo\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-b9brm\" (UID: \"83855e68-24ce-41e1-97be-79df68bc9505\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-b9brm"
Oct 31 12:40:52.058137 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:52.058069    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6r5r\" (UniqueName: \"kubernetes.io/projected/83855e68-24ce-41e1-97be-79df68bc9505-kube-api-access-p6r5r\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-b9brm\" (UID: \"83855e68-24ce-41e1-97be-79df68bc9505\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-b9brm"
Oct 31 12:40:52.074743 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:52.074673    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a4565dd-gcdht"
Oct 31 12:40:52.158842 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:52.158809    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f2bbf07-a93d-4238-a69a-3530a1cbe1e2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f2bbf07-a93d-4238-a69a-3530a1cbe1e2-maas-api-xks-ca\") pod \"0f2bbf07-a93d-4238-a69a-3530a1cbe1e2\" (UID: \"0f2bbf07-a93d-4238-a69a-3530a1cbe1e2\") "
Oct 31 12:40:52.159017 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:52.158931    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f2bbf07-a93d-4238-a69a-3530a1cbe1e2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f2bbf07-a93d-4238-a69a-3530a1cbe1e2-maas-api-openshift-ca\") pod \"0f2bbf07-a93d-4238-a69a-3530a1cbe1e2\" (UID: \"0f2bbf07-a93d-4238-a69a-3530a1cbe1e2\") "
Oct 31 12:40:52.159017 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:52.158961    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f2bbf07-a93d-4238-a69a-3530a1cbe1e2-kube-api-access-cl6vm\" (UniqueName: \"kubernetes.io/projected/0f2bbf07-a93d-4238-a69a-3530a1cbe1e2-kube-api-access-cl6vm\") pod \"0f2bbf07-a93d-4238-a69a-3530a1cbe1e2\" (UID: \"0f2bbf07-a93d-4238-a69a-3530a1cbe1e2\") "
Oct 31 12:40:52.159175 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:52.159156    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83855e68-24ce-41e1-97be-79df68bc9505-workload-socket\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-b9brm\" (UID: \"83855e68-24ce-41e1-97be-79df68bc9505\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-b9brm"
Oct 31 12:40:52.159243 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:52.159182    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83855e68-24ce-41e1-97be-79df68bc9505-istiod-ca-cert\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-b9brm\" (UID: \"83855e68-24ce-41e1-97be-79df68bc9505\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-b9brm"
Oct 31 12:40:52.159243 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:52.159210    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83855e68-24ce-41e1-97be-79df68bc9505-istio-envoy\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-b9brm\" (UID: \"83855e68-24ce-41e1-97be-79df68bc9505\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-b9brm"
Oct 31 12:40:52.159243 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:52.159240    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83855e68-24ce-41e1-97be-79df68bc9505-istio-data\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-b9brm\" (UID: \"83855e68-24ce-41e1-97be-79df68bc9505\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-b9brm"
Oct 31 12:40:52.159399 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:52.159269    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83855e68-24ce-41e1-97be-79df68bc9505-istio-podinfo\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-b9brm\" (UID: \"83855e68-24ce-41e1-97be-79df68bc9505\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-b9brm"
Oct 31 12:40:52.159399 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:52.159355    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6r5r\" (UniqueName: \"kubernetes.io/projected/83855e68-24ce-41e1-97be-79df68bc9505-kube-api-access-p6r5r\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-b9brm\" (UID: \"83855e68-24ce-41e1-97be-79df68bc9505\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-b9brm"
Oct 31 12:40:52.159545 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:52.159526    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83855e68-24ce-41e1-97be-79df68bc9505-credential-socket\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-b9brm\" (UID: \"83855e68-24ce-41e1-97be-79df68bc9505\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-b9brm"
Oct 31 12:40:52.159611 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:52.159558    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83855e68-24ce-41e1-97be-79df68bc9505-istio-token\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-b9brm\" (UID: \"83855e68-24ce-41e1-97be-79df68bc9505\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-b9brm"
Oct 31 12:40:52.159664 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:52.159616    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83855e68-24ce-41e1-97be-79df68bc9505-workload-certs\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-b9brm\" (UID: \"83855e68-24ce-41e1-97be-79df68bc9505\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-b9brm"
Oct 31 12:40:52.159744 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:52.159721    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83855e68-24ce-41e1-97be-79df68bc9505-workload-socket\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-b9brm\" (UID: \"83855e68-24ce-41e1-97be-79df68bc9505\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-b9brm"
Oct 31 12:40:52.159814 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:52.159786    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83855e68-24ce-41e1-97be-79df68bc9505-istio-data\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-b9brm\" (UID: \"83855e68-24ce-41e1-97be-79df68bc9505\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-b9brm"
Oct 31 12:40:52.159951 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:52.159927    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83855e68-24ce-41e1-97be-79df68bc9505-workload-certs\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-b9brm\" (UID: \"83855e68-24ce-41e1-97be-79df68bc9505\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-b9brm"
Oct 31 12:40:52.160122 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:52.160062    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83855e68-24ce-41e1-97be-79df68bc9505-credential-socket\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-b9brm\" (UID: \"83855e68-24ce-41e1-97be-79df68bc9505\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-b9brm"
Oct 31 12:40:52.160122 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:52.159352    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f2bbf07-a93d-4238-a69a-3530a1cbe1e2-maas-api-openshift-ca" pod "0f2bbf07-a93d-4238-a69a-3530a1cbe1e2" (UID: "0f2bbf07-a93d-4238-a69a-3530a1cbe1e2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:40:52.160596 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:52.160570    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83855e68-24ce-41e1-97be-79df68bc9505-istiod-ca-cert\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-b9brm\" (UID: \"83855e68-24ce-41e1-97be-79df68bc9505\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-b9brm"
Oct 31 12:40:52.162100 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:52.162071    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f2bbf07-a93d-4238-a69a-3530a1cbe1e2-maas-api-xks-ca" pod "0f2bbf07-a93d-4238-a69a-3530a1cbe1e2" (UID: "0f2bbf07-a93d-4238-a69a-3530a1cbe1e2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:40:52.162408 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:52.162381    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83855e68-24ce-41e1-97be-79df68bc9505-istio-envoy\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-b9brm\" (UID: \"83855e68-24ce-41e1-97be-79df68bc9505\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-b9brm"
Oct 31 12:40:52.162692 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:52.162673    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f2bbf07-a93d-4238-a69a-3530a1cbe1e2-kube-api-access-cl6vm" pod "0f2bbf07-a93d-4238-a69a-3530a1cbe1e2" (UID: "0f2bbf07-a93d-4238-a69a-3530a1cbe1e2"). InnerVolumeSpecName "kube-api-access-cl6vm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:40:52.163270 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:52.163248    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83855e68-24ce-41e1-97be-79df68bc9505-istio-podinfo\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-b9brm\" (UID: \"83855e68-24ce-41e1-97be-79df68bc9505\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-b9brm"
Oct 31 12:40:52.171982 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:52.171953    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83855e68-24ce-41e1-97be-79df68bc9505-istio-token\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-b9brm\" (UID: \"83855e68-24ce-41e1-97be-79df68bc9505\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-b9brm"
Oct 31 12:40:52.172176 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:52.172158    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6r5r\" (UniqueName: \"kubernetes.io/projected/83855e68-24ce-41e1-97be-79df68bc9505-kube-api-access-p6r5r\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-b9brm\" (UID: \"83855e68-24ce-41e1-97be-79df68bc9505\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-b9brm"
Oct 31 12:40:52.260954 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:52.260922    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f2bbf07-a93d-4238-a69a-3530a1cbe1e2-maas-api-xks-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:40:52.260954 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:52.260951    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f2bbf07-a93d-4238-a69a-3530a1cbe1e2-maas-api-openshift-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:40:52.260954 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:52.260962    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cl6vm\" (UniqueName: \"kubernetes.io/projected/0f2bbf07-a93d-4238-a69a-3530a1cbe1e2-kube-api-access-cl6vm\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:40:52.371106 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:52.371025    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-b9brm"
Oct 31 12:40:52.479824 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:52.478930    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-sv4w2"]
Oct 31 12:40:52.479824 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:52.479815    5060 cpu_manager.go:414] "containerMap: removing container" podUID="0f2bbf07-a93d-4238-a69a-3530a1cbe1e2" containerName="revoke-keys"
Oct 31 12:40:52.480060 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:52.479836    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f2bbf07-a93d-4238-a69a-3530a1cbe1e2" containerName="revoke-keys"
Oct 31 12:40:52.480060 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:52.479994    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f2bbf07-a93d-4238-a69a-3530a1cbe1e2" containerName="revoke-keys"
Oct 31 12:40:52.500806 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:52.500770    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-sv4w2"
Oct 31 12:40:52.504295 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:52.504257    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-sv4w2"]
Oct 31 12:40:52.520168 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:52.519862    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kj56r"]
Oct 31 12:40:52.525578 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:52.525554    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kj56r"
Oct 31 12:40:52.531356 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:52.531335    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Oct 31 12:40:52.531684 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:52.531668    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-mvht9\""
Oct 31 12:40:52.532112 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:52.532094    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Oct 31 12:40:52.535587 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:52.532158    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Oct 31 12:40:52.545121 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:52.545054    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kj56r"]
Oct 31 12:40:52.567096 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:52.567064    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/120c71cc-6b6d-4833-91a1-52bc36d759e8-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kj56r\" (UID: \"120c71cc-6b6d-4833-91a1-52bc36d759e8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kj56r"
Oct 31 12:40:52.567222 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:52.567105    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38bdca08-eb2b-43c5-8c07-7059e5c9e199-istio-podinfo\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-sv4w2\" (UID: \"38bdca08-eb2b-43c5-8c07-7059e5c9e199\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-sv4w2"
Oct 31 12:40:52.567222 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:52.567127    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/120c71cc-6b6d-4833-91a1-52bc36d759e8-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kj56r\" (UID: \"120c71cc-6b6d-4833-91a1-52bc36d759e8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kj56r"
Oct 31 12:40:52.567222 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:52.567153    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/120c71cc-6b6d-4833-91a1-52bc36d759e8-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kj56r\" (UID: \"120c71cc-6b6d-4833-91a1-52bc36d759e8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kj56r"
Oct 31 12:40:52.567222 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:52.567170    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/120c71cc-6b6d-4833-91a1-52bc36d759e8-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kj56r\" (UID: \"120c71cc-6b6d-4833-91a1-52bc36d759e8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kj56r"
Oct 31 12:40:52.567222 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:52.567203    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqgl6\" (UniqueName: \"kubernetes.io/projected/38bdca08-eb2b-43c5-8c07-7059e5c9e199-kube-api-access-nqgl6\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-sv4w2\" (UID: \"38bdca08-eb2b-43c5-8c07-7059e5c9e199\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-sv4w2"
Oct 31 12:40:52.567527 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:52.567228    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzhxn\" (UniqueName: \"kubernetes.io/projected/120c71cc-6b6d-4833-91a1-52bc36d759e8-kube-api-access-qzhxn\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kj56r\" (UID: \"120c71cc-6b6d-4833-91a1-52bc36d759e8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kj56r"
Oct 31 12:40:52.567527 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:52.567244    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38bdca08-eb2b-43c5-8c07-7059e5c9e199-istio-data\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-sv4w2\" (UID: \"38bdca08-eb2b-43c5-8c07-7059e5c9e199\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-sv4w2"
Oct 31 12:40:52.567527 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:52.567285    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38bdca08-eb2b-43c5-8c07-7059e5c9e199-workload-socket\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-sv4w2\" (UID: \"38bdca08-eb2b-43c5-8c07-7059e5c9e199\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-sv4w2"
Oct 31 12:40:52.567527 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:52.567301    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/120c71cc-6b6d-4833-91a1-52bc36d759e8-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kj56r\" (UID: \"120c71cc-6b6d-4833-91a1-52bc36d759e8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kj56r"
Oct 31 12:40:52.567527 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:52.567318    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38bdca08-eb2b-43c5-8c07-7059e5c9e199-istio-token\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-sv4w2\" (UID: \"38bdca08-eb2b-43c5-8c07-7059e5c9e199\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-sv4w2"
Oct 31 12:40:52.567527 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:52.567357    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38bdca08-eb2b-43c5-8c07-7059e5c9e199-istiod-ca-cert\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-sv4w2\" (UID: \"38bdca08-eb2b-43c5-8c07-7059e5c9e199\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-sv4w2"
Oct 31 12:40:52.567527 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:52.567394    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/120c71cc-6b6d-4833-91a1-52bc36d759e8-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kj56r\" (UID: \"120c71cc-6b6d-4833-91a1-52bc36d759e8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kj56r"
Oct 31 12:40:52.567527 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:52.567415    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38bdca08-eb2b-43c5-8c07-7059e5c9e199-workload-certs\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-sv4w2\" (UID: \"38bdca08-eb2b-43c5-8c07-7059e5c9e199\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-sv4w2"
Oct 31 12:40:52.567527 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:52.567447    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38bdca08-eb2b-43c5-8c07-7059e5c9e199-credential-socket\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-sv4w2\" (UID: \"38bdca08-eb2b-43c5-8c07-7059e5c9e199\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-sv4w2"
Oct 31 12:40:52.567527 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:52.567465    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38bdca08-eb2b-43c5-8c07-7059e5c9e199-istio-envoy\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-sv4w2\" (UID: \"38bdca08-eb2b-43c5-8c07-7059e5c9e199\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-sv4w2"
Oct 31 12:40:52.669020 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:52.668931    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38bdca08-eb2b-43c5-8c07-7059e5c9e199-credential-socket\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-sv4w2\" (UID: \"38bdca08-eb2b-43c5-8c07-7059e5c9e199\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-sv4w2"
Oct 31 12:40:52.669020 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:52.668983    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38bdca08-eb2b-43c5-8c07-7059e5c9e199-istio-envoy\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-sv4w2\" (UID: \"38bdca08-eb2b-43c5-8c07-7059e5c9e199\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-sv4w2"
Oct 31 12:40:52.669263 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:52.669060    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/120c71cc-6b6d-4833-91a1-52bc36d759e8-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kj56r\" (UID: \"120c71cc-6b6d-4833-91a1-52bc36d759e8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kj56r"
Oct 31 12:40:52.669263 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:52.669082    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38bdca08-eb2b-43c5-8c07-7059e5c9e199-istio-podinfo\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-sv4w2\" (UID: \"38bdca08-eb2b-43c5-8c07-7059e5c9e199\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-sv4w2"
Oct 31 12:40:52.669263 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:52.669102    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/120c71cc-6b6d-4833-91a1-52bc36d759e8-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kj56r\" (UID: \"120c71cc-6b6d-4833-91a1-52bc36d759e8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kj56r"
Oct 31 12:40:52.669263 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:52.669133    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/120c71cc-6b6d-4833-91a1-52bc36d759e8-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kj56r\" (UID: \"120c71cc-6b6d-4833-91a1-52bc36d759e8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kj56r"
Oct 31 12:40:52.669263 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:52.669151    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/120c71cc-6b6d-4833-91a1-52bc36d759e8-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kj56r\" (UID: \"120c71cc-6b6d-4833-91a1-52bc36d759e8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kj56r"
Oct 31 12:40:52.669263 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:52.669184    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqgl6\" (UniqueName: \"kubernetes.io/projected/38bdca08-eb2b-43c5-8c07-7059e5c9e199-kube-api-access-nqgl6\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-sv4w2\" (UID: \"38bdca08-eb2b-43c5-8c07-7059e5c9e199\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-sv4w2"
Oct 31 12:40:52.669263 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:52.669213    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzhxn\" (UniqueName: \"kubernetes.io/projected/120c71cc-6b6d-4833-91a1-52bc36d759e8-kube-api-access-qzhxn\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kj56r\" (UID: \"120c71cc-6b6d-4833-91a1-52bc36d759e8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kj56r"
Oct 31 12:40:52.669263 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:52.669231    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38bdca08-eb2b-43c5-8c07-7059e5c9e199-istio-data\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-sv4w2\" (UID: \"38bdca08-eb2b-43c5-8c07-7059e5c9e199\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-sv4w2"
Oct 31 12:40:52.669748 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:52.669274    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38bdca08-eb2b-43c5-8c07-7059e5c9e199-workload-socket\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-sv4w2\" (UID: \"38bdca08-eb2b-43c5-8c07-7059e5c9e199\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-sv4w2"
Oct 31 12:40:52.669748 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:52.669293    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/120c71cc-6b6d-4833-91a1-52bc36d759e8-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kj56r\" (UID: \"120c71cc-6b6d-4833-91a1-52bc36d759e8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kj56r"
Oct 31 12:40:52.669748 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:52.669312    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38bdca08-eb2b-43c5-8c07-7059e5c9e199-istio-token\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-sv4w2\" (UID: \"38bdca08-eb2b-43c5-8c07-7059e5c9e199\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-sv4w2"
Oct 31 12:40:52.669748 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:52.669354    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38bdca08-eb2b-43c5-8c07-7059e5c9e199-istiod-ca-cert\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-sv4w2\" (UID: \"38bdca08-eb2b-43c5-8c07-7059e5c9e199\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-sv4w2"
Oct 31 12:40:52.669748 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:52.669393    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/120c71cc-6b6d-4833-91a1-52bc36d759e8-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kj56r\" (UID: \"120c71cc-6b6d-4833-91a1-52bc36d759e8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kj56r"
Oct 31 12:40:52.669748 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:52.669414    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38bdca08-eb2b-43c5-8c07-7059e5c9e199-workload-certs\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-sv4w2\" (UID: \"38bdca08-eb2b-43c5-8c07-7059e5c9e199\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-sv4w2"
Oct 31 12:40:52.670056 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:52.669783    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38bdca08-eb2b-43c5-8c07-7059e5c9e199-workload-certs\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-sv4w2\" (UID: \"38bdca08-eb2b-43c5-8c07-7059e5c9e199\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-sv4w2"
Oct 31 12:40:52.671380 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:52.671361    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38bdca08-eb2b-43c5-8c07-7059e5c9e199-credential-socket\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-sv4w2\" (UID: \"38bdca08-eb2b-43c5-8c07-7059e5c9e199\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-sv4w2"
Oct 31 12:40:52.674582 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:52.674556    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/120c71cc-6b6d-4833-91a1-52bc36d759e8-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kj56r\" (UID: \"120c71cc-6b6d-4833-91a1-52bc36d759e8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kj56r"
Oct 31 12:40:52.675129 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:52.675098    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38bdca08-eb2b-43c5-8c07-7059e5c9e199-istio-data\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-sv4w2\" (UID: \"38bdca08-eb2b-43c5-8c07-7059e5c9e199\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-sv4w2"
Oct 31 12:40:52.675266 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:52.675236    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/120c71cc-6b6d-4833-91a1-52bc36d759e8-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kj56r\" (UID: \"120c71cc-6b6d-4833-91a1-52bc36d759e8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kj56r"
Oct 31 12:40:52.675988 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:52.675960    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38bdca08-eb2b-43c5-8c07-7059e5c9e199-workload-socket\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-sv4w2\" (UID: \"38bdca08-eb2b-43c5-8c07-7059e5c9e199\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-sv4w2"
Oct 31 12:40:52.676537 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:52.676484    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38bdca08-eb2b-43c5-8c07-7059e5c9e199-istiod-ca-cert\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-sv4w2\" (UID: \"38bdca08-eb2b-43c5-8c07-7059e5c9e199\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-sv4w2"
Oct 31 12:40:52.677150 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:52.677118    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-b9brm"]
Oct 31 12:40:52.679259 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:52.678023    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/120c71cc-6b6d-4833-91a1-52bc36d759e8-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kj56r\" (UID: \"120c71cc-6b6d-4833-91a1-52bc36d759e8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kj56r"
Oct 31 12:40:52.679259 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:52.678265    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38bdca08-eb2b-43c5-8c07-7059e5c9e199-istio-envoy\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-sv4w2\" (UID: \"38bdca08-eb2b-43c5-8c07-7059e5c9e199\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-sv4w2"
Oct 31 12:40:52.679259 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:52.678325    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/120c71cc-6b6d-4833-91a1-52bc36d759e8-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kj56r\" (UID: \"120c71cc-6b6d-4833-91a1-52bc36d759e8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kj56r"
Oct 31 12:40:52.679833 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:52.679440    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/120c71cc-6b6d-4833-91a1-52bc36d759e8-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kj56r\" (UID: \"120c71cc-6b6d-4833-91a1-52bc36d759e8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kj56r"
Oct 31 12:40:52.679833 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:52.679432    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:40:52.679833 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:52.679562    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:40:52.679833 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:52.679611    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:40:52.680338 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:52.680312    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38bdca08-eb2b-43c5-8c07-7059e5c9e199-istio-podinfo\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-sv4w2\" (UID: \"38bdca08-eb2b-43c5-8c07-7059e5c9e199\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-sv4w2"
Oct 31 12:40:52.686657 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:52.686629    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqgl6\" (UniqueName: \"kubernetes.io/projected/38bdca08-eb2b-43c5-8c07-7059e5c9e199-kube-api-access-nqgl6\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-sv4w2\" (UID: \"38bdca08-eb2b-43c5-8c07-7059e5c9e199\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-sv4w2"
Oct 31 12:40:52.697591 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:52.697553    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38bdca08-eb2b-43c5-8c07-7059e5c9e199-istio-token\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-sv4w2\" (UID: \"38bdca08-eb2b-43c5-8c07-7059e5c9e199\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-sv4w2"
Oct 31 12:40:52.698223 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:52.698202    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzhxn\" (UniqueName: \"kubernetes.io/projected/120c71cc-6b6d-4833-91a1-52bc36d759e8-kube-api-access-qzhxn\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kj56r\" (UID: \"120c71cc-6b6d-4833-91a1-52bc36d759e8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kj56r"
Oct 31 12:40:52.699028 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:52.698990    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/120c71cc-6b6d-4833-91a1-52bc36d759e8-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kj56r\" (UID: \"120c71cc-6b6d-4833-91a1-52bc36d759e8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kj56r"
Oct 31 12:40:52.825637 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:52.825593    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-sv4w2"
Oct 31 12:40:52.831425 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:52.831387    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-b9brm" event={"ID":"83855e68-24ce-41e1-97be-79df68bc9505","Type":"ContainerStarted","Data":"936c8c9326eed056654b98a9642430bc812b1b735c87dc6e38b8a4dc66023643"}
Oct 31 12:40:52.831605 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:52.831450    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-b9brm" event={"ID":"83855e68-24ce-41e1-97be-79df68bc9505","Type":"ContainerStarted","Data":"3a1859c37b15e081f389254ee8ac15727489369144b780c33abfef157c003fa8"}
Oct 31 12:40:52.834876 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:52.834250    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a4565dd-gcdht" event={"ID":"0f2bbf07-a93d-4238-a69a-3530a1cbe1e2","Type":"ContainerDied","Data":"119c824a9be624b7fe3c47db72f9d2a6b163051ec355a8d479e5ff088b58959b"}
Oct 31 12:40:52.834876 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:52.834299    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="119c824a9be624b7fe3c47db72f9d2a6b163051ec355a8d479e5ff088b58959b"
Oct 31 12:40:52.834876 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:52.834359    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a4565dd-gcdht"
Oct 31 12:40:52.856371 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:52.856342    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kj56r"
Oct 31 12:40:52.867940 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:52.867880    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-b9brm" podStartSLOduration=1.8678576329999999 podStartE2EDuration="1.867857633s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:40:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:40:52.862044019 +0000 UTC m=+815.965111732" watchObservedRunningTime="2026-10-31 12:40:52.867857633 +0000 UTC m=+815.970925346"
Oct 31 12:40:53.166178 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:53.166143    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-sv4w2"]
Oct 31 12:40:53.167148 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:40:53.167119    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38bdca08_eb2b_43c5_8c07_7059e5c9e199.slice/crio-184ceb960be55dc2f1bb257c3c9e1df3710eec41a06d9484d0ed29d40209fa5f WatchSource:0}: Error finding container 184ceb960be55dc2f1bb257c3c9e1df3710eec41a06d9484d0ed29d40209fa5f: Status 404 returned error can't find the container with id 184ceb960be55dc2f1bb257c3c9e1df3710eec41a06d9484d0ed29d40209fa5f
Oct 31 12:40:53.247523 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:53.238439    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:40:53.247523 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:53.238543    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:40:53.247523 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:53.238585    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:40:53.285067 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:53.285024    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kj56r"]
Oct 31 12:40:53.376741 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:53.376696    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-b9brm"
Oct 31 12:40:53.419686 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:53.419642    5060 patch_prober.go:29] interesting pod/e2e-worker-w6-67d410-openshift-default-676b978549-b9brm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body=
Oct 31 12:40:53.419890 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:53.419711    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-b9brm" podUID="83855e68-24ce-41e1-97be-79df68bc9505" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused"
Oct 31 12:40:53.900766 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:53.900726    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kj56r" event={"ID":"120c71cc-6b6d-4833-91a1-52bc36d759e8","Type":"ContainerStarted","Data":"550c21bcac92762e91ca89e81bec5dfe5441d94125255396e8d09c95b02e9004"}
Oct 31 12:40:53.921065 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:53.920963    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-sv4w2" event={"ID":"38bdca08-eb2b-43c5-8c07-7059e5c9e199","Type":"ContainerStarted","Data":"4c7d3ca2a9c8a503e740a602271d240f30dea66d2fed340dac15ec3cb71094ef"}
Oct 31 12:40:53.921065 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:53.921017    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-sv4w2" event={"ID":"38bdca08-eb2b-43c5-8c07-7059e5c9e199","Type":"ContainerStarted","Data":"184ceb960be55dc2f1bb257c3c9e1df3710eec41a06d9484d0ed29d40209fa5f"}
Oct 31 12:40:54.413515 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:54.403627    5060 patch_prober.go:29] interesting pod/e2e-worker-w6-67d410-openshift-default-676b978549-b9brm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body=
Oct 31 12:40:54.413515 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:54.403713    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-b9brm" podUID="83855e68-24ce-41e1-97be-79df68bc9505" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused"
Oct 31 12:40:54.836087 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:54.835694    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-sv4w2"
Oct 31 12:40:54.836524 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:54.836393    5060 patch_prober.go:29] interesting pod/e2e-worker-w1-85ce08-openshift-default-77d6458845-sv4w2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body=
Oct 31 12:40:54.836524 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:54.836447    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-sv4w2" podUID="38bdca08-eb2b-43c5-8c07-7059e5c9e199" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused"
Oct 31 12:40:55.335855 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:55.334501    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x2qzv" podUID="c247ed2a-aa90-424b-9b7e-4f2936d67e3f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:40:55.386876 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:55.386373    5060 patch_prober.go:29] interesting pod/e2e-worker-w6-67d410-openshift-default-676b978549-b9brm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body=
Oct 31 12:40:55.386876 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:55.386442    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-b9brm" podUID="83855e68-24ce-41e1-97be-79df68bc9505" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused"
Oct 31 12:40:55.421039 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:55.420601    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/simulator-w4-c67de2-kserve-868679cf9b-gqjbx" podUID="e104d7da-1fb2-4c99-a480-b4ae84c4f273" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:40:55.541657 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:55.541604    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-2-w5-b25d3b-kserve-76f798d59f-vvzhg" podUID="394ed76c-84ac-4612-9acb-a7e36cb68d89" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": context deadline exceeded"
Oct 31 12:40:55.829264 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:55.829222    5060 patch_prober.go:29] interesting pod/e2e-worker-w1-85ce08-openshift-default-77d6458845-sv4w2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body=
Oct 31 12:40:55.833614 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:55.833570    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-sv4w2" podUID="38bdca08-eb2b-43c5-8c07-7059e5c9e199" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused"
Oct 31 12:40:55.838639 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:55.838606    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/premium-w2-e6c79d-kserve-6468bf46d4-h66f8" podUID="cd6a266f-8d12-4089-83ee-c75c716bf26c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:40:56.348587 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:56.345584    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x2qzv" podUID="c247ed2a-aa90-424b-9b7e-4f2936d67e3f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded"
Oct 31 12:40:56.348587 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:56.346376    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x2qzv"
Oct 31 12:40:56.374516 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:56.372689    5060 patch_prober.go:29] interesting pod/e2e-worker-w6-67d410-openshift-default-676b978549-b9brm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body=
Oct 31 12:40:56.374516 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:56.372745    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-b9brm" podUID="83855e68-24ce-41e1-97be-79df68bc9505" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused"
Oct 31 12:40:56.397514 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:56.395296    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-sv4w2" podStartSLOduration=4.395274004 podStartE2EDuration="4.395274004s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:40:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:40:53.99810532 +0000 UTC m=+817.101173058" watchObservedRunningTime="2026-10-31 12:40:56.395274004 +0000 UTC m=+819.498341719"
Oct 31 12:40:56.435746 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:56.432829    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/simulator-w4-c67de2-kserve-868679cf9b-gqjbx" podUID="e104d7da-1fb2-4c99-a480-b4ae84c4f273" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:40:56.435746 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:56.433228    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c67de2/simulator-w4-c67de2-kserve-868679cf9b-gqjbx"
Oct 31 12:40:56.508066 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:56.507735    5060 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-2-w5-b25d3b-kserve-76f798d59f-vvzhg" podUID="394ed76c-84ac-4612-9acb-a7e36cb68d89" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:40:56.568609 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:56.548115    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-2-w5-b25d3b-kserve-76f798d59f-vvzhg" podUID="394ed76c-84ac-4612-9acb-a7e36cb68d89" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": context deadline exceeded"
Oct 31 12:40:56.568609 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:56.551956    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-2-w5-b25d3b-kserve-76f798d59f-vvzhg"
Oct 31 12:40:56.603594 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:56.594846    5060 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-c67de2/simulator-w4-c67de2-kserve-868679cf9b-gqjbx" podUID="e104d7da-1fb2-4c99-a480-b4ae84c4f273" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:40:56.850517 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:56.849593    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/premium-w2-e6c79d-kserve-6468bf46d4-h66f8" podUID="cd6a266f-8d12-4089-83ee-c75c716bf26c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Oct 31 12:40:56.881571 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:56.851913    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e6c79d/premium-w2-e6c79d-kserve-6468bf46d4-h66f8"
Oct 31 12:40:56.881571 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:56.870805    5060 patch_prober.go:29] interesting pod/e2e-worker-w1-85ce08-openshift-default-77d6458845-sv4w2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body=
Oct 31 12:40:56.881571 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:56.870856    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-sv4w2" podUID="38bdca08-eb2b-43c5-8c07-7059e5c9e199" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused"
Oct 31 12:40:57.381630 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:57.345798    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x2qzv" podUID="c247ed2a-aa90-424b-9b7e-4f2936d67e3f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded"
Oct 31 12:40:57.381630 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:57.372898    5060 patch_prober.go:29] interesting pod/e2e-worker-w6-67d410-openshift-default-676b978549-b9brm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body=
Oct 31 12:40:57.381630 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:57.372957    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-b9brm" podUID="83855e68-24ce-41e1-97be-79df68bc9505" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused"
Oct 31 12:40:57.444037 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:57.443949    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/simulator-w4-c67de2-kserve-868679cf9b-gqjbx" podUID="e104d7da-1fb2-4c99-a480-b4ae84c4f273" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded"
Oct 31 12:40:57.548283 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:57.544556    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e6c79d/premium-w2-e6c79d-kserve-6468bf46d4-h66f8"
Oct 31 12:40:57.549396 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:57.549368    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c67de2/simulator-w4-c67de2-kserve-868679cf9b-gqjbx"
Oct 31 12:40:57.550353 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:57.550325    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-2-w5-b25d3b-kserve-76f798d59f-vvzhg" podUID="394ed76c-84ac-4612-9acb-a7e36cb68d89" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:40:57.738348 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:57.738272    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x2qzv"
Oct 31 12:40:57.791513 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:57.772986    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-2-w5-b25d3b-kserve-76f798d59f-vvzhg"
Oct 31 12:40:57.830515 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:57.829728    5060 patch_prober.go:29] interesting pod/e2e-worker-w1-85ce08-openshift-default-77d6458845-sv4w2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body=
Oct 31 12:40:57.830515 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:57.829788    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-sv4w2" podUID="38bdca08-eb2b-43c5-8c07-7059e5c9e199" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused"
Oct 31 12:40:58.389513 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:58.381875    5060 patch_prober.go:29] interesting pod/e2e-worker-w6-67d410-openshift-default-676b978549-b9brm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body=
Oct 31 12:40:58.389513 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:58.381942    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-b9brm" podUID="83855e68-24ce-41e1-97be-79df68bc9505" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused"
Oct 31 12:40:58.843107 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:58.843057    5060 patch_prober.go:29] interesting pod/e2e-worker-w1-85ce08-openshift-default-77d6458845-sv4w2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body=
Oct 31 12:40:58.843654 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:58.843122    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-sv4w2" podUID="38bdca08-eb2b-43c5-8c07-7059e5c9e199" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused"
Oct 31 12:40:59.380340 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:59.379912    5060 patch_prober.go:29] interesting pod/e2e-worker-w6-67d410-openshift-default-676b978549-b9brm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body=
Oct 31 12:40:59.380340 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:59.379969    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-b9brm" podUID="83855e68-24ce-41e1-97be-79df68bc9505" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused"
Oct 31 12:40:59.830518 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:59.827617    5060 patch_prober.go:29] interesting pod/e2e-worker-w1-85ce08-openshift-default-77d6458845-sv4w2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body=
Oct 31 12:40:59.830518 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:40:59.827682    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-sv4w2" podUID="38bdca08-eb2b-43c5-8c07-7059e5c9e199" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused"
Oct 31 12:40:59.969912 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:40:59.969871    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7phb" containerName="extractor"
Oct 31 12:41:00.375314 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:00.374881    5060 patch_prober.go:29] interesting pod/e2e-worker-w6-67d410-openshift-default-676b978549-b9brm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body=
Oct 31 12:41:00.375314 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:00.374943    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-b9brm" podUID="83855e68-24ce-41e1-97be-79df68bc9505" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused"
Oct 31 12:41:00.831516 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:00.828897    5060 patch_prober.go:29] interesting pod/e2e-worker-w1-85ce08-openshift-default-77d6458845-sv4w2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body=
Oct 31 12:41:00.831516 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:00.828963    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-sv4w2" podUID="38bdca08-eb2b-43c5-8c07-7059e5c9e199" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused"
Oct 31 12:41:01.373455 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:01.372898    5060 patch_prober.go:29] interesting pod/e2e-worker-w6-67d410-openshift-default-676b978549-b9brm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body=
Oct 31 12:41:01.373455 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:01.372960    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-b9brm" podUID="83855e68-24ce-41e1-97be-79df68bc9505" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused"
Oct 31 12:41:01.827154 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:01.827115    5060 patch_prober.go:29] interesting pod/e2e-worker-w1-85ce08-openshift-default-77d6458845-sv4w2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body=
Oct 31 12:41:01.827345 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:01.827174    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-sv4w2" podUID="38bdca08-eb2b-43c5-8c07-7059e5c9e199" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused"
Oct 31 12:41:02.553317 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:02.553279    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-b9brm"
Oct 31 12:41:02.554356 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:02.554331    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-b9brm"
Oct 31 12:41:02.555805 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:02.555783    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-b9brm"
Oct 31 12:41:03.157557 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:03.157476    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-sv4w2"
Oct 31 12:41:03.158354 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:03.158321    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-sv4w2"
Oct 31 12:41:03.159238 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:03.159217    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-sv4w2"
Oct 31 12:41:03.343946 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:03.343906    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:41:03.344061 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:03.343977    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:41:03.458085 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:03.458049    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kj56r" event={"ID":"120c71cc-6b6d-4833-91a1-52bc36d759e8","Type":"ContainerStarted","Data":"4605fa9aff5b8d0a04e1ea8977eb486b17349d449855c518b62aa6255364cd85"}
Oct 31 12:41:03.458272 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:03.458118    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kj56r"
Oct 31 12:41:03.488939 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:03.488871    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kj56r" podStartSLOduration=1.478870278 podStartE2EDuration="11.488851601s" totalImagesPullingTime="10.009981323s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:40:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:41:03.481876574 +0000 UTC m=+826.584944287" watchObservedRunningTime="2026-10-31 12:41:03.488851601 +0000 UTC m=+826.591919313"
Oct 31 12:41:04.470464 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:04.470432    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kj56r"
Oct 31 12:41:11.374135 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:11.374100    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19b7740c-lj5z9"]
Oct 31 12:41:11.380061 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:11.380036    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19b7740c-lj5z9"
Oct 31 12:41:11.385375 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:11.385344    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 12:41:11.385985 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:11.385957    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 12:41:11.386113 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:11.386038    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 12:41:11.386178 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:11.386135    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7wsmb\""
Oct 31 12:41:11.389241 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:11.389013    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19b7740c-lj5z9"]
Oct 31 12:41:11.503180 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:11.503149    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88044eb8-3340-459c-8b36-f71cc2d3e79a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-19b7740c-lj5z9\" (UID: \"88044eb8-3340-459c-8b36-f71cc2d3e79a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19b7740c-lj5z9"
Oct 31 12:41:11.503388 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:11.503210    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88044eb8-3340-459c-8b36-f71cc2d3e79a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-19b7740c-lj5z9\" (UID: \"88044eb8-3340-459c-8b36-f71cc2d3e79a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19b7740c-lj5z9"
Oct 31 12:41:11.503388 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:11.503302    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4fdl\" (UniqueName: \"kubernetes.io/projected/88044eb8-3340-459c-8b36-f71cc2d3e79a-kube-api-access-v4fdl\") pod \"maas-api-revoke-sub-19b7740c-lj5z9\" (UID: \"88044eb8-3340-459c-8b36-f71cc2d3e79a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19b7740c-lj5z9"
Oct 31 12:41:11.604773 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:11.604734    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88044eb8-3340-459c-8b36-f71cc2d3e79a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-19b7740c-lj5z9\" (UID: \"88044eb8-3340-459c-8b36-f71cc2d3e79a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19b7740c-lj5z9"
Oct 31 12:41:11.604948 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:11.604825    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88044eb8-3340-459c-8b36-f71cc2d3e79a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-19b7740c-lj5z9\" (UID: \"88044eb8-3340-459c-8b36-f71cc2d3e79a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19b7740c-lj5z9"
Oct 31 12:41:11.604948 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:11.604871    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4fdl\" (UniqueName: \"kubernetes.io/projected/88044eb8-3340-459c-8b36-f71cc2d3e79a-kube-api-access-v4fdl\") pod \"maas-api-revoke-sub-19b7740c-lj5z9\" (UID: \"88044eb8-3340-459c-8b36-f71cc2d3e79a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19b7740c-lj5z9"
Oct 31 12:41:11.605620 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:11.605593    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88044eb8-3340-459c-8b36-f71cc2d3e79a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-19b7740c-lj5z9\" (UID: \"88044eb8-3340-459c-8b36-f71cc2d3e79a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19b7740c-lj5z9"
Oct 31 12:41:11.607132 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:11.607110    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88044eb8-3340-459c-8b36-f71cc2d3e79a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-19b7740c-lj5z9\" (UID: \"88044eb8-3340-459c-8b36-f71cc2d3e79a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19b7740c-lj5z9"
Oct 31 12:41:11.620602 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:11.620580    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4fdl\" (UniqueName: \"kubernetes.io/projected/88044eb8-3340-459c-8b36-f71cc2d3e79a-kube-api-access-v4fdl\") pod \"maas-api-revoke-sub-19b7740c-lj5z9\" (UID: \"88044eb8-3340-459c-8b36-f71cc2d3e79a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19b7740c-lj5z9"
Oct 31 12:41:11.698559 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:11.698447    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19b7740c-lj5z9"
Oct 31 12:41:11.986018 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:41:11.985953    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88044eb8_3340_459c_8b36_f71cc2d3e79a.slice/crio-a87315ba537f4af9a0684a7ec91f61107a9891fcec7a7b67fd63720368b37e1a WatchSource:0}: Error finding container a87315ba537f4af9a0684a7ec91f61107a9891fcec7a7b67fd63720368b37e1a: Status 404 returned error can't find the container with id a87315ba537f4af9a0684a7ec91f61107a9891fcec7a7b67fd63720368b37e1a
Oct 31 12:41:11.986875 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:11.986806    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19b7740c-lj5z9"]
Oct 31 12:41:12.546770 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:12.546718    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19b7740c-lj5z9" event={"ID":"88044eb8-3340-459c-8b36-f71cc2d3e79a","Type":"ContainerStarted","Data":"32249e761326f57144a6cce55f8bff9d6e63a69e7be5c737b65f7d73b52b8d8e"}
Oct 31 12:41:12.547262 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:12.546784    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19b7740c-lj5z9" event={"ID":"88044eb8-3340-459c-8b36-f71cc2d3e79a","Type":"ContainerStarted","Data":"a87315ba537f4af9a0684a7ec91f61107a9891fcec7a7b67fd63720368b37e1a"}
Oct 31 12:41:12.571226 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:12.571169    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19b7740c-lj5z9" podStartSLOduration=1.571155229 podStartE2EDuration="1.571155229s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:41:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:41:12.566986347 +0000 UTC m=+835.670054059" watchObservedRunningTime="2026-10-31 12:41:12.571155229 +0000 UTC m=+835.674222941"
Oct 31 12:41:13.563807 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:13.563758    5060 generic.go:356] "Generic (PLEG): container finished" podID="88044eb8-3340-459c-8b36-f71cc2d3e79a" containerID="32249e761326f57144a6cce55f8bff9d6e63a69e7be5c737b65f7d73b52b8d8e" exitCode=0
Oct 31 12:41:13.563807 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:13.563795    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19b7740c-lj5z9" event={"ID":"88044eb8-3340-459c-8b36-f71cc2d3e79a","Type":"ContainerDied","Data":"32249e761326f57144a6cce55f8bff9d6e63a69e7be5c737b65f7d73b52b8d8e"}
Oct 31 12:41:14.750483 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:14.750455    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19b7740c-lj5z9"
Oct 31 12:41:14.844763 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:14.844682    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/88044eb8-3340-459c-8b36-f71cc2d3e79a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88044eb8-3340-459c-8b36-f71cc2d3e79a-maas-api-xks-ca\") pod \"88044eb8-3340-459c-8b36-f71cc2d3e79a\" (UID: \"88044eb8-3340-459c-8b36-f71cc2d3e79a\") "
Oct 31 12:41:14.844763 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:14.844716    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88044eb8-3340-459c-8b36-f71cc2d3e79a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88044eb8-3340-459c-8b36-f71cc2d3e79a-maas-api-openshift-ca\") pod \"88044eb8-3340-459c-8b36-f71cc2d3e79a\" (UID: \"88044eb8-3340-459c-8b36-f71cc2d3e79a\") "
Oct 31 12:41:14.844763 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:14.844750    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88044eb8-3340-459c-8b36-f71cc2d3e79a-kube-api-access-v4fdl\" (UniqueName: \"kubernetes.io/projected/88044eb8-3340-459c-8b36-f71cc2d3e79a-kube-api-access-v4fdl\") pod \"88044eb8-3340-459c-8b36-f71cc2d3e79a\" (UID: \"88044eb8-3340-459c-8b36-f71cc2d3e79a\") "
Oct 31 12:41:14.845127 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:14.845098    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88044eb8-3340-459c-8b36-f71cc2d3e79a-maas-api-openshift-ca" pod "88044eb8-3340-459c-8b36-f71cc2d3e79a" (UID: "88044eb8-3340-459c-8b36-f71cc2d3e79a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:41:14.845434 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:14.845406    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88044eb8-3340-459c-8b36-f71cc2d3e79a-maas-api-openshift-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:41:14.846888 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:14.846859    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88044eb8-3340-459c-8b36-f71cc2d3e79a-maas-api-xks-ca" pod "88044eb8-3340-459c-8b36-f71cc2d3e79a" (UID: "88044eb8-3340-459c-8b36-f71cc2d3e79a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:41:14.846980 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:14.846951    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88044eb8-3340-459c-8b36-f71cc2d3e79a-kube-api-access-v4fdl" pod "88044eb8-3340-459c-8b36-f71cc2d3e79a" (UID: "88044eb8-3340-459c-8b36-f71cc2d3e79a"). InnerVolumeSpecName "kube-api-access-v4fdl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:41:14.947170 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:14.947136    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88044eb8-3340-459c-8b36-f71cc2d3e79a-maas-api-xks-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:41:14.947170 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:14.947166    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4fdl\" (UniqueName: \"kubernetes.io/projected/88044eb8-3340-459c-8b36-f71cc2d3e79a-kube-api-access-v4fdl\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:41:15.589359 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:15.589323    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19b7740c-lj5z9"
Oct 31 12:41:15.589359 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:15.589340    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19b7740c-lj5z9" event={"ID":"88044eb8-3340-459c-8b36-f71cc2d3e79a","Type":"ContainerDied","Data":"a87315ba537f4af9a0684a7ec91f61107a9891fcec7a7b67fd63720368b37e1a"}
Oct 31 12:41:15.589644 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:15.589404    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a87315ba537f4af9a0684a7ec91f61107a9891fcec7a7b67fd63720368b37e1a"
Oct 31 12:41:16.042566 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:16.042527    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff5915db-km2qc"]
Oct 31 12:41:16.043155 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:16.043141    5060 cpu_manager.go:414] "containerMap: removing container" podUID="88044eb8-3340-459c-8b36-f71cc2d3e79a" containerName="revoke-keys"
Oct 31 12:41:16.043203 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:16.043156    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88044eb8-3340-459c-8b36-f71cc2d3e79a" containerName="revoke-keys"
Oct 31 12:41:16.043303 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:16.043295    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="88044eb8-3340-459c-8b36-f71cc2d3e79a" containerName="revoke-keys"
Oct 31 12:41:16.047486 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:16.047468    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff5915db-km2qc"
Oct 31 12:41:16.052268 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:16.052228    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 12:41:16.052268 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:16.052258    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 12:41:16.052466 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:16.052284    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 12:41:16.052570 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:16.052529    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7wsmb\""
Oct 31 12:41:16.058062 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:16.057634    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff5915db-km2qc"]
Oct 31 12:41:16.162118 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:16.162078    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36cecbf2-83f6-42d3-9846-ac361a13e427-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ff5915db-km2qc\" (UID: \"36cecbf2-83f6-42d3-9846-ac361a13e427\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff5915db-km2qc"
Oct 31 12:41:16.162358 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:16.162334    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zj2v\" (UniqueName: \"kubernetes.io/projected/36cecbf2-83f6-42d3-9846-ac361a13e427-kube-api-access-9zj2v\") pod \"maas-api-revoke-sub-ff5915db-km2qc\" (UID: \"36cecbf2-83f6-42d3-9846-ac361a13e427\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff5915db-km2qc"
Oct 31 12:41:16.162726 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:16.162694    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36cecbf2-83f6-42d3-9846-ac361a13e427-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ff5915db-km2qc\" (UID: \"36cecbf2-83f6-42d3-9846-ac361a13e427\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff5915db-km2qc"
Oct 31 12:41:16.263955 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:16.263924    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zj2v\" (UniqueName: \"kubernetes.io/projected/36cecbf2-83f6-42d3-9846-ac361a13e427-kube-api-access-9zj2v\") pod \"maas-api-revoke-sub-ff5915db-km2qc\" (UID: \"36cecbf2-83f6-42d3-9846-ac361a13e427\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff5915db-km2qc"
Oct 31 12:41:16.264120 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:16.264031    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36cecbf2-83f6-42d3-9846-ac361a13e427-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ff5915db-km2qc\" (UID: \"36cecbf2-83f6-42d3-9846-ac361a13e427\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff5915db-km2qc"
Oct 31 12:41:16.264120 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:16.264079    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36cecbf2-83f6-42d3-9846-ac361a13e427-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ff5915db-km2qc\" (UID: \"36cecbf2-83f6-42d3-9846-ac361a13e427\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff5915db-km2qc"
Oct 31 12:41:16.264885 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:16.264853    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36cecbf2-83f6-42d3-9846-ac361a13e427-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ff5915db-km2qc\" (UID: \"36cecbf2-83f6-42d3-9846-ac361a13e427\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff5915db-km2qc"
Oct 31 12:41:16.266598 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:16.266570    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36cecbf2-83f6-42d3-9846-ac361a13e427-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ff5915db-km2qc\" (UID: \"36cecbf2-83f6-42d3-9846-ac361a13e427\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff5915db-km2qc"
Oct 31 12:41:16.276923 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:16.276879    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zj2v\" (UniqueName: \"kubernetes.io/projected/36cecbf2-83f6-42d3-9846-ac361a13e427-kube-api-access-9zj2v\") pod \"maas-api-revoke-sub-ff5915db-km2qc\" (UID: \"36cecbf2-83f6-42d3-9846-ac361a13e427\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff5915db-km2qc"
Oct 31 12:41:16.368088 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:16.368002    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff5915db-km2qc"
Oct 31 12:41:16.663372 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:41:16.663341    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36cecbf2_83f6_42d3_9846_ac361a13e427.slice/crio-ea4f87bcadeb1352d6601fb3f425177072483c86f751b03ff50050bb34be0292 WatchSource:0}: Error finding container ea4f87bcadeb1352d6601fb3f425177072483c86f751b03ff50050bb34be0292: Status 404 returned error can't find the container with id ea4f87bcadeb1352d6601fb3f425177072483c86f751b03ff50050bb34be0292
Oct 31 12:41:16.668001 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:16.667972    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff5915db-km2qc"]
Oct 31 12:41:17.611957 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:17.611916    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff5915db-km2qc" event={"ID":"36cecbf2-83f6-42d3-9846-ac361a13e427","Type":"ContainerStarted","Data":"b72780e0b86d0ec07ac0274ca6268ef2e0924a34df4b4ef8f569cee1b098f8f9"}
Oct 31 12:41:17.612349 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:17.611971    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff5915db-km2qc" event={"ID":"36cecbf2-83f6-42d3-9846-ac361a13e427","Type":"ContainerStarted","Data":"ea4f87bcadeb1352d6601fb3f425177072483c86f751b03ff50050bb34be0292"}
Oct 31 12:41:17.636403 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:17.636352    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff5915db-km2qc" podStartSLOduration=1.636338491 podStartE2EDuration="1.636338491s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:41:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:41:17.631428196 +0000 UTC m=+840.734495912" watchObservedRunningTime="2026-10-31 12:41:17.636338491 +0000 UTC m=+840.739406247"
Oct 31 12:41:18.644517 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:18.643794    5060 generic.go:356] "Generic (PLEG): container finished" podID="36cecbf2-83f6-42d3-9846-ac361a13e427" containerID="b72780e0b86d0ec07ac0274ca6268ef2e0924a34df4b4ef8f569cee1b098f8f9" exitCode=0
Oct 31 12:41:18.644517 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:18.643852    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff5915db-km2qc" event={"ID":"36cecbf2-83f6-42d3-9846-ac361a13e427","Type":"ContainerDied","Data":"b72780e0b86d0ec07ac0274ca6268ef2e0924a34df4b4ef8f569cee1b098f8f9"}
Oct 31 12:41:20.237515 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:20.236979    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff5915db-km2qc"
Oct 31 12:41:20.346615 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:20.321856    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36cecbf2-83f6-42d3-9846-ac361a13e427-kube-api-access-9zj2v\" (UniqueName: \"kubernetes.io/projected/36cecbf2-83f6-42d3-9846-ac361a13e427-kube-api-access-9zj2v\") pod \"36cecbf2-83f6-42d3-9846-ac361a13e427\" (UID: \"36cecbf2-83f6-42d3-9846-ac361a13e427\") "
Oct 31 12:41:20.346615 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:20.321975    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/36cecbf2-83f6-42d3-9846-ac361a13e427-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36cecbf2-83f6-42d3-9846-ac361a13e427-maas-api-xks-ca\") pod \"36cecbf2-83f6-42d3-9846-ac361a13e427\" (UID: \"36cecbf2-83f6-42d3-9846-ac361a13e427\") "
Oct 31 12:41:20.346615 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:20.322157    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36cecbf2-83f6-42d3-9846-ac361a13e427-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36cecbf2-83f6-42d3-9846-ac361a13e427-maas-api-openshift-ca\") pod \"36cecbf2-83f6-42d3-9846-ac361a13e427\" (UID: \"36cecbf2-83f6-42d3-9846-ac361a13e427\") "
Oct 31 12:41:20.346615 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:20.322834    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36cecbf2-83f6-42d3-9846-ac361a13e427-maas-api-openshift-ca" pod "36cecbf2-83f6-42d3-9846-ac361a13e427" (UID: "36cecbf2-83f6-42d3-9846-ac361a13e427"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:41:20.346615 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:20.338107    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36cecbf2-83f6-42d3-9846-ac361a13e427-kube-api-access-9zj2v" pod "36cecbf2-83f6-42d3-9846-ac361a13e427" (UID: "36cecbf2-83f6-42d3-9846-ac361a13e427"). InnerVolumeSpecName "kube-api-access-9zj2v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:41:20.346615 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:20.341827    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36cecbf2-83f6-42d3-9846-ac361a13e427-maas-api-openshift-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:41:20.346615 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:20.341855    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9zj2v\" (UniqueName: \"kubernetes.io/projected/36cecbf2-83f6-42d3-9846-ac361a13e427-kube-api-access-9zj2v\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:41:20.488856 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:20.488796    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36cecbf2-83f6-42d3-9846-ac361a13e427-maas-api-xks-ca" pod "36cecbf2-83f6-42d3-9846-ac361a13e427" (UID: "36cecbf2-83f6-42d3-9846-ac361a13e427"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:41:20.582512 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:20.559482    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36cecbf2-83f6-42d3-9846-ac361a13e427-maas-api-xks-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:41:20.752566 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:20.749255    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff5915db-km2qc" event={"ID":"36cecbf2-83f6-42d3-9846-ac361a13e427","Type":"ContainerDied","Data":"ea4f87bcadeb1352d6601fb3f425177072483c86f751b03ff50050bb34be0292"}
Oct 31 12:41:20.752566 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:20.749322    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ea4f87bcadeb1352d6601fb3f425177072483c86f751b03ff50050bb34be0292"
Oct 31 12:41:20.752566 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:20.749393    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff5915db-km2qc"
Oct 31 12:41:21.547664 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:21.547613    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/premium-w2-e6c79d-kserve-6468bf46d4-h66f8" podUID="cd6a266f-8d12-4089-83ee-c75c716bf26c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:41:21.561615 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:21.561566    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/simulator-w4-c67de2-kserve-868679cf9b-gqjbx" podUID="e104d7da-1fb2-4c99-a480-b4ae84c4f273" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:41:21.755514 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:21.754686    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x2qzv" podUID="c247ed2a-aa90-424b-9b7e-4f2936d67e3f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:41:21.793640 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:21.793591    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-2-w5-b25d3b-kserve-76f798d59f-vvzhg" podUID="394ed76c-84ac-4612-9acb-a7e36cb68d89" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:41:22.431528 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:22.369384    5060 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x2qzv" podUID="c247ed2a-aa90-424b-9b7e-4f2936d67e3f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": context deadline exceeded"
Oct 31 12:41:22.572681 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:22.572634    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/simulator-w4-c67de2-kserve-868679cf9b-gqjbx" podUID="e104d7da-1fb2-4c99-a480-b4ae84c4f273" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:41:22.575422 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:22.575398    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-kmx7z"]
Oct 31 12:41:22.595525 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:22.595459    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/premium-w2-e6c79d-kserve-6468bf46d4-h66f8" podUID="cd6a266f-8d12-4089-83ee-c75c716bf26c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:41:22.937520 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:22.934548    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x2qzv" podUID="c247ed2a-aa90-424b-9b7e-4f2936d67e3f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:41:22.937520 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:22.935639    5060 cpu_manager.go:414] "containerMap: removing container" podUID="36cecbf2-83f6-42d3-9846-ac361a13e427" containerName="revoke-keys"
Oct 31 12:41:22.937520 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:22.935663    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36cecbf2-83f6-42d3-9846-ac361a13e427" containerName="revoke-keys"
Oct 31 12:41:22.937520 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:22.935894    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="36cecbf2-83f6-42d3-9846-ac361a13e427" containerName="revoke-keys"
Oct 31 12:41:22.983512 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:22.955436    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-2-w5-b25d3b-kserve-76f798d59f-vvzhg" podUID="394ed76c-84ac-4612-9acb-a7e36cb68d89" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:41:23.137091 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:23.117277    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-kmx7z"
Oct 31 12:41:23.535700 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:23.535652    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-kmx7z"]
Oct 31 12:41:23.548712 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:23.548673    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c67de2/simulator-w4-c67de2-kserve-868679cf9b-gqjbx"
Oct 31 12:41:23.548933 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:23.548915    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75668bd6-vwl6q"]
Oct 31 12:41:23.606548 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:23.581484    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/simulator-w4-c67de2-kserve-868679cf9b-gqjbx" podUID="e104d7da-1fb2-4c99-a480-b4ae84c4f273" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:41:23.622868 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:23.612699    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b8d44ec-1b81-4ca2-9337-6294db29bb5b-istio-data\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-kmx7z\" (UID: \"9b8d44ec-1b81-4ca2-9337-6294db29bb5b\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-kmx7z"
Oct 31 12:41:23.760611 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:23.690995    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/premium-w2-e6c79d-kserve-6468bf46d4-h66f8" podUID="cd6a266f-8d12-4089-83ee-c75c716bf26c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:41:23.760611 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:23.723308    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75668bd6-vwl6q"]
Oct 31 12:41:23.760611 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:23.754273    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e6c79d/premium-w2-e6c79d-kserve-6468bf46d4-h66f8"
Oct 31 12:41:23.760611 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:23.754334    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x2qzv"
Oct 31 12:41:23.760611 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:23.754383    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-2-w5-b25d3b-kserve-76f798d59f-vvzhg"
Oct 31 12:41:23.760611 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:23.754417    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-2-w5-b25d3b-kserve-76f798d59f-vvzhg"
Oct 31 12:41:23.760611 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:23.754448    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x2qzv"
Oct 31 12:41:23.760611 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:23.754478    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c67de2/simulator-w4-c67de2-kserve-868679cf9b-gqjbx"
Oct 31 12:41:23.760611 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:23.723391    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75668bd6-vwl6q"
Oct 31 12:41:23.770118 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:23.769677    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 12:41:23.770118 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:23.769823    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 12:41:23.770118 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:23.769926    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7wsmb\""
Oct 31 12:41:23.770118 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:23.769976    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 12:41:23.859604 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:23.859475    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e6c79d/premium-w2-e6c79d-kserve-6468bf46d4-h66f8"
Oct 31 12:41:23.885154 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:23.885111    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b8d44ec-1b81-4ca2-9337-6294db29bb5b-istio-podinfo\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-kmx7z\" (UID: \"9b8d44ec-1b81-4ca2-9337-6294db29bb5b\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-kmx7z"
Oct 31 12:41:23.885409 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:23.885385    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b8d44ec-1b81-4ca2-9337-6294db29bb5b-workload-socket\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-kmx7z\" (UID: \"9b8d44ec-1b81-4ca2-9337-6294db29bb5b\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-kmx7z"
Oct 31 12:41:23.885738 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:23.885715    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b8d44ec-1b81-4ca2-9337-6294db29bb5b-credential-socket\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-kmx7z\" (UID: \"9b8d44ec-1b81-4ca2-9337-6294db29bb5b\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-kmx7z"
Oct 31 12:41:23.885880 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:23.885863    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b8d44ec-1b81-4ca2-9337-6294db29bb5b-workload-certs\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-kmx7z\" (UID: \"9b8d44ec-1b81-4ca2-9337-6294db29bb5b\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-kmx7z"
Oct 31 12:41:23.886118 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:23.886100    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b8d44ec-1b81-4ca2-9337-6294db29bb5b-istio-token\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-kmx7z\" (UID: \"9b8d44ec-1b81-4ca2-9337-6294db29bb5b\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-kmx7z"
Oct 31 12:41:23.892312 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:23.892282    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b8d44ec-1b81-4ca2-9337-6294db29bb5b-istio-envoy\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-kmx7z\" (UID: \"9b8d44ec-1b81-4ca2-9337-6294db29bb5b\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-kmx7z"
Oct 31 12:41:23.892984 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:23.892961    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b8d44ec-1b81-4ca2-9337-6294db29bb5b-istiod-ca-cert\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-kmx7z\" (UID: \"9b8d44ec-1b81-4ca2-9337-6294db29bb5b\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-kmx7z"
Oct 31 12:41:23.997098 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:23.997059    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b8d44ec-1b81-4ca2-9337-6294db29bb5b-istio-envoy\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-kmx7z\" (UID: \"9b8d44ec-1b81-4ca2-9337-6294db29bb5b\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-kmx7z"
Oct 31 12:41:23.998766 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:23.998739    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b8d44ec-1b81-4ca2-9337-6294db29bb5b-istiod-ca-cert\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-kmx7z\" (UID: \"9b8d44ec-1b81-4ca2-9337-6294db29bb5b\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-kmx7z"
Oct 31 12:41:24.015404 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:24.006847    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b8d44ec-1b81-4ca2-9337-6294db29bb5b-istiod-ca-cert\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-kmx7z\" (UID: \"9b8d44ec-1b81-4ca2-9337-6294db29bb5b\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-kmx7z"
Oct 31 12:41:24.015404 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:24.007676    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b8d44ec-1b81-4ca2-9337-6294db29bb5b-istio-data\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-kmx7z\" (UID: \"9b8d44ec-1b81-4ca2-9337-6294db29bb5b\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-kmx7z"
Oct 31 12:41:24.015404 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:24.007729    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8h5mb\" (UniqueName: \"kubernetes.io/projected/9b8d44ec-1b81-4ca2-9337-6294db29bb5b-kube-api-access-8h5mb\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-kmx7z\" (UID: \"9b8d44ec-1b81-4ca2-9337-6294db29bb5b\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-kmx7z"
Oct 31 12:41:24.015404 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:24.007812    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b8d44ec-1b81-4ca2-9337-6294db29bb5b-istio-podinfo\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-kmx7z\" (UID: \"9b8d44ec-1b81-4ca2-9337-6294db29bb5b\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-kmx7z"
Oct 31 12:41:24.015404 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:24.007878    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b8d44ec-1b81-4ca2-9337-6294db29bb5b-workload-socket\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-kmx7z\" (UID: \"9b8d44ec-1b81-4ca2-9337-6294db29bb5b\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-kmx7z"
Oct 31 12:41:24.015404 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:24.007990    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b8d44ec-1b81-4ca2-9337-6294db29bb5b-credential-socket\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-kmx7z\" (UID: \"9b8d44ec-1b81-4ca2-9337-6294db29bb5b\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-kmx7z"
Oct 31 12:41:24.015404 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:24.008025    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b8d44ec-1b81-4ca2-9337-6294db29bb5b-workload-certs\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-kmx7z\" (UID: \"9b8d44ec-1b81-4ca2-9337-6294db29bb5b\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-kmx7z"
Oct 31 12:41:24.015404 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:24.008065    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1b6f29e-44fd-49de-a88e-ae6cf2efcfef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-75668bd6-vwl6q\" (UID: \"b1b6f29e-44fd-49de-a88e-ae6cf2efcfef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75668bd6-vwl6q"
Oct 31 12:41:24.015404 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:24.008180    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b8d44ec-1b81-4ca2-9337-6294db29bb5b-istio-token\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-kmx7z\" (UID: \"9b8d44ec-1b81-4ca2-9337-6294db29bb5b\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-kmx7z"
Oct 31 12:41:24.015404 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:24.008210    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1b6f29e-44fd-49de-a88e-ae6cf2efcfef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-75668bd6-vwl6q\" (UID: \"b1b6f29e-44fd-49de-a88e-ae6cf2efcfef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75668bd6-vwl6q"
Oct 31 12:41:24.015404 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:24.008231    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98p2t\" (UniqueName: \"kubernetes.io/projected/b1b6f29e-44fd-49de-a88e-ae6cf2efcfef-kube-api-access-98p2t\") pod \"maas-api-revoke-sub-75668bd6-vwl6q\" (UID: \"b1b6f29e-44fd-49de-a88e-ae6cf2efcfef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75668bd6-vwl6q"
Oct 31 12:41:24.015404 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:24.008738    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b8d44ec-1b81-4ca2-9337-6294db29bb5b-istio-data\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-kmx7z\" (UID: \"9b8d44ec-1b81-4ca2-9337-6294db29bb5b\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-kmx7z"
Oct 31 12:41:24.015404 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:24.009048    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b8d44ec-1b81-4ca2-9337-6294db29bb5b-credential-socket\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-kmx7z\" (UID: \"9b8d44ec-1b81-4ca2-9337-6294db29bb5b\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-kmx7z"
Oct 31 12:41:24.015404 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:24.009188    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b8d44ec-1b81-4ca2-9337-6294db29bb5b-workload-certs\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-kmx7z\" (UID: \"9b8d44ec-1b81-4ca2-9337-6294db29bb5b\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-kmx7z"
Oct 31 12:41:24.015404 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:24.010533    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b8d44ec-1b81-4ca2-9337-6294db29bb5b-workload-socket\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-kmx7z\" (UID: \"9b8d44ec-1b81-4ca2-9337-6294db29bb5b\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-kmx7z"
Oct 31 12:41:24.035546 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:24.030360    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b8d44ec-1b81-4ca2-9337-6294db29bb5b-istio-podinfo\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-kmx7z\" (UID: \"9b8d44ec-1b81-4ca2-9337-6294db29bb5b\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-kmx7z"
Oct 31 12:41:24.035546 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:24.030744    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b8d44ec-1b81-4ca2-9337-6294db29bb5b-istio-envoy\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-kmx7z\" (UID: \"9b8d44ec-1b81-4ca2-9337-6294db29bb5b\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-kmx7z"
Oct 31 12:41:24.042193 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:24.042166    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b8d44ec-1b81-4ca2-9337-6294db29bb5b-istio-token\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-kmx7z\" (UID: \"9b8d44ec-1b81-4ca2-9337-6294db29bb5b\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-kmx7z"
Oct 31 12:41:24.109592 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:24.109547    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8h5mb\" (UniqueName: \"kubernetes.io/projected/9b8d44ec-1b81-4ca2-9337-6294db29bb5b-kube-api-access-8h5mb\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-kmx7z\" (UID: \"9b8d44ec-1b81-4ca2-9337-6294db29bb5b\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-kmx7z"
Oct 31 12:41:24.109929 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:24.109858    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1b6f29e-44fd-49de-a88e-ae6cf2efcfef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-75668bd6-vwl6q\" (UID: \"b1b6f29e-44fd-49de-a88e-ae6cf2efcfef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75668bd6-vwl6q"
Oct 31 12:41:24.110214 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:24.110191    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1b6f29e-44fd-49de-a88e-ae6cf2efcfef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-75668bd6-vwl6q\" (UID: \"b1b6f29e-44fd-49de-a88e-ae6cf2efcfef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75668bd6-vwl6q"
Oct 31 12:41:24.110317 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:24.110224    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98p2t\" (UniqueName: \"kubernetes.io/projected/b1b6f29e-44fd-49de-a88e-ae6cf2efcfef-kube-api-access-98p2t\") pod \"maas-api-revoke-sub-75668bd6-vwl6q\" (UID: \"b1b6f29e-44fd-49de-a88e-ae6cf2efcfef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75668bd6-vwl6q"
Oct 31 12:41:24.110995 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:24.110975    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1b6f29e-44fd-49de-a88e-ae6cf2efcfef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-75668bd6-vwl6q\" (UID: \"b1b6f29e-44fd-49de-a88e-ae6cf2efcfef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75668bd6-vwl6q"
Oct 31 12:41:24.112770 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:24.112746    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1b6f29e-44fd-49de-a88e-ae6cf2efcfef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-75668bd6-vwl6q\" (UID: \"b1b6f29e-44fd-49de-a88e-ae6cf2efcfef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75668bd6-vwl6q"
Oct 31 12:41:24.122231 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:24.122205    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8h5mb\" (UniqueName: \"kubernetes.io/projected/9b8d44ec-1b81-4ca2-9337-6294db29bb5b-kube-api-access-8h5mb\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-kmx7z\" (UID: \"9b8d44ec-1b81-4ca2-9337-6294db29bb5b\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-kmx7z"
Oct 31 12:41:24.137538 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:24.137506    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98p2t\" (UniqueName: \"kubernetes.io/projected/b1b6f29e-44fd-49de-a88e-ae6cf2efcfef-kube-api-access-98p2t\") pod \"maas-api-revoke-sub-75668bd6-vwl6q\" (UID: \"b1b6f29e-44fd-49de-a88e-ae6cf2efcfef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75668bd6-vwl6q"
Oct 31 12:41:24.181570 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:24.181514    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-kmx7z"
Oct 31 12:41:24.217405 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:24.217368    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75668bd6-vwl6q"
Oct 31 12:41:24.564130 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:24.564088    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-kmx7z"]
Oct 31 12:41:24.568334 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:41:24.568167    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b8d44ec_1b81_4ca2_9337_6294db29bb5b.slice/crio-c263cb3a737d0fd06a44d32b13b7c9d2df85563729ce11284e8ced59d219bd46 WatchSource:0}: Error finding container c263cb3a737d0fd06a44d32b13b7c9d2df85563729ce11284e8ced59d219bd46: Status 404 returned error can't find the container with id c263cb3a737d0fd06a44d32b13b7c9d2df85563729ce11284e8ced59d219bd46
Oct 31 12:41:24.571764 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:24.571723    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:41:24.571974 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:24.571952    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:41:24.572174 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:24.572155    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:41:24.578615 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:24.578577    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75668bd6-vwl6q"]
Oct 31 12:41:24.778412 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:24.778376    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96d6e803-8ckml"]
Oct 31 12:41:24.795706 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:24.795678    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96d6e803-8ckml"]
Oct 31 12:41:24.796230 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:24.796163    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96d6e803-8ckml"
Oct 31 12:41:24.828509 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:24.828407    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wt2jw\" (UniqueName: \"kubernetes.io/projected/48fd0a20-b1ec-47cd-9f68-4638965a5acc-kube-api-access-wt2jw\") pod \"maas-api-revoke-sub-96d6e803-8ckml\" (UID: \"48fd0a20-b1ec-47cd-9f68-4638965a5acc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96d6e803-8ckml"
Oct 31 12:41:24.828706 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:24.828585    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48fd0a20-b1ec-47cd-9f68-4638965a5acc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-96d6e803-8ckml\" (UID: \"48fd0a20-b1ec-47cd-9f68-4638965a5acc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96d6e803-8ckml"
Oct 31 12:41:24.828783 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:24.828724    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48fd0a20-b1ec-47cd-9f68-4638965a5acc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-96d6e803-8ckml\" (UID: \"48fd0a20-b1ec-47cd-9f68-4638965a5acc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96d6e803-8ckml"
Oct 31 12:41:24.930185 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:24.930151    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48fd0a20-b1ec-47cd-9f68-4638965a5acc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-96d6e803-8ckml\" (UID: \"48fd0a20-b1ec-47cd-9f68-4638965a5acc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96d6e803-8ckml"
Oct 31 12:41:24.930344 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:24.930259    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48fd0a20-b1ec-47cd-9f68-4638965a5acc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-96d6e803-8ckml\" (UID: \"48fd0a20-b1ec-47cd-9f68-4638965a5acc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96d6e803-8ckml"
Oct 31 12:41:24.930344 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:24.930333    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wt2jw\" (UniqueName: \"kubernetes.io/projected/48fd0a20-b1ec-47cd-9f68-4638965a5acc-kube-api-access-wt2jw\") pod \"maas-api-revoke-sub-96d6e803-8ckml\" (UID: \"48fd0a20-b1ec-47cd-9f68-4638965a5acc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96d6e803-8ckml"
Oct 31 12:41:24.931383 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:24.931353    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48fd0a20-b1ec-47cd-9f68-4638965a5acc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-96d6e803-8ckml\" (UID: \"48fd0a20-b1ec-47cd-9f68-4638965a5acc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96d6e803-8ckml"
Oct 31 12:41:24.933081 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:24.933058    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48fd0a20-b1ec-47cd-9f68-4638965a5acc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-96d6e803-8ckml\" (UID: \"48fd0a20-b1ec-47cd-9f68-4638965a5acc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96d6e803-8ckml"
Oct 31 12:41:24.945107 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:24.942733    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wt2jw\" (UniqueName: \"kubernetes.io/projected/48fd0a20-b1ec-47cd-9f68-4638965a5acc-kube-api-access-wt2jw\") pod \"maas-api-revoke-sub-96d6e803-8ckml\" (UID: \"48fd0a20-b1ec-47cd-9f68-4638965a5acc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96d6e803-8ckml"
Oct 31 12:41:25.061762 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:25.061725    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75668bd6-vwl6q" event={"ID":"b1b6f29e-44fd-49de-a88e-ae6cf2efcfef","Type":"ContainerStarted","Data":"cda9d8a0265f4fff259ab967919f458ca7f2b54682f31537efb2d25ff00fd8ab"}
Oct 31 12:41:25.064967 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:25.064931    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-kmx7z" event={"ID":"9b8d44ec-1b81-4ca2-9337-6294db29bb5b","Type":"ContainerStarted","Data":"7f53fd2543e6a9686da79fddfa2f812edc241f2cc4cb638a5678d8679506870a"}
Oct 31 12:41:25.065131 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:25.064987    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-kmx7z" event={"ID":"9b8d44ec-1b81-4ca2-9337-6294db29bb5b","Type":"ContainerStarted","Data":"c263cb3a737d0fd06a44d32b13b7c9d2df85563729ce11284e8ced59d219bd46"}
Oct 31 12:41:25.097291 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:25.097174    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-kmx7z" podStartSLOduration=4.097154349 podStartE2EDuration="4.097154349s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:41:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:41:25.092706684 +0000 UTC m=+848.195774403" watchObservedRunningTime="2026-10-31 12:41:25.097154349 +0000 UTC m=+848.200222061"
Oct 31 12:41:25.126384 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:25.126344    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96d6e803-8ckml"
Oct 31 12:41:25.182353 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:25.182317    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-kmx7z"
Oct 31 12:41:25.185520 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:25.185479    5060 patch_prober.go:29] interesting pod/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-kmx7z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body=
Oct 31 12:41:25.185727 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:25.185705    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-kmx7z" podUID="9b8d44ec-1b81-4ca2-9337-6294db29bb5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused"
Oct 31 12:41:25.473723 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:41:25.473680    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48fd0a20_b1ec_47cd_9f68_4638965a5acc.slice/crio-a8aa2558507f00520634d6ce1fcc4a217e2b1d96137f988721f82cb49ccddaac WatchSource:0}: Error finding container a8aa2558507f00520634d6ce1fcc4a217e2b1d96137f988721f82cb49ccddaac: Status 404 returned error can't find the container with id a8aa2558507f00520634d6ce1fcc4a217e2b1d96137f988721f82cb49ccddaac
Oct 31 12:41:25.477438 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:25.477403    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96d6e803-8ckml"]
Oct 31 12:41:26.096721 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:26.096634    5060 generic.go:356] "Generic (PLEG): container finished" podID="b1b6f29e-44fd-49de-a88e-ae6cf2efcfef" containerID="1ddef93f9665b2e472ee2ef78c968665ca2dc31a0208f750b3cfb1c9fb258c58" exitCode=0
Oct 31 12:41:26.096721 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:26.096677    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75668bd6-vwl6q" event={"ID":"b1b6f29e-44fd-49de-a88e-ae6cf2efcfef","Type":"ContainerDied","Data":"1ddef93f9665b2e472ee2ef78c968665ca2dc31a0208f750b3cfb1c9fb258c58"}
Oct 31 12:41:26.098729 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:26.098700    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96d6e803-8ckml" event={"ID":"48fd0a20-b1ec-47cd-9f68-4638965a5acc","Type":"ContainerStarted","Data":"df534668b3fcb84b8d19acb83414417e4ceefa93fe467899e36586ec15857337"}
Oct 31 12:41:26.098879 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:26.098750    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96d6e803-8ckml" event={"ID":"48fd0a20-b1ec-47cd-9f68-4638965a5acc","Type":"ContainerStarted","Data":"a8aa2558507f00520634d6ce1fcc4a217e2b1d96137f988721f82cb49ccddaac"}
Oct 31 12:41:26.135466 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:26.135409    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96d6e803-8ckml" podStartSLOduration=2.13539589 podStartE2EDuration="2.13539589s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:41:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:41:26.132568907 +0000 UTC m=+849.235636616" watchObservedRunningTime="2026-10-31 12:41:26.13539589 +0000 UTC m=+849.238463661"
Oct 31 12:41:26.186679 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:26.186637    5060 patch_prober.go:29] interesting pod/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-kmx7z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body=
Oct 31 12:41:26.186845 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:26.186696    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-kmx7z" podUID="9b8d44ec-1b81-4ca2-9337-6294db29bb5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused"
Oct 31 12:41:27.111972 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:27.111938    5060 generic.go:356] "Generic (PLEG): container finished" podID="48fd0a20-b1ec-47cd-9f68-4638965a5acc" containerID="df534668b3fcb84b8d19acb83414417e4ceefa93fe467899e36586ec15857337" exitCode=0
Oct 31 12:41:27.112437 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:27.112025    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96d6e803-8ckml" event={"ID":"48fd0a20-b1ec-47cd-9f68-4638965a5acc","Type":"ContainerDied","Data":"df534668b3fcb84b8d19acb83414417e4ceefa93fe467899e36586ec15857337"}
Oct 31 12:41:27.183348 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:27.182912    5060 patch_prober.go:29] interesting pod/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-kmx7z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body=
Oct 31 12:41:27.183348 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:27.182973    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-kmx7z" podUID="9b8d44ec-1b81-4ca2-9337-6294db29bb5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused"
Oct 31 12:41:27.343148 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:27.343002    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75668bd6-vwl6q"
Oct 31 12:41:27.498277 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:27.498169    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1b6f29e-44fd-49de-a88e-ae6cf2efcfef-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1b6f29e-44fd-49de-a88e-ae6cf2efcfef-maas-api-openshift-ca\") pod \"b1b6f29e-44fd-49de-a88e-ae6cf2efcfef\" (UID: \"b1b6f29e-44fd-49de-a88e-ae6cf2efcfef\") "
Oct 31 12:41:27.498277 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:27.498271    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1b6f29e-44fd-49de-a88e-ae6cf2efcfef-kube-api-access-98p2t\" (UniqueName: \"kubernetes.io/projected/b1b6f29e-44fd-49de-a88e-ae6cf2efcfef-kube-api-access-98p2t\") pod \"b1b6f29e-44fd-49de-a88e-ae6cf2efcfef\" (UID: \"b1b6f29e-44fd-49de-a88e-ae6cf2efcfef\") "
Oct 31 12:41:27.498567 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:27.498353    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b1b6f29e-44fd-49de-a88e-ae6cf2efcfef-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1b6f29e-44fd-49de-a88e-ae6cf2efcfef-maas-api-xks-ca\") pod \"b1b6f29e-44fd-49de-a88e-ae6cf2efcfef\" (UID: \"b1b6f29e-44fd-49de-a88e-ae6cf2efcfef\") "
Oct 31 12:41:27.499075 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:27.499045    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1b6f29e-44fd-49de-a88e-ae6cf2efcfef-maas-api-openshift-ca" pod "b1b6f29e-44fd-49de-a88e-ae6cf2efcfef" (UID: "b1b6f29e-44fd-49de-a88e-ae6cf2efcfef"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:41:27.500854 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:27.500822    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1b6f29e-44fd-49de-a88e-ae6cf2efcfef-kube-api-access-98p2t" pod "b1b6f29e-44fd-49de-a88e-ae6cf2efcfef" (UID: "b1b6f29e-44fd-49de-a88e-ae6cf2efcfef"). InnerVolumeSpecName "kube-api-access-98p2t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:41:27.500971 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:27.500919    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1b6f29e-44fd-49de-a88e-ae6cf2efcfef-maas-api-xks-ca" pod "b1b6f29e-44fd-49de-a88e-ae6cf2efcfef" (UID: "b1b6f29e-44fd-49de-a88e-ae6cf2efcfef"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:41:27.600226 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:27.600177    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98p2t\" (UniqueName: \"kubernetes.io/projected/b1b6f29e-44fd-49de-a88e-ae6cf2efcfef-kube-api-access-98p2t\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:41:27.600226 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:27.600222    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1b6f29e-44fd-49de-a88e-ae6cf2efcfef-maas-api-xks-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:41:27.600476 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:27.600237    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1b6f29e-44fd-49de-a88e-ae6cf2efcfef-maas-api-openshift-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:41:28.123857 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:28.123773    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75668bd6-vwl6q"
Oct 31 12:41:28.123857 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:28.123818    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75668bd6-vwl6q" event={"ID":"b1b6f29e-44fd-49de-a88e-ae6cf2efcfef","Type":"ContainerDied","Data":"cda9d8a0265f4fff259ab967919f458ca7f2b54682f31537efb2d25ff00fd8ab"}
Oct 31 12:41:28.124387 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:28.123865    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cda9d8a0265f4fff259ab967919f458ca7f2b54682f31537efb2d25ff00fd8ab"
Oct 31 12:41:28.192526 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:28.182980    5060 patch_prober.go:29] interesting pod/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-kmx7z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body=
Oct 31 12:41:28.192526 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:28.183054    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-kmx7z" podUID="9b8d44ec-1b81-4ca2-9337-6294db29bb5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused"
Oct 31 12:41:28.343972 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:28.343943    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96d6e803-8ckml"
Oct 31 12:41:28.422763 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:28.422675    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/48fd0a20-b1ec-47cd-9f68-4638965a5acc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48fd0a20-b1ec-47cd-9f68-4638965a5acc-maas-api-xks-ca\") pod \"48fd0a20-b1ec-47cd-9f68-4638965a5acc\" (UID: \"48fd0a20-b1ec-47cd-9f68-4638965a5acc\") "
Oct 31 12:41:28.422763 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:28.422719    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48fd0a20-b1ec-47cd-9f68-4638965a5acc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48fd0a20-b1ec-47cd-9f68-4638965a5acc-maas-api-openshift-ca\") pod \"48fd0a20-b1ec-47cd-9f68-4638965a5acc\" (UID: \"48fd0a20-b1ec-47cd-9f68-4638965a5acc\") "
Oct 31 12:41:28.423004 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:28.422827    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48fd0a20-b1ec-47cd-9f68-4638965a5acc-kube-api-access-wt2jw\" (UniqueName: \"kubernetes.io/projected/48fd0a20-b1ec-47cd-9f68-4638965a5acc-kube-api-access-wt2jw\") pod \"48fd0a20-b1ec-47cd-9f68-4638965a5acc\" (UID: \"48fd0a20-b1ec-47cd-9f68-4638965a5acc\") "
Oct 31 12:41:28.423395 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:28.423365    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48fd0a20-b1ec-47cd-9f68-4638965a5acc-maas-api-openshift-ca" pod "48fd0a20-b1ec-47cd-9f68-4638965a5acc" (UID: "48fd0a20-b1ec-47cd-9f68-4638965a5acc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:41:28.425796 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:28.425765    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48fd0a20-b1ec-47cd-9f68-4638965a5acc-maas-api-xks-ca" pod "48fd0a20-b1ec-47cd-9f68-4638965a5acc" (UID: "48fd0a20-b1ec-47cd-9f68-4638965a5acc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:41:28.425796 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:28.425787    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48fd0a20-b1ec-47cd-9f68-4638965a5acc-kube-api-access-wt2jw" pod "48fd0a20-b1ec-47cd-9f68-4638965a5acc" (UID: "48fd0a20-b1ec-47cd-9f68-4638965a5acc"). InnerVolumeSpecName "kube-api-access-wt2jw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:41:28.524219 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:28.524182    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48fd0a20-b1ec-47cd-9f68-4638965a5acc-maas-api-xks-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:41:28.524219 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:28.524217    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48fd0a20-b1ec-47cd-9f68-4638965a5acc-maas-api-openshift-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:41:28.524219 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:28.524229    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wt2jw\" (UniqueName: \"kubernetes.io/projected/48fd0a20-b1ec-47cd-9f68-4638965a5acc-kube-api-access-wt2jw\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:41:29.134817 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:29.134785    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96d6e803-8ckml" event={"ID":"48fd0a20-b1ec-47cd-9f68-4638965a5acc","Type":"ContainerDied","Data":"a8aa2558507f00520634d6ce1fcc4a217e2b1d96137f988721f82cb49ccddaac"}
Oct 31 12:41:29.135216 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:29.134833    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a8aa2558507f00520634d6ce1fcc4a217e2b1d96137f988721f82cb49ccddaac"
Oct 31 12:41:29.135216 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:29.134879    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96d6e803-8ckml"
Oct 31 12:41:29.188521 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:29.187570    5060 patch_prober.go:29] interesting pod/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-kmx7z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body=
Oct 31 12:41:29.188521 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:29.187629    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-kmx7z" podUID="9b8d44ec-1b81-4ca2-9337-6294db29bb5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused"
Oct 31 12:41:29.923549 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:41:29.923481    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7phb" containerName="extractor"
Oct 31 12:41:30.187569 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:30.186006    5060 patch_prober.go:29] interesting pod/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-kmx7z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body=
Oct 31 12:41:30.187569 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:30.186081    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-kmx7z" podUID="9b8d44ec-1b81-4ca2-9337-6294db29bb5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused"
Oct 31 12:41:31.182888 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:31.182847    5060 patch_prober.go:29] interesting pod/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-kmx7z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body=
Oct 31 12:41:31.183060 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:31.182928    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-kmx7z" podUID="9b8d44ec-1b81-4ca2-9337-6294db29bb5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused"
Oct 31 12:41:32.393917 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:32.393880    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-kmx7z"
Oct 31 12:41:32.395514 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:32.395464    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-kmx7z"
Oct 31 12:41:32.396362 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:32.396341    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-kmx7z"
Oct 31 12:41:34.201364 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:34.201329    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da149b49-bhdgd"]
Oct 31 12:41:34.202847 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:34.202801    5060 cpu_manager.go:414] "containerMap: removing container" podUID="b1b6f29e-44fd-49de-a88e-ae6cf2efcfef" containerName="revoke-keys"
Oct 31 12:41:34.203022 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:34.203007    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1b6f29e-44fd-49de-a88e-ae6cf2efcfef" containerName="revoke-keys"
Oct 31 12:41:34.203120 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:34.203109    5060 cpu_manager.go:414] "containerMap: removing container" podUID="48fd0a20-b1ec-47cd-9f68-4638965a5acc" containerName="revoke-keys"
Oct 31 12:41:34.203194 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:34.203182    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48fd0a20-b1ec-47cd-9f68-4638965a5acc" containerName="revoke-keys"
Oct 31 12:41:34.203540 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:34.203520    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1b6f29e-44fd-49de-a88e-ae6cf2efcfef" containerName="revoke-keys"
Oct 31 12:41:34.203670 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:34.203659    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="48fd0a20-b1ec-47cd-9f68-4638965a5acc" containerName="revoke-keys"
Oct 31 12:41:34.218525 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:34.217929    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da149b49-bhdgd"]
Oct 31 12:41:34.218525 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:34.217968    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da149b49-bhdgd"
Oct 31 12:41:34.223439 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:34.223396    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7wsmb\""
Oct 31 12:41:34.228822 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:34.228798    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 12:41:34.256478 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:34.256433    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 12:41:34.259034 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:34.258975    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 12:41:34.314846 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:34.314814    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d8fa218-8592-4a2a-8efb-db09efab23ab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-da149b49-bhdgd\" (UID: \"8d8fa218-8592-4a2a-8efb-db09efab23ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da149b49-bhdgd"
Oct 31 12:41:34.315041 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:34.315016    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d8fa218-8592-4a2a-8efb-db09efab23ab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-da149b49-bhdgd\" (UID: \"8d8fa218-8592-4a2a-8efb-db09efab23ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da149b49-bhdgd"
Oct 31 12:41:34.315112 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:34.315078    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88vsf\" (UniqueName: \"kubernetes.io/projected/8d8fa218-8592-4a2a-8efb-db09efab23ab-kube-api-access-88vsf\") pod \"maas-api-revoke-sub-da149b49-bhdgd\" (UID: \"8d8fa218-8592-4a2a-8efb-db09efab23ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da149b49-bhdgd"
Oct 31 12:41:34.416075 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:34.416041    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d8fa218-8592-4a2a-8efb-db09efab23ab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-da149b49-bhdgd\" (UID: \"8d8fa218-8592-4a2a-8efb-db09efab23ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da149b49-bhdgd"
Oct 31 12:41:34.416253 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:34.416132    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d8fa218-8592-4a2a-8efb-db09efab23ab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-da149b49-bhdgd\" (UID: \"8d8fa218-8592-4a2a-8efb-db09efab23ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da149b49-bhdgd"
Oct 31 12:41:34.416253 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:34.416156    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88vsf\" (UniqueName: \"kubernetes.io/projected/8d8fa218-8592-4a2a-8efb-db09efab23ab-kube-api-access-88vsf\") pod \"maas-api-revoke-sub-da149b49-bhdgd\" (UID: \"8d8fa218-8592-4a2a-8efb-db09efab23ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da149b49-bhdgd"
Oct 31 12:41:34.416757 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:34.416733    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d8fa218-8592-4a2a-8efb-db09efab23ab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-da149b49-bhdgd\" (UID: \"8d8fa218-8592-4a2a-8efb-db09efab23ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da149b49-bhdgd"
Oct 31 12:41:34.418526 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:34.418479    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d8fa218-8592-4a2a-8efb-db09efab23ab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-da149b49-bhdgd\" (UID: \"8d8fa218-8592-4a2a-8efb-db09efab23ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da149b49-bhdgd"
Oct 31 12:41:34.428424 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:34.428400    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88vsf\" (UniqueName: \"kubernetes.io/projected/8d8fa218-8592-4a2a-8efb-db09efab23ab-kube-api-access-88vsf\") pod \"maas-api-revoke-sub-da149b49-bhdgd\" (UID: \"8d8fa218-8592-4a2a-8efb-db09efab23ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da149b49-bhdgd"
Oct 31 12:41:34.578748 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:34.578714    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da149b49-bhdgd"
Oct 31 12:41:34.870926 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:41:34.870890    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d8fa218_8592_4a2a_8efb_db09efab23ab.slice/crio-97e37c9cae5f130c62f28a7f51ee1ebdce0ceedbcfae56a5fe62e905a61ee4b2 WatchSource:0}: Error finding container 97e37c9cae5f130c62f28a7f51ee1ebdce0ceedbcfae56a5fe62e905a61ee4b2: Status 404 returned error can't find the container with id 97e37c9cae5f130c62f28a7f51ee1ebdce0ceedbcfae56a5fe62e905a61ee4b2
Oct 31 12:41:34.874344 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:34.874037    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da149b49-bhdgd"]
Oct 31 12:41:35.225162 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:35.225125    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da149b49-bhdgd" event={"ID":"8d8fa218-8592-4a2a-8efb-db09efab23ab","Type":"ContainerStarted","Data":"e0c4693aff8c581be0c33040fe2aaf18843e8f01e4f5f5c9ce79dd1ca7ebebd0"}
Oct 31 12:41:35.225664 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:35.225199    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da149b49-bhdgd" event={"ID":"8d8fa218-8592-4a2a-8efb-db09efab23ab","Type":"ContainerStarted","Data":"97e37c9cae5f130c62f28a7f51ee1ebdce0ceedbcfae56a5fe62e905a61ee4b2"}
Oct 31 12:41:35.248321 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:35.248271    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da149b49-bhdgd" podStartSLOduration=1.24825688 podStartE2EDuration="1.24825688s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:41:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:41:35.242574909 +0000 UTC m=+858.345642632" watchObservedRunningTime="2026-10-31 12:41:35.24825688 +0000 UTC m=+858.351324610"
Oct 31 12:41:36.244162 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:36.244048    5060 generic.go:356] "Generic (PLEG): container finished" podID="8d8fa218-8592-4a2a-8efb-db09efab23ab" containerID="e0c4693aff8c581be0c33040fe2aaf18843e8f01e4f5f5c9ce79dd1ca7ebebd0" exitCode=0
Oct 31 12:41:36.244687 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:36.244155    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da149b49-bhdgd" event={"ID":"8d8fa218-8592-4a2a-8efb-db09efab23ab","Type":"ContainerDied","Data":"e0c4693aff8c581be0c33040fe2aaf18843e8f01e4f5f5c9ce79dd1ca7ebebd0"}
Oct 31 12:41:37.421526 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:37.421499    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da149b49-bhdgd"
Oct 31 12:41:37.498139 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:37.498101    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-njl9z"]
Oct 31 12:41:37.499172 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:37.499153    5060 cpu_manager.go:414] "containerMap: removing container" podUID="8d8fa218-8592-4a2a-8efb-db09efab23ab" containerName="revoke-keys"
Oct 31 12:41:37.499225 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:37.499175    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d8fa218-8592-4a2a-8efb-db09efab23ab" containerName="revoke-keys"
Oct 31 12:41:37.499354 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:37.499342    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d8fa218-8592-4a2a-8efb-db09efab23ab" containerName="revoke-keys"
Oct 31 12:41:37.503760 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:37.503705    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-njl9z"
Oct 31 12:41:37.546734 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:37.546702    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-njl9z"]
Oct 31 12:41:37.555250 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:37.555218    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-56grg"]
Oct 31 12:41:37.556365 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:37.556335    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d8fa218-8592-4a2a-8efb-db09efab23ab-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d8fa218-8592-4a2a-8efb-db09efab23ab-maas-api-xks-ca\") pod \"8d8fa218-8592-4a2a-8efb-db09efab23ab\" (UID: \"8d8fa218-8592-4a2a-8efb-db09efab23ab\") "
Oct 31 12:41:37.556580 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:37.556445    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d8fa218-8592-4a2a-8efb-db09efab23ab-kube-api-access-88vsf\" (UniqueName: \"kubernetes.io/projected/8d8fa218-8592-4a2a-8efb-db09efab23ab-kube-api-access-88vsf\") pod \"8d8fa218-8592-4a2a-8efb-db09efab23ab\" (UID: \"8d8fa218-8592-4a2a-8efb-db09efab23ab\") "
Oct 31 12:41:37.556776 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:37.556597    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d8fa218-8592-4a2a-8efb-db09efab23ab-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d8fa218-8592-4a2a-8efb-db09efab23ab-maas-api-openshift-ca\") pod \"8d8fa218-8592-4a2a-8efb-db09efab23ab\" (UID: \"8d8fa218-8592-4a2a-8efb-db09efab23ab\") "
Oct 31 12:41:37.557577 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:37.557545    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d8fa218-8592-4a2a-8efb-db09efab23ab-maas-api-openshift-ca" pod "8d8fa218-8592-4a2a-8efb-db09efab23ab" (UID: "8d8fa218-8592-4a2a-8efb-db09efab23ab"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:41:37.560365 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:37.560341    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d8fa218-8592-4a2a-8efb-db09efab23ab-kube-api-access-88vsf" pod "8d8fa218-8592-4a2a-8efb-db09efab23ab" (UID: "8d8fa218-8592-4a2a-8efb-db09efab23ab"). InnerVolumeSpecName "kube-api-access-88vsf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:41:37.564011 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:37.563484    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-56grg"
Oct 31 12:41:37.568108 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:37.568000    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d8fa218-8592-4a2a-8efb-db09efab23ab-maas-api-xks-ca" pod "8d8fa218-8592-4a2a-8efb-db09efab23ab" (UID: "8d8fa218-8592-4a2a-8efb-db09efab23ab"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:41:37.587938 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:37.582475    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-56grg"]
Oct 31 12:41:37.654210 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:37.654173    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-f8gp2"]
Oct 31 12:41:37.660359 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:37.660251    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-f8gp2"
Oct 31 12:41:37.660692 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:37.659536    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29b176fa-229b-433e-928f-41daf33d1f37-workload-socket\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-njl9z\" (UID: \"29b176fa-229b-433e-928f-41daf33d1f37\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-njl9z"
Oct 31 12:41:37.665602 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:37.665569    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f43cdb3b-9bd6-46a7-a14c-feec07c2c755-credential-socket\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-56grg\" (UID: \"f43cdb3b-9bd6-46a7-a14c-feec07c2c755\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-56grg"
Oct 31 12:41:37.665703 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:37.665616    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29b176fa-229b-433e-928f-41daf33d1f37-istiod-ca-cert\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-njl9z\" (UID: \"29b176fa-229b-433e-928f-41daf33d1f37\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-njl9z"
Oct 31 12:41:37.665779 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:37.665697    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29b176fa-229b-433e-928f-41daf33d1f37-istio-podinfo\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-njl9z\" (UID: \"29b176fa-229b-433e-928f-41daf33d1f37\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-njl9z"
Oct 31 12:41:37.665779 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:37.665736    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f43cdb3b-9bd6-46a7-a14c-feec07c2c755-istio-data\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-56grg\" (UID: \"f43cdb3b-9bd6-46a7-a14c-feec07c2c755\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-56grg"
Oct 31 12:41:37.665890 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:37.665835    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f43cdb3b-9bd6-46a7-a14c-feec07c2c755-workload-certs\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-56grg\" (UID: \"f43cdb3b-9bd6-46a7-a14c-feec07c2c755\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-56grg"
Oct 31 12:41:37.665890 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:37.665868    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29b176fa-229b-433e-928f-41daf33d1f37-istio-envoy\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-njl9z\" (UID: \"29b176fa-229b-433e-928f-41daf33d1f37\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-njl9z"
Oct 31 12:41:37.665987 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:37.665916    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f43cdb3b-9bd6-46a7-a14c-feec07c2c755-workload-socket\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-56grg\" (UID: \"f43cdb3b-9bd6-46a7-a14c-feec07c2c755\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-56grg"
Oct 31 12:41:37.666044 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:37.665985    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f43cdb3b-9bd6-46a7-a14c-feec07c2c755-istio-podinfo\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-56grg\" (UID: \"f43cdb3b-9bd6-46a7-a14c-feec07c2c755\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-56grg"
Oct 31 12:41:37.666044 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:37.666033    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f43cdb3b-9bd6-46a7-a14c-feec07c2c755-istio-token\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-56grg\" (UID: \"f43cdb3b-9bd6-46a7-a14c-feec07c2c755\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-56grg"
Oct 31 12:41:37.666141 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:37.666064    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9rjz\" (UniqueName: \"kubernetes.io/projected/29b176fa-229b-433e-928f-41daf33d1f37-kube-api-access-g9rjz\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-njl9z\" (UID: \"29b176fa-229b-433e-928f-41daf33d1f37\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-njl9z"
Oct 31 12:41:37.666141 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:37.666093    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f43cdb3b-9bd6-46a7-a14c-feec07c2c755-istio-envoy\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-56grg\" (UID: \"f43cdb3b-9bd6-46a7-a14c-feec07c2c755\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-56grg"
Oct 31 12:41:37.666141 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:37.666125    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7ksj\" (UniqueName: \"kubernetes.io/projected/f43cdb3b-9bd6-46a7-a14c-feec07c2c755-kube-api-access-f7ksj\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-56grg\" (UID: \"f43cdb3b-9bd6-46a7-a14c-feec07c2c755\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-56grg"
Oct 31 12:41:37.666298 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:37.666155    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29b176fa-229b-433e-928f-41daf33d1f37-credential-socket\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-njl9z\" (UID: \"29b176fa-229b-433e-928f-41daf33d1f37\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-njl9z"
Oct 31 12:41:37.666298 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:37.666200    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29b176fa-229b-433e-928f-41daf33d1f37-workload-certs\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-njl9z\" (UID: \"29b176fa-229b-433e-928f-41daf33d1f37\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-njl9z"
Oct 31 12:41:37.666298 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:37.666251    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29b176fa-229b-433e-928f-41daf33d1f37-istio-data\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-njl9z\" (UID: \"29b176fa-229b-433e-928f-41daf33d1f37\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-njl9z"
Oct 31 12:41:37.666298 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:37.666290    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f43cdb3b-9bd6-46a7-a14c-feec07c2c755-istiod-ca-cert\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-56grg\" (UID: \"f43cdb3b-9bd6-46a7-a14c-feec07c2c755\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-56grg"
Oct 31 12:41:37.666511 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:37.666345    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29b176fa-229b-433e-928f-41daf33d1f37-istio-token\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-njl9z\" (UID: \"29b176fa-229b-433e-928f-41daf33d1f37\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-njl9z"
Oct 31 12:41:37.666511 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:37.666456    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88vsf\" (UniqueName: \"kubernetes.io/projected/8d8fa218-8592-4a2a-8efb-db09efab23ab-kube-api-access-88vsf\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:41:37.666511 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:37.666475    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d8fa218-8592-4a2a-8efb-db09efab23ab-maas-api-openshift-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:41:37.666511 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:37.666509    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d8fa218-8592-4a2a-8efb-db09efab23ab-maas-api-xks-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:41:37.701611 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:37.701576    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-f8gp2"]
Oct 31 12:41:37.720453 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:37.720418    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-bbjl7"]
Oct 31 12:41:37.731588 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:37.731557    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-bbjl7"
Oct 31 12:41:37.774548 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:37.766626    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-bbjl7"]
Oct 31 12:41:37.774548 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:37.768258    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f43cdb3b-9bd6-46a7-a14c-feec07c2c755-workload-socket\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-56grg\" (UID: \"f43cdb3b-9bd6-46a7-a14c-feec07c2c755\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-56grg"
Oct 31 12:41:37.774548 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:37.768333    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f43cdb3b-9bd6-46a7-a14c-feec07c2c755-istio-podinfo\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-56grg\" (UID: \"f43cdb3b-9bd6-46a7-a14c-feec07c2c755\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-56grg"
Oct 31 12:41:37.774548 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:37.768366    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df05433b-b6c1-4bd0-88a1-11e769461d69-istiod-ca-cert\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-f8gp2\" (UID: \"df05433b-b6c1-4bd0-88a1-11e769461d69\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-f8gp2"
Oct 31 12:41:37.774548 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:37.768411    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df05433b-b6c1-4bd0-88a1-11e769461d69-credential-socket\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-f8gp2\" (UID: \"df05433b-b6c1-4bd0-88a1-11e769461d69\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-f8gp2"
Oct 31 12:41:37.774548 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:37.768443    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f43cdb3b-9bd6-46a7-a14c-feec07c2c755-istio-token\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-56grg\" (UID: \"f43cdb3b-9bd6-46a7-a14c-feec07c2c755\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-56grg"
Oct 31 12:41:37.774548 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:37.768475    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9rjz\" (UniqueName: \"kubernetes.io/projected/29b176fa-229b-433e-928f-41daf33d1f37-kube-api-access-g9rjz\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-njl9z\" (UID: \"29b176fa-229b-433e-928f-41daf33d1f37\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-njl9z"
Oct 31 12:41:37.774548 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:37.768526    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pthhn\" (UniqueName: \"kubernetes.io/projected/df05433b-b6c1-4bd0-88a1-11e769461d69-kube-api-access-pthhn\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-f8gp2\" (UID: \"df05433b-b6c1-4bd0-88a1-11e769461d69\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-f8gp2"
Oct 31 12:41:37.774548 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:37.768574    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f43cdb3b-9bd6-46a7-a14c-feec07c2c755-istio-envoy\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-56grg\" (UID: \"f43cdb3b-9bd6-46a7-a14c-feec07c2c755\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-56grg"
Oct 31 12:41:37.774548 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:37.768606    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7ksj\" (UniqueName: \"kubernetes.io/projected/f43cdb3b-9bd6-46a7-a14c-feec07c2c755-kube-api-access-f7ksj\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-56grg\" (UID: \"f43cdb3b-9bd6-46a7-a14c-feec07c2c755\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-56grg"
Oct 31 12:41:37.774548 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:37.768637    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29b176fa-229b-433e-928f-41daf33d1f37-credential-socket\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-njl9z\" (UID: \"29b176fa-229b-433e-928f-41daf33d1f37\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-njl9z"
Oct 31 12:41:37.774548 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:37.768679    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29b176fa-229b-433e-928f-41daf33d1f37-workload-certs\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-njl9z\" (UID: \"29b176fa-229b-433e-928f-41daf33d1f37\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-njl9z"
Oct 31 12:41:37.774548 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:37.768725    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29b176fa-229b-433e-928f-41daf33d1f37-istio-data\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-njl9z\" (UID: \"29b176fa-229b-433e-928f-41daf33d1f37\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-njl9z"
Oct 31 12:41:37.774548 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:37.768761    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f43cdb3b-9bd6-46a7-a14c-feec07c2c755-istiod-ca-cert\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-56grg\" (UID: \"f43cdb3b-9bd6-46a7-a14c-feec07c2c755\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-56grg"
Oct 31 12:41:37.774548 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:37.768789    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df05433b-b6c1-4bd0-88a1-11e769461d69-istio-envoy\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-f8gp2\" (UID: \"df05433b-b6c1-4bd0-88a1-11e769461d69\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-f8gp2"
Oct 31 12:41:37.774548 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:37.768820    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df05433b-b6c1-4bd0-88a1-11e769461d69-workload-socket\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-f8gp2\" (UID: \"df05433b-b6c1-4bd0-88a1-11e769461d69\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-f8gp2"
Oct 31 12:41:37.774548 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:37.768868    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29b176fa-229b-433e-928f-41daf33d1f37-istio-token\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-njl9z\" (UID: \"29b176fa-229b-433e-928f-41daf33d1f37\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-njl9z"
Oct 31 12:41:37.774548 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:37.768899    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df05433b-b6c1-4bd0-88a1-11e769461d69-istio-token\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-f8gp2\" (UID: \"df05433b-b6c1-4bd0-88a1-11e769461d69\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-f8gp2"
Oct 31 12:41:37.774548 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:37.768970    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29b176fa-229b-433e-928f-41daf33d1f37-workload-socket\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-njl9z\" (UID: \"29b176fa-229b-433e-928f-41daf33d1f37\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-njl9z"
Oct 31 12:41:37.774548 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:37.769000    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f43cdb3b-9bd6-46a7-a14c-feec07c2c755-credential-socket\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-56grg\" (UID: \"f43cdb3b-9bd6-46a7-a14c-feec07c2c755\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-56grg"
Oct 31 12:41:37.774548 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:37.769025    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29b176fa-229b-433e-928f-41daf33d1f37-istiod-ca-cert\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-njl9z\" (UID: \"29b176fa-229b-433e-928f-41daf33d1f37\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-njl9z"
Oct 31 12:41:37.774548 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:37.769080    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29b176fa-229b-433e-928f-41daf33d1f37-istio-podinfo\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-njl9z\" (UID: \"29b176fa-229b-433e-928f-41daf33d1f37\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-njl9z"
Oct 31 12:41:37.774548 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:37.769120    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f43cdb3b-9bd6-46a7-a14c-feec07c2c755-istio-data\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-56grg\" (UID: \"f43cdb3b-9bd6-46a7-a14c-feec07c2c755\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-56grg"
Oct 31 12:41:37.774548 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:37.769144    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df05433b-b6c1-4bd0-88a1-11e769461d69-workload-certs\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-f8gp2\" (UID: \"df05433b-b6c1-4bd0-88a1-11e769461d69\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-f8gp2"
Oct 31 12:41:37.774548 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:37.769263    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f43cdb3b-9bd6-46a7-a14c-feec07c2c755-workload-certs\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-56grg\" (UID: \"f43cdb3b-9bd6-46a7-a14c-feec07c2c755\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-56grg"
Oct 31 12:41:37.774548 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:37.769294    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df05433b-b6c1-4bd0-88a1-11e769461d69-istio-data\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-f8gp2\" (UID: \"df05433b-b6c1-4bd0-88a1-11e769461d69\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-f8gp2"
Oct 31 12:41:37.774548 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:37.769319    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df05433b-b6c1-4bd0-88a1-11e769461d69-istio-podinfo\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-f8gp2\" (UID: \"df05433b-b6c1-4bd0-88a1-11e769461d69\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-f8gp2"
Oct 31 12:41:37.774548 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:37.769351    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29b176fa-229b-433e-928f-41daf33d1f37-istio-envoy\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-njl9z\" (UID: \"29b176fa-229b-433e-928f-41daf33d1f37\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-njl9z"
Oct 31 12:41:37.774548 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:37.770636    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29b176fa-229b-433e-928f-41daf33d1f37-istiod-ca-cert\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-njl9z\" (UID: \"29b176fa-229b-433e-928f-41daf33d1f37\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-njl9z"
Oct 31 12:41:37.774548 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:37.771208    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29b176fa-229b-433e-928f-41daf33d1f37-workload-socket\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-njl9z\" (UID: \"29b176fa-229b-433e-928f-41daf33d1f37\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-njl9z"
Oct 31 12:41:37.774548 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:37.771520    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f43cdb3b-9bd6-46a7-a14c-feec07c2c755-credential-socket\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-56grg\" (UID: \"f43cdb3b-9bd6-46a7-a14c-feec07c2c755\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-56grg"
Oct 31 12:41:37.774548 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:37.771970    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f43cdb3b-9bd6-46a7-a14c-feec07c2c755-workload-socket\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-56grg\" (UID: \"f43cdb3b-9bd6-46a7-a14c-feec07c2c755\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-56grg"
Oct 31 12:41:37.783164 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:37.776828    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f43cdb3b-9bd6-46a7-a14c-feec07c2c755-istiod-ca-cert\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-56grg\" (UID: \"f43cdb3b-9bd6-46a7-a14c-feec07c2c755\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-56grg"
Oct 31 12:41:37.783164 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:37.778788    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29b176fa-229b-433e-928f-41daf33d1f37-credential-socket\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-njl9z\" (UID: \"29b176fa-229b-433e-928f-41daf33d1f37\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-njl9z"
Oct 31 12:41:37.783164 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:37.782725    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29b176fa-229b-433e-928f-41daf33d1f37-workload-certs\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-njl9z\" (UID: \"29b176fa-229b-433e-928f-41daf33d1f37\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-njl9z"
Oct 31 12:41:37.787095 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:37.785698    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29b176fa-229b-433e-928f-41daf33d1f37-istio-token\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-njl9z\" (UID: \"29b176fa-229b-433e-928f-41daf33d1f37\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-njl9z"
Oct 31 12:41:37.787209 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:37.787126    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f43cdb3b-9bd6-46a7-a14c-feec07c2c755-workload-certs\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-56grg\" (UID: \"f43cdb3b-9bd6-46a7-a14c-feec07c2c755\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-56grg"
Oct 31 12:41:37.787795 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:37.787502    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f43cdb3b-9bd6-46a7-a14c-feec07c2c755-istio-data\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-56grg\" (UID: \"f43cdb3b-9bd6-46a7-a14c-feec07c2c755\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-56grg"
Oct 31 12:41:37.787795 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:37.787727    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29b176fa-229b-433e-928f-41daf33d1f37-istio-data\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-njl9z\" (UID: \"29b176fa-229b-433e-928f-41daf33d1f37\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-njl9z"
Oct 31 12:41:37.788894 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:37.788827    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f43cdb3b-9bd6-46a7-a14c-feec07c2c755-istio-envoy\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-56grg\" (UID: \"f43cdb3b-9bd6-46a7-a14c-feec07c2c755\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-56grg"
Oct 31 12:41:37.790223 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:37.789066    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9rjz\" (UniqueName: \"kubernetes.io/projected/29b176fa-229b-433e-928f-41daf33d1f37-kube-api-access-g9rjz\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-njl9z\" (UID: \"29b176fa-229b-433e-928f-41daf33d1f37\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-njl9z"
Oct 31 12:41:37.791093 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:37.791059    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7ksj\" (UniqueName: \"kubernetes.io/projected/f43cdb3b-9bd6-46a7-a14c-feec07c2c755-kube-api-access-f7ksj\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-56grg\" (UID: \"f43cdb3b-9bd6-46a7-a14c-feec07c2c755\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-56grg"
Oct 31 12:41:37.791093 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:37.791072    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29b176fa-229b-433e-928f-41daf33d1f37-istio-podinfo\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-njl9z\" (UID: \"29b176fa-229b-433e-928f-41daf33d1f37\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-njl9z"
Oct 31 12:41:37.794811 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:37.794785    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29b176fa-229b-433e-928f-41daf33d1f37-istio-envoy\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-njl9z\" (UID: \"29b176fa-229b-433e-928f-41daf33d1f37\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-njl9z"
Oct 31 12:41:37.795909 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:37.795886    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f43cdb3b-9bd6-46a7-a14c-feec07c2c755-istio-podinfo\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-56grg\" (UID: \"f43cdb3b-9bd6-46a7-a14c-feec07c2c755\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-56grg"
Oct 31 12:41:37.799710 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:37.799684    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f43cdb3b-9bd6-46a7-a14c-feec07c2c755-istio-token\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-56grg\" (UID: \"f43cdb3b-9bd6-46a7-a14c-feec07c2c755\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-56grg"
Oct 31 12:41:37.821231 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:37.821197    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-njl9z"
Oct 31 12:41:37.871289 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:37.871189    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09b9b34e-eb53-4829-aed9-abb0b81675d0-credential-socket\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-bbjl7\" (UID: \"09b9b34e-eb53-4829-aed9-abb0b81675d0\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-bbjl7"
Oct 31 12:41:37.871289 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:37.871251    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df05433b-b6c1-4bd0-88a1-11e769461d69-istio-data\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-f8gp2\" (UID: \"df05433b-b6c1-4bd0-88a1-11e769461d69\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-f8gp2"
Oct 31 12:41:37.871476 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:37.871295    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df05433b-b6c1-4bd0-88a1-11e769461d69-istio-podinfo\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-f8gp2\" (UID: \"df05433b-b6c1-4bd0-88a1-11e769461d69\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-f8gp2"
Oct 31 12:41:37.871729 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:37.871614    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09b9b34e-eb53-4829-aed9-abb0b81675d0-istio-token\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-bbjl7\" (UID: \"09b9b34e-eb53-4829-aed9-abb0b81675d0\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-bbjl7"
Oct 31 12:41:37.871729 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:37.871672    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09b9b34e-eb53-4829-aed9-abb0b81675d0-workload-socket\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-bbjl7\" (UID: \"09b9b34e-eb53-4829-aed9-abb0b81675d0\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-bbjl7"
Oct 31 12:41:37.871895 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:37.871776    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df05433b-b6c1-4bd0-88a1-11e769461d69-istiod-ca-cert\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-f8gp2\" (UID: \"df05433b-b6c1-4bd0-88a1-11e769461d69\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-f8gp2"
Oct 31 12:41:37.871895 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:37.871821    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09b9b34e-eb53-4829-aed9-abb0b81675d0-istio-podinfo\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-bbjl7\" (UID: \"09b9b34e-eb53-4829-aed9-abb0b81675d0\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-bbjl7"
Oct 31 12:41:37.871895 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:37.871866    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df05433b-b6c1-4bd0-88a1-11e769461d69-istio-data\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-f8gp2\" (UID: \"df05433b-b6c1-4bd0-88a1-11e769461d69\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-f8gp2"
Oct 31 12:41:37.872040 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:37.871872    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df05433b-b6c1-4bd0-88a1-11e769461d69-credential-socket\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-f8gp2\" (UID: \"df05433b-b6c1-4bd0-88a1-11e769461d69\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-f8gp2"
Oct 31 12:41:37.872040 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:37.871929    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09b9b34e-eb53-4829-aed9-abb0b81675d0-istiod-ca-cert\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-bbjl7\" (UID: \"09b9b34e-eb53-4829-aed9-abb0b81675d0\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-bbjl7"
Oct 31 12:41:37.872040 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:37.871973    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pthhn\" (UniqueName: \"kubernetes.io/projected/df05433b-b6c1-4bd0-88a1-11e769461d69-kube-api-access-pthhn\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-f8gp2\" (UID: \"df05433b-b6c1-4bd0-88a1-11e769461d69\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-f8gp2"
Oct 31 12:41:37.872040 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:37.872013    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsxn6\" (UniqueName: \"kubernetes.io/projected/09b9b34e-eb53-4829-aed9-abb0b81675d0-kube-api-access-wsxn6\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-bbjl7\" (UID: \"09b9b34e-eb53-4829-aed9-abb0b81675d0\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-bbjl7"
Oct 31 12:41:37.872238 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:37.872066    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09b9b34e-eb53-4829-aed9-abb0b81675d0-istio-data\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-bbjl7\" (UID: \"09b9b34e-eb53-4829-aed9-abb0b81675d0\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-bbjl7"
Oct 31 12:41:37.872238 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:37.872127    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df05433b-b6c1-4bd0-88a1-11e769461d69-istio-envoy\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-f8gp2\" (UID: \"df05433b-b6c1-4bd0-88a1-11e769461d69\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-f8gp2"
Oct 31 12:41:37.872238 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:37.872155    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df05433b-b6c1-4bd0-88a1-11e769461d69-credential-socket\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-f8gp2\" (UID: \"df05433b-b6c1-4bd0-88a1-11e769461d69\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-f8gp2"
Oct 31 12:41:37.872238 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:37.872158    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df05433b-b6c1-4bd0-88a1-11e769461d69-workload-socket\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-f8gp2\" (UID: \"df05433b-b6c1-4bd0-88a1-11e769461d69\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-f8gp2"
Oct 31 12:41:37.872238 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:37.872228    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df05433b-b6c1-4bd0-88a1-11e769461d69-istio-token\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-f8gp2\" (UID: \"df05433b-b6c1-4bd0-88a1-11e769461d69\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-f8gp2"
Oct 31 12:41:37.872483 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:37.872317    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09b9b34e-eb53-4829-aed9-abb0b81675d0-istio-envoy\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-bbjl7\" (UID: \"09b9b34e-eb53-4829-aed9-abb0b81675d0\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-bbjl7"
Oct 31 12:41:37.872483 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:37.872385    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df05433b-b6c1-4bd0-88a1-11e769461d69-workload-certs\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-f8gp2\" (UID: \"df05433b-b6c1-4bd0-88a1-11e769461d69\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-f8gp2"
Oct 31 12:41:37.872483 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:37.872401    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df05433b-b6c1-4bd0-88a1-11e769461d69-workload-socket\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-f8gp2\" (UID: \"df05433b-b6c1-4bd0-88a1-11e769461d69\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-f8gp2"
Oct 31 12:41:37.872483 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:37.872417    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09b9b34e-eb53-4829-aed9-abb0b81675d0-workload-certs\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-bbjl7\" (UID: \"09b9b34e-eb53-4829-aed9-abb0b81675d0\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-bbjl7"
Oct 31 12:41:37.873357 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:37.873322    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df05433b-b6c1-4bd0-88a1-11e769461d69-workload-certs\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-f8gp2\" (UID: \"df05433b-b6c1-4bd0-88a1-11e769461d69\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-f8gp2"
Oct 31 12:41:37.873638 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:37.873613    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df05433b-b6c1-4bd0-88a1-11e769461d69-istiod-ca-cert\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-f8gp2\" (UID: \"df05433b-b6c1-4bd0-88a1-11e769461d69\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-f8gp2"
Oct 31 12:41:37.873715 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:37.873656    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df05433b-b6c1-4bd0-88a1-11e769461d69-istio-podinfo\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-f8gp2\" (UID: \"df05433b-b6c1-4bd0-88a1-11e769461d69\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-f8gp2"
Oct 31 12:41:37.876144 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:37.876120    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df05433b-b6c1-4bd0-88a1-11e769461d69-istio-envoy\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-f8gp2\" (UID: \"df05433b-b6c1-4bd0-88a1-11e769461d69\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-f8gp2"
Oct 31 12:41:37.885542 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:37.885483    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df05433b-b6c1-4bd0-88a1-11e769461d69-istio-token\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-f8gp2\" (UID: \"df05433b-b6c1-4bd0-88a1-11e769461d69\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-f8gp2"
Oct 31 12:41:37.886739 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:37.886712    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pthhn\" (UniqueName: \"kubernetes.io/projected/df05433b-b6c1-4bd0-88a1-11e769461d69-kube-api-access-pthhn\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-f8gp2\" (UID: \"df05433b-b6c1-4bd0-88a1-11e769461d69\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-f8gp2"
Oct 31 12:41:37.929511 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:37.929460    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-56grg"
Oct 31 12:41:37.975561 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:37.974358    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09b9b34e-eb53-4829-aed9-abb0b81675d0-istiod-ca-cert\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-bbjl7\" (UID: \"09b9b34e-eb53-4829-aed9-abb0b81675d0\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-bbjl7"
Oct 31 12:41:37.975561 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:37.974427    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsxn6\" (UniqueName: \"kubernetes.io/projected/09b9b34e-eb53-4829-aed9-abb0b81675d0-kube-api-access-wsxn6\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-bbjl7\" (UID: \"09b9b34e-eb53-4829-aed9-abb0b81675d0\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-bbjl7"
Oct 31 12:41:37.975561 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:37.974475    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09b9b34e-eb53-4829-aed9-abb0b81675d0-istio-data\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-bbjl7\" (UID: \"09b9b34e-eb53-4829-aed9-abb0b81675d0\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-bbjl7"
Oct 31 12:41:37.975561 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:37.974617    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09b9b34e-eb53-4829-aed9-abb0b81675d0-istio-envoy\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-bbjl7\" (UID: \"09b9b34e-eb53-4829-aed9-abb0b81675d0\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-bbjl7"
Oct 31 12:41:37.975561 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:37.974679    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09b9b34e-eb53-4829-aed9-abb0b81675d0-workload-certs\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-bbjl7\" (UID: \"09b9b34e-eb53-4829-aed9-abb0b81675d0\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-bbjl7"
Oct 31 12:41:37.975561 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:37.974760    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09b9b34e-eb53-4829-aed9-abb0b81675d0-credential-socket\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-bbjl7\" (UID: \"09b9b34e-eb53-4829-aed9-abb0b81675d0\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-bbjl7"
Oct 31 12:41:37.975561 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:37.974825    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09b9b34e-eb53-4829-aed9-abb0b81675d0-istio-token\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-bbjl7\" (UID: \"09b9b34e-eb53-4829-aed9-abb0b81675d0\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-bbjl7"
Oct 31 12:41:37.975561 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:37.974880    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09b9b34e-eb53-4829-aed9-abb0b81675d0-workload-socket\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-bbjl7\" (UID: \"09b9b34e-eb53-4829-aed9-abb0b81675d0\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-bbjl7"
Oct 31 12:41:37.975561 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:37.974929    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09b9b34e-eb53-4829-aed9-abb0b81675d0-istio-podinfo\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-bbjl7\" (UID: \"09b9b34e-eb53-4829-aed9-abb0b81675d0\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-bbjl7"
Oct 31 12:41:37.982555 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:37.982113    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09b9b34e-eb53-4829-aed9-abb0b81675d0-istio-podinfo\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-bbjl7\" (UID: \"09b9b34e-eb53-4829-aed9-abb0b81675d0\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-bbjl7"
Oct 31 12:41:37.982555 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:37.982397    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09b9b34e-eb53-4829-aed9-abb0b81675d0-workload-certs\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-bbjl7\" (UID: \"09b9b34e-eb53-4829-aed9-abb0b81675d0\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-bbjl7"
Oct 31 12:41:37.984215 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:37.982832    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09b9b34e-eb53-4829-aed9-abb0b81675d0-workload-socket\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-bbjl7\" (UID: \"09b9b34e-eb53-4829-aed9-abb0b81675d0\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-bbjl7"
Oct 31 12:41:37.984215 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:37.983084    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09b9b34e-eb53-4829-aed9-abb0b81675d0-credential-socket\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-bbjl7\" (UID: \"09b9b34e-eb53-4829-aed9-abb0b81675d0\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-bbjl7"
Oct 31 12:41:37.984215 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:37.983152    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09b9b34e-eb53-4829-aed9-abb0b81675d0-istio-data\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-bbjl7\" (UID: \"09b9b34e-eb53-4829-aed9-abb0b81675d0\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-bbjl7"
Oct 31 12:41:37.984215 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:37.983794    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09b9b34e-eb53-4829-aed9-abb0b81675d0-istiod-ca-cert\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-bbjl7\" (UID: \"09b9b34e-eb53-4829-aed9-abb0b81675d0\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-bbjl7"
Oct 31 12:41:37.984215 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:37.983992    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-f8gp2"
Oct 31 12:41:37.984666 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:37.984388    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09b9b34e-eb53-4829-aed9-abb0b81675d0-istio-envoy\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-bbjl7\" (UID: \"09b9b34e-eb53-4829-aed9-abb0b81675d0\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-bbjl7"
Oct 31 12:41:37.996328 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:37.996181    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09b9b34e-eb53-4829-aed9-abb0b81675d0-istio-token\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-bbjl7\" (UID: \"09b9b34e-eb53-4829-aed9-abb0b81675d0\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-bbjl7"
Oct 31 12:41:37.996736 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:37.996711    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsxn6\" (UniqueName: \"kubernetes.io/projected/09b9b34e-eb53-4829-aed9-abb0b81675d0-kube-api-access-wsxn6\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-bbjl7\" (UID: \"09b9b34e-eb53-4829-aed9-abb0b81675d0\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-bbjl7"
Oct 31 12:41:38.019115 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:38.018839    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25afd7d2-4vdmm"]
Oct 31 12:41:38.033437 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:38.030390    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25afd7d2-4vdmm"
Oct 31 12:41:38.050251 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:38.049754    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25afd7d2-4vdmm"]
Oct 31 12:41:38.091228 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:38.091192    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-bbjl7"
Oct 31 12:41:38.138388 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:38.138300    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-njl9z"]
Oct 31 12:41:38.149473 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:38.149427    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:41:38.149631 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:38.149543    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:41:38.149631 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:38.149625    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:41:38.185253 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:38.178256    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9829v\" (UniqueName: \"kubernetes.io/projected/3aa9227a-b604-4644-a5ee-97b5ccc1f33e-kube-api-access-9829v\") pod \"maas-api-revoke-sub-25afd7d2-4vdmm\" (UID: \"3aa9227a-b604-4644-a5ee-97b5ccc1f33e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25afd7d2-4vdmm"
Oct 31 12:41:38.185253 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:38.178375    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3aa9227a-b604-4644-a5ee-97b5ccc1f33e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-25afd7d2-4vdmm\" (UID: \"3aa9227a-b604-4644-a5ee-97b5ccc1f33e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25afd7d2-4vdmm"
Oct 31 12:41:38.185253 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:38.178407    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3aa9227a-b604-4644-a5ee-97b5ccc1f33e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-25afd7d2-4vdmm\" (UID: \"3aa9227a-b604-4644-a5ee-97b5ccc1f33e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25afd7d2-4vdmm"
Oct 31 12:41:38.265980 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:38.264855    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:41:38.265980 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:38.264940    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:41:38.265980 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:38.264967    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:41:38.268598 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:38.267514    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-56grg"]
Oct 31 12:41:38.269482 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:38.269205    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da149b49-bhdgd"
Oct 31 12:41:38.269482 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:38.269393    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da149b49-bhdgd" event={"ID":"8d8fa218-8592-4a2a-8efb-db09efab23ab","Type":"ContainerDied","Data":"97e37c9cae5f130c62f28a7f51ee1ebdce0ceedbcfae56a5fe62e905a61ee4b2"}
Oct 31 12:41:38.269482 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:38.269439    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="97e37c9cae5f130c62f28a7f51ee1ebdce0ceedbcfae56a5fe62e905a61ee4b2"
Oct 31 12:41:38.271473 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:38.271420    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-njl9z" event={"ID":"29b176fa-229b-433e-928f-41daf33d1f37","Type":"ContainerStarted","Data":"0ff6d98fc0142fabd43a8291115c4a043efbd1b829dd1790a1fa3daf5ee1d88c"}
Oct 31 12:41:38.279680 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:38.279653    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9829v\" (UniqueName: \"kubernetes.io/projected/3aa9227a-b604-4644-a5ee-97b5ccc1f33e-kube-api-access-9829v\") pod \"maas-api-revoke-sub-25afd7d2-4vdmm\" (UID: \"3aa9227a-b604-4644-a5ee-97b5ccc1f33e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25afd7d2-4vdmm"
Oct 31 12:41:38.279785 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:38.279749    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3aa9227a-b604-4644-a5ee-97b5ccc1f33e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-25afd7d2-4vdmm\" (UID: \"3aa9227a-b604-4644-a5ee-97b5ccc1f33e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25afd7d2-4vdmm"
Oct 31 12:41:38.279848 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:38.279781    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3aa9227a-b604-4644-a5ee-97b5ccc1f33e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-25afd7d2-4vdmm\" (UID: \"3aa9227a-b604-4644-a5ee-97b5ccc1f33e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25afd7d2-4vdmm"
Oct 31 12:41:38.281105 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:38.280713    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3aa9227a-b604-4644-a5ee-97b5ccc1f33e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-25afd7d2-4vdmm\" (UID: \"3aa9227a-b604-4644-a5ee-97b5ccc1f33e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25afd7d2-4vdmm"
Oct 31 12:41:38.285862 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:38.285391    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3aa9227a-b604-4644-a5ee-97b5ccc1f33e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-25afd7d2-4vdmm\" (UID: \"3aa9227a-b604-4644-a5ee-97b5ccc1f33e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25afd7d2-4vdmm"
Oct 31 12:41:38.297984 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:38.297953    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9829v\" (UniqueName: \"kubernetes.io/projected/3aa9227a-b604-4644-a5ee-97b5ccc1f33e-kube-api-access-9829v\") pod \"maas-api-revoke-sub-25afd7d2-4vdmm\" (UID: \"3aa9227a-b604-4644-a5ee-97b5ccc1f33e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25afd7d2-4vdmm"
Oct 31 12:41:38.356699 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:38.356376    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:41:38.356699 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:38.356461    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:41:38.356699 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:38.356527    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:41:38.360843 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:38.360812    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-f8gp2"]
Oct 31 12:41:38.360986 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:38.360861    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25afd7d2-4vdmm"
Oct 31 12:41:38.412697 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:38.412667    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-bbjl7"]
Oct 31 12:41:38.438168 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:38.437641    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:41:38.438168 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:38.437728    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:41:38.438168 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:38.437772    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:41:38.819888 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:38.819798    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25afd7d2-4vdmm"]
Oct 31 12:41:38.822678 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:41:38.821818    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3aa9227a_b604_4644_a5ee_97b5ccc1f33e.slice/crio-813dddfd8eaf983de2e6d1e605fa2e2b5078e086bddfbdf31e0687c517ffc3fc WatchSource:0}: Error finding container 813dddfd8eaf983de2e6d1e605fa2e2b5078e086bddfbdf31e0687c517ffc3fc: Status 404 returned error can't find the container with id 813dddfd8eaf983de2e6d1e605fa2e2b5078e086bddfbdf31e0687c517ffc3fc
Oct 31 12:41:39.416521 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:39.414253    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-bbjl7" event={"ID":"09b9b34e-eb53-4829-aed9-abb0b81675d0","Type":"ContainerStarted","Data":"07912adfcfe0dac99f69adef566eb21ca4e3b211b8b78efefb0506f1e7a058db"}
Oct 31 12:41:39.416521 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:39.414324    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-bbjl7" event={"ID":"09b9b34e-eb53-4829-aed9-abb0b81675d0","Type":"ContainerStarted","Data":"32887e49b1db12e505b63ee10bc42a6305fe40c25b8050fcca1f35cd7d28d1c9"}
Oct 31 12:41:39.435571 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:39.432628    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-njl9z" event={"ID":"29b176fa-229b-433e-928f-41daf33d1f37","Type":"ContainerStarted","Data":"6fafb77a1324904f28fa906f139fc8ee088f838268da7825463b6b19073c703d"}
Oct 31 12:41:39.491330 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:39.491287    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-56grg" event={"ID":"f43cdb3b-9bd6-46a7-a14c-feec07c2c755","Type":"ContainerStarted","Data":"5dbbe131ebca981cfcb804c34551921810cfbf21be15aa15636d6888f9170867"}
Oct 31 12:41:39.504025 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:39.503991    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-56grg" event={"ID":"f43cdb3b-9bd6-46a7-a14c-feec07c2c755","Type":"ContainerStarted","Data":"9cb68c72dafb6d570b2e653e09a2044fa290142ffbf15863c2addcc364af45e0"}
Oct 31 12:41:39.504277 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:39.504256    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25afd7d2-4vdmm" event={"ID":"3aa9227a-b604-4644-a5ee-97b5ccc1f33e","Type":"ContainerStarted","Data":"813dddfd8eaf983de2e6d1e605fa2e2b5078e086bddfbdf31e0687c517ffc3fc"}
Oct 31 12:41:39.504445 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:39.504429    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-f8gp2" event={"ID":"df05433b-b6c1-4bd0-88a1-11e769461d69","Type":"ContainerStarted","Data":"be887fbca1475bb75a00814d220dfaf71c3ea6f0884c58c1ac59ae8c2b8730cd"}
Oct 31 12:41:39.504618 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:39.504600    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-f8gp2" event={"ID":"df05433b-b6c1-4bd0-88a1-11e769461d69","Type":"ContainerStarted","Data":"86f7523e6a052e987cc7c8a070144118d6f5d57526670bdf9f8fcb7df7308bc9"}
Oct 31 12:41:39.504736 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:39.498366    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-bbjl7" podStartSLOduration=2.498343163 podStartE2EDuration="2.498343163s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:41:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:41:39.472352118 +0000 UTC m=+862.575419834" watchObservedRunningTime="2026-10-31 12:41:39.498343163 +0000 UTC m=+862.601410909"
Oct 31 12:41:39.548922 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:39.548853    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-njl9z" podStartSLOduration=2.5488273599999998 podStartE2EDuration="2.54882736s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:41:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:41:39.528138664 +0000 UTC m=+862.631206377" watchObservedRunningTime="2026-10-31 12:41:39.54882736 +0000 UTC m=+862.651895094"
Oct 31 12:41:39.572524 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:39.572427    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-56grg" podStartSLOduration=2.572403672 podStartE2EDuration="2.572403672s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:41:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:41:39.551788534 +0000 UTC m=+862.654856249" watchObservedRunningTime="2026-10-31 12:41:39.572403672 +0000 UTC m=+862.675471384"
Oct 31 12:41:39.835780 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:39.835744    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-njl9z"
Oct 31 12:41:39.836126 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:39.836103    5060 patch_prober.go:29] interesting pod/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-njl9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body=
Oct 31 12:41:39.836307 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:39.836270    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-njl9z" podUID="29b176fa-229b-433e-928f-41daf33d1f37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused"
Oct 31 12:41:39.940713 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:39.940678    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-56grg"
Oct 31 12:41:39.952193 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:39.952157    5060 patch_prober.go:29] interesting pod/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-56grg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body=
Oct 31 12:41:39.952443 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:39.952417    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-56grg" podUID="f43cdb3b-9bd6-46a7-a14c-feec07c2c755" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused"
Oct 31 12:41:39.993742 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:39.993703    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-f8gp2"
Oct 31 12:41:39.995888 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:39.995857    5060 patch_prober.go:29] interesting pod/e2e-worker-w1-85ce08-openshift-default-77d6458845-f8gp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body=
Oct 31 12:41:39.996133 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:39.996105    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-f8gp2" podUID="df05433b-b6c1-4bd0-88a1-11e769461d69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused"
Oct 31 12:41:40.097452 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:40.097362    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-bbjl7"
Oct 31 12:41:40.097814 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:40.097787    5060 patch_prober.go:29] interesting pod/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-bbjl7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body=
Oct 31 12:41:40.097984 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:40.097959    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-bbjl7" podUID="09b9b34e-eb53-4829-aed9-abb0b81675d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused"
Oct 31 12:41:40.545544 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:40.540300    5060 generic.go:356] "Generic (PLEG): container finished" podID="3aa9227a-b604-4644-a5ee-97b5ccc1f33e" containerID="971de069a484c920446fef53a5cb36ac5446585d39329505dd716333b77d09cb" exitCode=0
Oct 31 12:41:40.545544 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:40.541920    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25afd7d2-4vdmm" event={"ID":"3aa9227a-b604-4644-a5ee-97b5ccc1f33e","Type":"ContainerDied","Data":"971de069a484c920446fef53a5cb36ac5446585d39329505dd716333b77d09cb"}
Oct 31 12:41:40.587530 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:40.585432    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-f8gp2" podStartSLOduration=3.585410576 podStartE2EDuration="3.585410576s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:41:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:41:39.607550765 +0000 UTC m=+862.710618472" watchObservedRunningTime="2026-10-31 12:41:40.585410576 +0000 UTC m=+863.688478287"
Oct 31 12:41:40.730528 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:40.727337    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/simulator-w4-c67de2-kserve-868679cf9b-gqjbx" podUID="e104d7da-1fb2-4c99-a480-b4ae84c4f273" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded"
Oct 31 12:41:40.763614 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:40.762587    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x2qzv" podUID="c247ed2a-aa90-424b-9b7e-4f2936d67e3f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:41:40.790040 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:40.789985    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-2-w5-b25d3b-kserve-76f798d59f-vvzhg" podUID="394ed76c-84ac-4612-9acb-a7e36cb68d89" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:41:40.822848 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:40.822749    5060 patch_prober.go:29] interesting pod/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-njl9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body=
Oct 31 12:41:40.823113 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:40.823068    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-njl9z" podUID="29b176fa-229b-433e-928f-41daf33d1f37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused"
Oct 31 12:41:40.915529 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:40.902009    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/premium-w2-e6c79d-kserve-6468bf46d4-h66f8" podUID="cd6a266f-8d12-4089-83ee-c75c716bf26c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Oct 31 12:41:40.945258 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:40.945210    5060 patch_prober.go:29] interesting pod/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-56grg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body=
Oct 31 12:41:40.945948 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:40.945897    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-56grg" podUID="f43cdb3b-9bd6-46a7-a14c-feec07c2c755" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused"
Oct 31 12:41:41.004557 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:40.993102    5060 patch_prober.go:29] interesting pod/e2e-worker-w1-85ce08-openshift-default-77d6458845-f8gp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body=
Oct 31 12:41:41.004557 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:40.993184    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-f8gp2" podUID="df05433b-b6c1-4bd0-88a1-11e769461d69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused"
Oct 31 12:41:41.266577 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:41.266244    5060 patch_prober.go:29] interesting pod/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-bbjl7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body=
Oct 31 12:41:41.266577 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:41.266304    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-bbjl7" podUID="09b9b34e-eb53-4829-aed9-abb0b81675d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused"
Oct 31 12:41:41.733930 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:41.727864    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/simulator-w4-c67de2-kserve-868679cf9b-gqjbx" podUID="e104d7da-1fb2-4c99-a480-b4ae84c4f273" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded"
Oct 31 12:41:41.790532 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:41.769256    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x2qzv" podUID="c247ed2a-aa90-424b-9b7e-4f2936d67e3f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:41:41.790882 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:41.790849    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c67de2/simulator-w4-c67de2-kserve-868679cf9b-gqjbx"
Oct 31 12:41:41.791056 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:41.791042    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x2qzv"
Oct 31 12:41:41.843546 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:41.831795    5060 patch_prober.go:29] interesting pod/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-njl9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body=
Oct 31 12:41:41.843546 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:41.831859    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-njl9z" podUID="29b176fa-229b-433e-928f-41daf33d1f37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused"
Oct 31 12:41:41.976614 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:41.907076    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/premium-w2-e6c79d-kserve-6468bf46d4-h66f8" podUID="cd6a266f-8d12-4089-83ee-c75c716bf26c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:41:42.115285 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:42.108557    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e6c79d/premium-w2-e6c79d-kserve-6468bf46d4-h66f8"
Oct 31 12:41:42.197516 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:42.117591    5060 patch_prober.go:29] interesting pod/e2e-worker-w1-85ce08-openshift-default-77d6458845-f8gp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body=
Oct 31 12:41:42.340646 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:42.340592    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-f8gp2" podUID="df05433b-b6c1-4bd0-88a1-11e769461d69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused"
Oct 31 12:41:42.366565 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:42.127133    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x2qzv"
Oct 31 12:41:42.660581 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:42.124018    5060 patch_prober.go:29] interesting pod/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-bbjl7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body=
Oct 31 12:41:42.660581 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:42.653073    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-bbjl7" podUID="09b9b34e-eb53-4829-aed9-abb0b81675d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused"
Oct 31 12:41:42.660581 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:42.158664    5060 patch_prober.go:29] interesting pod/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-56grg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body=
Oct 31 12:41:42.660581 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:42.653159    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-56grg" podUID="f43cdb3b-9bd6-46a7-a14c-feec07c2c755" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused"
Oct 31 12:41:42.902463 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:42.901754    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-2-w5-b25d3b-kserve-76f798d59f-vvzhg" podUID="394ed76c-84ac-4612-9acb-a7e36cb68d89" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": context deadline exceeded"
Oct 31 12:41:42.950571 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:42.937038    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/simulator-w4-c67de2-kserve-868679cf9b-gqjbx" podUID="e104d7da-1fb2-4c99-a480-b4ae84c4f273" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:41:43.024520 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:43.022351    5060 patch_prober.go:29] interesting pod/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-njl9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body=
Oct 31 12:41:43.024520 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:43.022416    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-njl9z" podUID="29b176fa-229b-433e-928f-41daf33d1f37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused"
Oct 31 12:41:43.270581 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:43.232064    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/premium-w2-e6c79d-kserve-6468bf46d4-h66f8" podUID="cd6a266f-8d12-4089-83ee-c75c716bf26c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:41:43.375660 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:43.375429    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x2qzv" podUID="c247ed2a-aa90-424b-9b7e-4f2936d67e3f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:41:44.012577 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:44.011575    5060 patch_prober.go:29] interesting pod/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-bbjl7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body=
Oct 31 12:41:44.012577 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:44.011642    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-bbjl7" podUID="09b9b34e-eb53-4829-aed9-abb0b81675d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused"
Oct 31 12:41:44.019222 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:44.019187    5060 patch_prober.go:29] interesting pod/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-56grg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body=
Oct 31 12:41:44.019461 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:44.019436    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-56grg" podUID="f43cdb3b-9bd6-46a7-a14c-feec07c2c755" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused"
Oct 31 12:41:44.033515 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:44.019871    5060 patch_prober.go:29] interesting pod/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-56grg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body=
Oct 31 12:41:44.033747 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:44.033714    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-56grg" podUID="f43cdb3b-9bd6-46a7-a14c-feec07c2c755" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused"
Oct 31 12:41:44.100874 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:44.100826    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/simulator-w4-c67de2-kserve-868679cf9b-gqjbx" podUID="e104d7da-1fb2-4c99-a480-b4ae84c4f273" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:41:44.429256 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:44.103074    5060 patch_prober.go:29] interesting pod/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-bbjl7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body=
Oct 31 12:41:44.429256 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:44.424602    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-bbjl7" podUID="09b9b34e-eb53-4829-aed9-abb0b81675d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused"
Oct 31 12:41:44.429256 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:44.359677    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/premium-w2-e6c79d-kserve-6468bf46d4-h66f8" podUID="cd6a266f-8d12-4089-83ee-c75c716bf26c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:41:44.429256 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:44.103110    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-2-w5-b25d3b-kserve-76f798d59f-vvzhg" podUID="394ed76c-84ac-4612-9acb-a7e36cb68d89" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": context deadline exceeded"
Oct 31 12:41:44.452547 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:44.151312    5060 patch_prober.go:29] interesting pod/e2e-worker-w1-85ce08-openshift-default-77d6458845-f8gp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body=
Oct 31 12:41:44.452547 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:44.432636    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-f8gp2" podUID="df05433b-b6c1-4bd0-88a1-11e769461d69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused"
Oct 31 12:41:44.452547 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:41:44.363584    5060 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.18s"
Oct 31 12:41:44.452547 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:44.433138    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-2-w5-b25d3b-kserve-76f798d59f-vvzhg"
Oct 31 12:41:44.452547 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:44.390326    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x2qzv" podUID="c247ed2a-aa90-424b-9b7e-4f2936d67e3f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:41:44.452547 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:44.436890    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x2qzv"
Oct 31 12:41:44.508579 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:44.508529    5060 patch_prober.go:29] interesting pod/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-njl9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body=
Oct 31 12:41:44.508859 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:44.508828    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-njl9z" podUID="29b176fa-229b-433e-928f-41daf33d1f37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused"
Oct 31 12:41:44.535515 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:44.531722    5060 patch_prober.go:29] interesting pod/e2e-worker-w1-85ce08-openshift-default-77d6458845-f8gp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body=
Oct 31 12:41:44.535515 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:44.531803    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-f8gp2" podUID="df05433b-b6c1-4bd0-88a1-11e769461d69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused"
Oct 31 12:41:44.872984 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:44.872949    5060 patch_prober.go:29] interesting pod/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-njl9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body=
Oct 31 12:41:44.881772 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:44.881644    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-njl9z" podUID="29b176fa-229b-433e-928f-41daf33d1f37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused"
Oct 31 12:41:44.929784 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:44.929675    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25afd7d2-4vdmm"
Oct 31 12:41:44.937678 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:44.937635    5060 patch_prober.go:29] interesting pod/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-56grg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body=
Oct 31 12:41:44.941620 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:44.941566    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-56grg" podUID="f43cdb3b-9bd6-46a7-a14c-feec07c2c755" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused"
Oct 31 12:41:45.022709 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:45.022653    5060 patch_prober.go:29] interesting pod/e2e-worker-w1-85ce08-openshift-default-77d6458845-f8gp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body=
Oct 31 12:41:45.082537 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:45.023271    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-f8gp2" podUID="df05433b-b6c1-4bd0-88a1-11e769461d69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused"
Oct 31 12:41:45.108540 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:45.107704    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3aa9227a-b604-4644-a5ee-97b5ccc1f33e-kube-api-access-9829v\" (UniqueName: \"kubernetes.io/projected/3aa9227a-b604-4644-a5ee-97b5ccc1f33e-kube-api-access-9829v\") pod \"3aa9227a-b604-4644-a5ee-97b5ccc1f33e\" (UID: \"3aa9227a-b604-4644-a5ee-97b5ccc1f33e\") "
Oct 31 12:41:45.108540 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:45.107794    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3aa9227a-b604-4644-a5ee-97b5ccc1f33e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3aa9227a-b604-4644-a5ee-97b5ccc1f33e-maas-api-xks-ca\") pod \"3aa9227a-b604-4644-a5ee-97b5ccc1f33e\" (UID: \"3aa9227a-b604-4644-a5ee-97b5ccc1f33e\") "
Oct 31 12:41:45.108540 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:45.107856    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3aa9227a-b604-4644-a5ee-97b5ccc1f33e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3aa9227a-b604-4644-a5ee-97b5ccc1f33e-maas-api-openshift-ca\") pod \"3aa9227a-b604-4644-a5ee-97b5ccc1f33e\" (UID: \"3aa9227a-b604-4644-a5ee-97b5ccc1f33e\") "
Oct 31 12:41:45.184642 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:45.149177    5060 patch_prober.go:29] interesting pod/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-bbjl7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body=
Oct 31 12:41:45.184642 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:45.149235    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-bbjl7" podUID="09b9b34e-eb53-4829-aed9-abb0b81675d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused"
Oct 31 12:41:45.184642 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:45.159746    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3aa9227a-b604-4644-a5ee-97b5ccc1f33e-maas-api-openshift-ca" pod "3aa9227a-b604-4644-a5ee-97b5ccc1f33e" (UID: "3aa9227a-b604-4644-a5ee-97b5ccc1f33e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:41:45.281142 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:45.281107    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3aa9227a-b604-4644-a5ee-97b5ccc1f33e-maas-api-openshift-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:41:45.354522 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:45.346552    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3aa9227a-b604-4644-a5ee-97b5ccc1f33e-kube-api-access-9829v" pod "3aa9227a-b604-4644-a5ee-97b5ccc1f33e" (UID: "3aa9227a-b604-4644-a5ee-97b5ccc1f33e"). InnerVolumeSpecName "kube-api-access-9829v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:41:45.373027 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:45.372969    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3aa9227a-b604-4644-a5ee-97b5ccc1f33e-maas-api-xks-ca" pod "3aa9227a-b604-4644-a5ee-97b5ccc1f33e" (UID: "3aa9227a-b604-4644-a5ee-97b5ccc1f33e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:41:45.392102 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:45.391666    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9829v\" (UniqueName: \"kubernetes.io/projected/3aa9227a-b604-4644-a5ee-97b5ccc1f33e-kube-api-access-9829v\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:41:45.392102 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:45.391705    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3aa9227a-b604-4644-a5ee-97b5ccc1f33e-maas-api-xks-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:41:45.426415 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:45.423845    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/simulator-w4-c67de2-kserve-868679cf9b-gqjbx" podUID="e104d7da-1fb2-4c99-a480-b4ae84c4f273" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:41:45.426415 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:45.425034    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/premium-w2-e6c79d-kserve-6468bf46d4-h66f8" podUID="cd6a266f-8d12-4089-83ee-c75c716bf26c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:41:45.426415 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:45.425758    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-2-w5-b25d3b-kserve-76f798d59f-vvzhg" podUID="394ed76c-84ac-4612-9acb-a7e36cb68d89" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:41:45.561524 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:45.555607    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x2qzv" podUID="c247ed2a-aa90-424b-9b7e-4f2936d67e3f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:41:45.820719 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:45.820632    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25afd7d2-4vdmm"
Oct 31 12:41:45.845708 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:45.832734    5060 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 31 12:41:45.845708 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:45.838010    5060 patch_prober.go:29] interesting pod/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-njl9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body=
Oct 31 12:41:45.845708 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:45.838060    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-njl9z" podUID="29b176fa-229b-433e-928f-41daf33d1f37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused"
Oct 31 12:41:45.845708 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:45.838427    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25afd7d2-4vdmm" event={"ID":"3aa9227a-b604-4644-a5ee-97b5ccc1f33e","Type":"ContainerDied","Data":"813dddfd8eaf983de2e6d1e605fa2e2b5078e086bddfbdf31e0687c517ffc3fc"}
Oct 31 12:41:45.845708 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:45.838477    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="813dddfd8eaf983de2e6d1e605fa2e2b5078e086bddfbdf31e0687c517ffc3fc"
Oct 31 12:41:45.943829 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:45.943792    5060 patch_prober.go:29] interesting pod/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-56grg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body=
Oct 31 12:41:45.944095 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:45.944065    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-56grg" podUID="f43cdb3b-9bd6-46a7-a14c-feec07c2c755" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused"
Oct 31 12:41:46.019697 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:46.017408    5060 patch_prober.go:29] interesting pod/e2e-worker-w1-85ce08-openshift-default-77d6458845-f8gp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body=
Oct 31 12:41:46.019697 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:46.017463    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-f8gp2" podUID="df05433b-b6c1-4bd0-88a1-11e769461d69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused"
Oct 31 12:41:46.112230 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:46.112141    5060 patch_prober.go:29] interesting pod/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-bbjl7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body=
Oct 31 12:41:46.134863 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:46.134813    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-bbjl7" podUID="09b9b34e-eb53-4829-aed9-abb0b81675d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused"
Oct 31 12:41:46.429298 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:46.429195    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-2-w5-b25d3b-kserve-76f798d59f-vvzhg" podUID="394ed76c-84ac-4612-9acb-a7e36cb68d89" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:41:46.433545 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:46.433476    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/premium-w2-e6c79d-kserve-6468bf46d4-h66f8" podUID="cd6a266f-8d12-4089-83ee-c75c716bf26c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Oct 31 12:41:46.461417 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:46.456385    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/simulator-w4-c67de2-kserve-868679cf9b-gqjbx" podUID="e104d7da-1fb2-4c99-a480-b4ae84c4f273" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded"
Oct 31 12:41:46.544683 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:46.544633    5060 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-2-w5-b25d3b-kserve-76f798d59f-vvzhg" podUID="394ed76c-84ac-4612-9acb-a7e36cb68d89" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:41:46.576682 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:46.576619    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x2qzv" podUID="c247ed2a-aa90-424b-9b7e-4f2936d67e3f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:41:46.578008 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:46.576939    5060 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-c67de2/simulator-w4-c67de2-kserve-868679cf9b-gqjbx" podUID="e104d7da-1fb2-4c99-a480-b4ae84c4f273" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": context deadline exceeded"
Oct 31 12:41:46.904548 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:46.883714    5060 patch_prober.go:29] interesting pod/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-njl9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body=
Oct 31 12:41:46.930525 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:46.904821    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-njl9z" podUID="29b176fa-229b-433e-928f-41daf33d1f37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused"
Oct 31 12:41:46.948379 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:46.948341    5060 patch_prober.go:29] interesting pod/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-56grg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body=
Oct 31 12:41:46.957615 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:46.957565    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-56grg" podUID="f43cdb3b-9bd6-46a7-a14c-feec07c2c755" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused"
Oct 31 12:41:47.058539 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:47.050070    5060 patch_prober.go:29] interesting pod/e2e-worker-w1-85ce08-openshift-default-77d6458845-f8gp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body=
Oct 31 12:41:47.058539 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:47.050128    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-f8gp2" podUID="df05433b-b6c1-4bd0-88a1-11e769461d69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused"
Oct 31 12:41:47.116392 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:47.116342    5060 patch_prober.go:29] interesting pod/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-bbjl7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body=
Oct 31 12:41:47.117037 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:47.116983    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-bbjl7" podUID="09b9b34e-eb53-4829-aed9-abb0b81675d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused"
Oct 31 12:41:47.440814 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:47.440764    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-2-w5-b25d3b-kserve-76f798d59f-vvzhg" podUID="394ed76c-84ac-4612-9acb-a7e36cb68d89" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": context deadline exceeded"
Oct 31 12:41:47.482541 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:47.474842    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/simulator-w4-c67de2-kserve-868679cf9b-gqjbx" podUID="e104d7da-1fb2-4c99-a480-b4ae84c4f273" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:41:47.623513 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:47.585034    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x2qzv" podUID="c247ed2a-aa90-424b-9b7e-4f2936d67e3f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded"
Oct 31 12:41:47.670671 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:47.670639    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e6c79d/premium-w2-e6c79d-kserve-6468bf46d4-h66f8"
Oct 31 12:41:47.670953 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:47.670940    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-2-w5-b25d3b-kserve-76f798d59f-vvzhg"
Oct 31 12:41:47.688165 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:47.688133    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x2qzv"
Oct 31 12:41:47.853647 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:47.853602    5060 patch_prober.go:29] interesting pod/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-njl9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body=
Oct 31 12:41:47.853923 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:47.853898    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-njl9z" podUID="29b176fa-229b-433e-928f-41daf33d1f37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused"
Oct 31 12:41:47.882514 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:47.857471    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c67de2/simulator-w4-c67de2-kserve-868679cf9b-gqjbx"
Oct 31 12:41:47.948575 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:47.938800    5060 patch_prober.go:29] interesting pod/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-56grg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body=
Oct 31 12:41:47.948575 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:47.938872    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-56grg" podUID="f43cdb3b-9bd6-46a7-a14c-feec07c2c755" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused"
Oct 31 12:41:47.984811 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:47.984690    5060 patch_prober.go:29] interesting pod/e2e-worker-w1-85ce08-openshift-default-77d6458845-f8gp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body=
Oct 31 12:41:47.985281 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:47.985245    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-f8gp2" podUID="df05433b-b6c1-4bd0-88a1-11e769461d69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused"
Oct 31 12:41:48.113480 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:48.112951    5060 patch_prober.go:29] interesting pod/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-bbjl7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body=
Oct 31 12:41:48.113480 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:48.113019    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-bbjl7" podUID="09b9b34e-eb53-4829-aed9-abb0b81675d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused"
Oct 31 12:41:48.845855 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:48.845816    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-njl9z"
Oct 31 12:41:48.847462 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:48.847441    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-njl9z"
Oct 31 12:41:48.850528 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:48.850484    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-njl9z"
Oct 31 12:41:48.985328 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:48.984866    5060 patch_prober.go:29] interesting pod/e2e-worker-w1-85ce08-openshift-default-77d6458845-f8gp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body=
Oct 31 12:41:48.985328 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:48.984932    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-f8gp2" podUID="df05433b-b6c1-4bd0-88a1-11e769461d69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused"
Oct 31 12:41:49.095155 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:49.094859    5060 patch_prober.go:29] interesting pod/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-bbjl7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body=
Oct 31 12:41:49.095155 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:49.094920    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-bbjl7" podUID="09b9b34e-eb53-4829-aed9-abb0b81675d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused"
Oct 31 12:41:49.222678 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:49.222580    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-56grg"
Oct 31 12:41:49.224315 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:49.224292    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-56grg"
Oct 31 12:41:49.225330 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:49.225307    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-56grg"
Oct 31 12:41:49.991136 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:49.991103    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-f8gp2"
Oct 31 12:41:49.992253 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:49.992226    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-f8gp2"
Oct 31 12:41:49.994743 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:49.994719    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-f8gp2"
Oct 31 12:41:50.203740 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:50.203707    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-bbjl7"
Oct 31 12:41:50.204638 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:50.204615    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-bbjl7"
Oct 31 12:41:50.206145 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:50.206001    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-bbjl7"
Oct 31 12:41:50.331404 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:50.331370    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4505ecc2-xbmxd"]
Oct 31 12:41:50.332112 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:50.332095    5060 cpu_manager.go:414] "containerMap: removing container" podUID="3aa9227a-b604-4644-a5ee-97b5ccc1f33e" containerName="revoke-keys"
Oct 31 12:41:50.332112 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:50.332112    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3aa9227a-b604-4644-a5ee-97b5ccc1f33e" containerName="revoke-keys"
Oct 31 12:41:50.332255 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:50.332246    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="3aa9227a-b604-4644-a5ee-97b5ccc1f33e" containerName="revoke-keys"
Oct 31 12:41:50.336421 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:50.336394    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4505ecc2-xbmxd"
Oct 31 12:41:50.340320 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:50.340269    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 12:41:50.341361 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:50.341339    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7wsmb\""
Oct 31 12:41:50.341478 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:50.341397    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 12:41:50.341597 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:50.341577    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 12:41:50.344937 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:50.344915    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4505ecc2-xbmxd"]
Oct 31 12:41:50.411481 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:50.411448    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-26c69f97-npzvq"]
Oct 31 12:41:50.416213 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:50.416193    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26c69f97-npzvq"
Oct 31 12:41:50.428022 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:50.427993    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-26c69f97-npzvq"]
Oct 31 12:41:50.464014 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:50.463979    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a40f13f-5cbe-4cfe-bbf2-d1875c6537c4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4505ecc2-xbmxd\" (UID: \"0a40f13f-5cbe-4cfe-bbf2-d1875c6537c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4505ecc2-xbmxd"
Oct 31 12:41:50.464193 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:50.464029    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a40f13f-5cbe-4cfe-bbf2-d1875c6537c4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4505ecc2-xbmxd\" (UID: \"0a40f13f-5cbe-4cfe-bbf2-d1875c6537c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4505ecc2-xbmxd"
Oct 31 12:41:50.464193 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:50.464089    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfk92\" (UniqueName: \"kubernetes.io/projected/0a40f13f-5cbe-4cfe-bbf2-d1875c6537c4-kube-api-access-kfk92\") pod \"maas-api-revoke-sub-4505ecc2-xbmxd\" (UID: \"0a40f13f-5cbe-4cfe-bbf2-d1875c6537c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4505ecc2-xbmxd"
Oct 31 12:41:50.518187 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:50.507328    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f549b09-4lfmf"]
Oct 31 12:41:50.525556 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:50.525138    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f549b09-4lfmf"
Oct 31 12:41:50.531902 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:50.531870    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f549b09-4lfmf"]
Oct 31 12:41:50.565629 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:50.565598    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/798f63df-6d7f-442a-b8fc-9611c67040f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-26c69f97-npzvq\" (UID: \"798f63df-6d7f-442a-b8fc-9611c67040f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26c69f97-npzvq"
Oct 31 12:41:50.565788 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:50.565644    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnzf4\" (UniqueName: \"kubernetes.io/projected/798f63df-6d7f-442a-b8fc-9611c67040f8-kube-api-access-tnzf4\") pod \"maas-api-revoke-sub-26c69f97-npzvq\" (UID: \"798f63df-6d7f-442a-b8fc-9611c67040f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26c69f97-npzvq"
Oct 31 12:41:50.565842 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:50.565793    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/798f63df-6d7f-442a-b8fc-9611c67040f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-26c69f97-npzvq\" (UID: \"798f63df-6d7f-442a-b8fc-9611c67040f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26c69f97-npzvq"
Oct 31 12:41:50.565899 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:50.565843    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a40f13f-5cbe-4cfe-bbf2-d1875c6537c4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4505ecc2-xbmxd\" (UID: \"0a40f13f-5cbe-4cfe-bbf2-d1875c6537c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4505ecc2-xbmxd"
Oct 31 12:41:50.565899 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:50.565863    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a40f13f-5cbe-4cfe-bbf2-d1875c6537c4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4505ecc2-xbmxd\" (UID: \"0a40f13f-5cbe-4cfe-bbf2-d1875c6537c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4505ecc2-xbmxd"
Oct 31 12:41:50.565961 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:50.565927    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfk92\" (UniqueName: \"kubernetes.io/projected/0a40f13f-5cbe-4cfe-bbf2-d1875c6537c4-kube-api-access-kfk92\") pod \"maas-api-revoke-sub-4505ecc2-xbmxd\" (UID: \"0a40f13f-5cbe-4cfe-bbf2-d1875c6537c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4505ecc2-xbmxd"
Oct 31 12:41:50.567104 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:50.567075    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a40f13f-5cbe-4cfe-bbf2-d1875c6537c4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4505ecc2-xbmxd\" (UID: \"0a40f13f-5cbe-4cfe-bbf2-d1875c6537c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4505ecc2-xbmxd"
Oct 31 12:41:50.568811 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:50.568785    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a40f13f-5cbe-4cfe-bbf2-d1875c6537c4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4505ecc2-xbmxd\" (UID: \"0a40f13f-5cbe-4cfe-bbf2-d1875c6537c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4505ecc2-xbmxd"
Oct 31 12:41:50.578774 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:50.578747    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfk92\" (UniqueName: \"kubernetes.io/projected/0a40f13f-5cbe-4cfe-bbf2-d1875c6537c4-kube-api-access-kfk92\") pod \"maas-api-revoke-sub-4505ecc2-xbmxd\" (UID: \"0a40f13f-5cbe-4cfe-bbf2-d1875c6537c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4505ecc2-xbmxd"
Oct 31 12:41:50.653229 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:50.653055    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4505ecc2-xbmxd"
Oct 31 12:41:50.678474 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:50.677466    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96617f2d-e89b-43b1-8b6c-a580a2d372be-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7f549b09-4lfmf\" (UID: \"96617f2d-e89b-43b1-8b6c-a580a2d372be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f549b09-4lfmf"
Oct 31 12:41:50.678474 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:50.677568    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/798f63df-6d7f-442a-b8fc-9611c67040f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-26c69f97-npzvq\" (UID: \"798f63df-6d7f-442a-b8fc-9611c67040f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26c69f97-npzvq"
Oct 31 12:41:50.678474 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:50.677699    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnzf4\" (UniqueName: \"kubernetes.io/projected/798f63df-6d7f-442a-b8fc-9611c67040f8-kube-api-access-tnzf4\") pod \"maas-api-revoke-sub-26c69f97-npzvq\" (UID: \"798f63df-6d7f-442a-b8fc-9611c67040f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26c69f97-npzvq"
Oct 31 12:41:50.678474 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:50.677792    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96617f2d-e89b-43b1-8b6c-a580a2d372be-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7f549b09-4lfmf\" (UID: \"96617f2d-e89b-43b1-8b6c-a580a2d372be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f549b09-4lfmf"
Oct 31 12:41:50.678474 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:50.677891    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvcvv\" (UniqueName: \"kubernetes.io/projected/96617f2d-e89b-43b1-8b6c-a580a2d372be-kube-api-access-vvcvv\") pod \"maas-api-revoke-sub-7f549b09-4lfmf\" (UID: \"96617f2d-e89b-43b1-8b6c-a580a2d372be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f549b09-4lfmf"
Oct 31 12:41:50.678474 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:50.677958    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/798f63df-6d7f-442a-b8fc-9611c67040f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-26c69f97-npzvq\" (UID: \"798f63df-6d7f-442a-b8fc-9611c67040f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26c69f97-npzvq"
Oct 31 12:41:50.687726 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:50.687665    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/798f63df-6d7f-442a-b8fc-9611c67040f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-26c69f97-npzvq\" (UID: \"798f63df-6d7f-442a-b8fc-9611c67040f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26c69f97-npzvq"
Oct 31 12:41:50.687875 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:50.687851    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/798f63df-6d7f-442a-b8fc-9611c67040f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-26c69f97-npzvq\" (UID: \"798f63df-6d7f-442a-b8fc-9611c67040f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26c69f97-npzvq"
Oct 31 12:41:50.701121 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:50.701085    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnzf4\" (UniqueName: \"kubernetes.io/projected/798f63df-6d7f-442a-b8fc-9611c67040f8-kube-api-access-tnzf4\") pod \"maas-api-revoke-sub-26c69f97-npzvq\" (UID: \"798f63df-6d7f-442a-b8fc-9611c67040f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26c69f97-npzvq"
Oct 31 12:41:50.740232 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:50.740201    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26c69f97-npzvq"
Oct 31 12:41:50.782008 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:50.780037    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96617f2d-e89b-43b1-8b6c-a580a2d372be-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7f549b09-4lfmf\" (UID: \"96617f2d-e89b-43b1-8b6c-a580a2d372be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f549b09-4lfmf"
Oct 31 12:41:50.782008 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:50.780148    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvcvv\" (UniqueName: \"kubernetes.io/projected/96617f2d-e89b-43b1-8b6c-a580a2d372be-kube-api-access-vvcvv\") pod \"maas-api-revoke-sub-7f549b09-4lfmf\" (UID: \"96617f2d-e89b-43b1-8b6c-a580a2d372be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f549b09-4lfmf"
Oct 31 12:41:50.782008 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:50.780234    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96617f2d-e89b-43b1-8b6c-a580a2d372be-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7f549b09-4lfmf\" (UID: \"96617f2d-e89b-43b1-8b6c-a580a2d372be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f549b09-4lfmf"
Oct 31 12:41:50.782008 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:50.781690    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96617f2d-e89b-43b1-8b6c-a580a2d372be-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7f549b09-4lfmf\" (UID: \"96617f2d-e89b-43b1-8b6c-a580a2d372be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f549b09-4lfmf"
Oct 31 12:41:50.787082 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:50.787032    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96617f2d-e89b-43b1-8b6c-a580a2d372be-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7f549b09-4lfmf\" (UID: \"96617f2d-e89b-43b1-8b6c-a580a2d372be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f549b09-4lfmf"
Oct 31 12:41:50.805763 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:50.805160    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvcvv\" (UniqueName: \"kubernetes.io/projected/96617f2d-e89b-43b1-8b6c-a580a2d372be-kube-api-access-vvcvv\") pod \"maas-api-revoke-sub-7f549b09-4lfmf\" (UID: \"96617f2d-e89b-43b1-8b6c-a580a2d372be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f549b09-4lfmf"
Oct 31 12:41:50.851599 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:50.851304    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f549b09-4lfmf"
Oct 31 12:41:50.979452 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:50.979406    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4505ecc2-xbmxd"]
Oct 31 12:41:51.004801 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:41:51.004749    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a40f13f_5cbe_4cfe_bbf2_d1875c6537c4.slice/crio-0a4fb273ec1e020c21ada3fa903fb2cd1db5123370bb1e92fb81f9188b74ce7d WatchSource:0}: Error finding container 0a4fb273ec1e020c21ada3fa903fb2cd1db5123370bb1e92fb81f9188b74ce7d: Status 404 returned error can't find the container with id 0a4fb273ec1e020c21ada3fa903fb2cd1db5123370bb1e92fb81f9188b74ce7d
Oct 31 12:41:51.065171 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:51.065117    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-26c69f97-npzvq"]
Oct 31 12:41:51.158484 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:41:51.158445    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod798f63df_6d7f_442a_b8fc_9611c67040f8.slice/crio-f0531e1f459a604fdbe4f887578d2416aeae1d56a1e6daf1671eb4145bda140a WatchSource:0}: Error finding container f0531e1f459a604fdbe4f887578d2416aeae1d56a1e6daf1671eb4145bda140a: Status 404 returned error can't find the container with id f0531e1f459a604fdbe4f887578d2416aeae1d56a1e6daf1671eb4145bda140a
Oct 31 12:41:51.170876 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:41:51.169891    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96617f2d_e89b_43b1_8b6c_a580a2d372be.slice/crio-7e8d4ae6dabfa26095030af607b3d695359565bb2940678ded974f929cddfb2b WatchSource:0}: Error finding container 7e8d4ae6dabfa26095030af607b3d695359565bb2940678ded974f929cddfb2b: Status 404 returned error can't find the container with id 7e8d4ae6dabfa26095030af607b3d695359565bb2940678ded974f929cddfb2b
Oct 31 12:41:51.172257 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:51.172230    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f549b09-4lfmf"]
Oct 31 12:41:51.190398 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:51.189428    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26c69f97-npzvq" event={"ID":"798f63df-6d7f-442a-b8fc-9611c67040f8","Type":"ContainerStarted","Data":"f0531e1f459a604fdbe4f887578d2416aeae1d56a1e6daf1671eb4145bda140a"}
Oct 31 12:41:51.193112 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:51.192361    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f549b09-4lfmf" event={"ID":"96617f2d-e89b-43b1-8b6c-a580a2d372be","Type":"ContainerStarted","Data":"7e8d4ae6dabfa26095030af607b3d695359565bb2940678ded974f929cddfb2b"}
Oct 31 12:41:51.194464 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:51.194431    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4505ecc2-xbmxd" event={"ID":"0a40f13f-5cbe-4cfe-bbf2-d1875c6537c4","Type":"ContainerStarted","Data":"0a4fb273ec1e020c21ada3fa903fb2cd1db5123370bb1e92fb81f9188b74ce7d"}
Oct 31 12:41:52.353445 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:52.353405    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4505ecc2-xbmxd" event={"ID":"0a40f13f-5cbe-4cfe-bbf2-d1875c6537c4","Type":"ContainerStarted","Data":"63024a86359dffc58137ad85d25b325d6b23be61a1eebf1de36f0f738c9cb4e9"}
Oct 31 12:41:52.391649 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:52.391608    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26c69f97-npzvq" event={"ID":"798f63df-6d7f-442a-b8fc-9611c67040f8","Type":"ContainerStarted","Data":"f168eb5d6e15c60542c75f1826b8ba6c0d5646b25cd64214fd3630a623aa0d0c"}
Oct 31 12:41:52.406555 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:52.402852    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f549b09-4lfmf" event={"ID":"96617f2d-e89b-43b1-8b6c-a580a2d372be","Type":"ContainerStarted","Data":"7a043a877747c7ce8331704c988beb5ea63e41a2b8d6ca0fe9460811ec1174e2"}
Oct 31 12:41:52.409794 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:52.409763    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-28t2l"]
Oct 31 12:41:52.441855 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:52.441818    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-28t2l"
Oct 31 12:41:52.599529 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:52.594448    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66295870-6cfc-4f48-b353-566fe73a6bf0-workload-socket\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-28t2l\" (UID: \"66295870-6cfc-4f48-b353-566fe73a6bf0\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-28t2l"
Oct 31 12:41:52.599529 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:52.594609    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgjp7\" (UniqueName: \"kubernetes.io/projected/66295870-6cfc-4f48-b353-566fe73a6bf0-kube-api-access-fgjp7\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-28t2l\" (UID: \"66295870-6cfc-4f48-b353-566fe73a6bf0\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-28t2l"
Oct 31 12:41:52.599529 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:52.594661    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66295870-6cfc-4f48-b353-566fe73a6bf0-workload-certs\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-28t2l\" (UID: \"66295870-6cfc-4f48-b353-566fe73a6bf0\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-28t2l"
Oct 31 12:41:52.599529 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:52.594754    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66295870-6cfc-4f48-b353-566fe73a6bf0-istio-envoy\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-28t2l\" (UID: \"66295870-6cfc-4f48-b353-566fe73a6bf0\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-28t2l"
Oct 31 12:41:52.599529 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:52.594864    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66295870-6cfc-4f48-b353-566fe73a6bf0-istio-podinfo\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-28t2l\" (UID: \"66295870-6cfc-4f48-b353-566fe73a6bf0\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-28t2l"
Oct 31 12:41:52.599529 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:52.594926    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66295870-6cfc-4f48-b353-566fe73a6bf0-credential-socket\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-28t2l\" (UID: \"66295870-6cfc-4f48-b353-566fe73a6bf0\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-28t2l"
Oct 31 12:41:52.599529 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:52.594990    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66295870-6cfc-4f48-b353-566fe73a6bf0-istio-token\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-28t2l\" (UID: \"66295870-6cfc-4f48-b353-566fe73a6bf0\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-28t2l"
Oct 31 12:41:52.599529 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:52.595136    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66295870-6cfc-4f48-b353-566fe73a6bf0-istio-data\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-28t2l\" (UID: \"66295870-6cfc-4f48-b353-566fe73a6bf0\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-28t2l"
Oct 31 12:41:52.599529 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:52.595199    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66295870-6cfc-4f48-b353-566fe73a6bf0-istiod-ca-cert\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-28t2l\" (UID: \"66295870-6cfc-4f48-b353-566fe73a6bf0\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-28t2l"
Oct 31 12:41:52.721562 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:52.705766    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66295870-6cfc-4f48-b353-566fe73a6bf0-istio-envoy\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-28t2l\" (UID: \"66295870-6cfc-4f48-b353-566fe73a6bf0\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-28t2l"
Oct 31 12:41:52.721562 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:52.705883    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66295870-6cfc-4f48-b353-566fe73a6bf0-istio-podinfo\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-28t2l\" (UID: \"66295870-6cfc-4f48-b353-566fe73a6bf0\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-28t2l"
Oct 31 12:41:52.721562 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:52.705943    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66295870-6cfc-4f48-b353-566fe73a6bf0-credential-socket\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-28t2l\" (UID: \"66295870-6cfc-4f48-b353-566fe73a6bf0\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-28t2l"
Oct 31 12:41:52.721562 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:52.705989    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66295870-6cfc-4f48-b353-566fe73a6bf0-istio-token\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-28t2l\" (UID: \"66295870-6cfc-4f48-b353-566fe73a6bf0\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-28t2l"
Oct 31 12:41:52.721562 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:52.706067    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66295870-6cfc-4f48-b353-566fe73a6bf0-istio-data\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-28t2l\" (UID: \"66295870-6cfc-4f48-b353-566fe73a6bf0\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-28t2l"
Oct 31 12:41:52.721562 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:52.706111    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66295870-6cfc-4f48-b353-566fe73a6bf0-istiod-ca-cert\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-28t2l\" (UID: \"66295870-6cfc-4f48-b353-566fe73a6bf0\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-28t2l"
Oct 31 12:41:52.721562 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:52.706179    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66295870-6cfc-4f48-b353-566fe73a6bf0-workload-socket\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-28t2l\" (UID: \"66295870-6cfc-4f48-b353-566fe73a6bf0\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-28t2l"
Oct 31 12:41:52.721562 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:52.706259    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgjp7\" (UniqueName: \"kubernetes.io/projected/66295870-6cfc-4f48-b353-566fe73a6bf0-kube-api-access-fgjp7\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-28t2l\" (UID: \"66295870-6cfc-4f48-b353-566fe73a6bf0\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-28t2l"
Oct 31 12:41:52.721562 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:52.706301    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66295870-6cfc-4f48-b353-566fe73a6bf0-workload-certs\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-28t2l\" (UID: \"66295870-6cfc-4f48-b353-566fe73a6bf0\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-28t2l"
Oct 31 12:41:52.721562 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:52.706916    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66295870-6cfc-4f48-b353-566fe73a6bf0-workload-certs\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-28t2l\" (UID: \"66295870-6cfc-4f48-b353-566fe73a6bf0\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-28t2l"
Oct 31 12:41:52.775516 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:52.755470    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66295870-6cfc-4f48-b353-566fe73a6bf0-istio-data\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-28t2l\" (UID: \"66295870-6cfc-4f48-b353-566fe73a6bf0\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-28t2l"
Oct 31 12:41:52.775516 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:52.768737    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66295870-6cfc-4f48-b353-566fe73a6bf0-workload-socket\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-28t2l\" (UID: \"66295870-6cfc-4f48-b353-566fe73a6bf0\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-28t2l"
Oct 31 12:41:52.775516 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:52.769389    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66295870-6cfc-4f48-b353-566fe73a6bf0-istiod-ca-cert\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-28t2l\" (UID: \"66295870-6cfc-4f48-b353-566fe73a6bf0\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-28t2l"
Oct 31 12:41:52.775516 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:52.769967    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66295870-6cfc-4f48-b353-566fe73a6bf0-credential-socket\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-28t2l\" (UID: \"66295870-6cfc-4f48-b353-566fe73a6bf0\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-28t2l"
Oct 31 12:41:52.932517 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:52.925794    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66295870-6cfc-4f48-b353-566fe73a6bf0-istio-token\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-28t2l\" (UID: \"66295870-6cfc-4f48-b353-566fe73a6bf0\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-28t2l"
Oct 31 12:41:52.932517 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:52.925870    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-28t2l"]
Oct 31 12:41:53.002582 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:53.002448    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4505ecc2-xbmxd" podStartSLOduration=3.002428332 podStartE2EDuration="3.002428332s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:41:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:41:52.401841645 +0000 UTC m=+875.504909359" watchObservedRunningTime="2026-10-31 12:41:53.002428332 +0000 UTC m=+876.105496046"
Oct 31 12:41:53.185538 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:53.178161    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66295870-6cfc-4f48-b353-566fe73a6bf0-istio-podinfo\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-28t2l\" (UID: \"66295870-6cfc-4f48-b353-566fe73a6bf0\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-28t2l"
Oct 31 12:41:53.278964 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:53.257204    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66295870-6cfc-4f48-b353-566fe73a6bf0-istio-envoy\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-28t2l\" (UID: \"66295870-6cfc-4f48-b353-566fe73a6bf0\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-28t2l"
Oct 31 12:41:53.279275 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:53.263436    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgjp7\" (UniqueName: \"kubernetes.io/projected/66295870-6cfc-4f48-b353-566fe73a6bf0-kube-api-access-fgjp7\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-28t2l\" (UID: \"66295870-6cfc-4f48-b353-566fe73a6bf0\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-28t2l"
Oct 31 12:41:53.355524 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:53.355476    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-28t2l"
Oct 31 12:41:53.440473 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:53.440394    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-2-w5-b25d3b-kserve-76f798d59f-vvzhg" podUID="394ed76c-84ac-4612-9acb-a7e36cb68d89" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:41:53.442222 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:53.441405    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26c69f97-npzvq" podStartSLOduration=3.441386221 podStartE2EDuration="3.441386221s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:41:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:41:52.424979184 +0000 UTC m=+875.528046897" watchObservedRunningTime="2026-10-31 12:41:53.441386221 +0000 UTC m=+876.544453946"
Oct 31 12:41:53.482571 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:53.442438    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/premium-w2-e6c79d-kserve-6468bf46d4-h66f8" podUID="cd6a266f-8d12-4089-83ee-c75c716bf26c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Oct 31 12:41:53.482571 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:53.442714    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/simulator-w4-c67de2-kserve-868679cf9b-gqjbx" podUID="e104d7da-1fb2-4c99-a480-b4ae84c4f273" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded"
Oct 31 12:41:53.611876 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:53.611771    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x2qzv" podUID="c247ed2a-aa90-424b-9b7e-4f2936d67e3f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:41:53.754524 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:53.749043    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f549b09-4lfmf" podStartSLOduration=3.749022591 podStartE2EDuration="3.749022591s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:41:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:41:52.463598293 +0000 UTC m=+875.566666006" watchObservedRunningTime="2026-10-31 12:41:53.749022591 +0000 UTC m=+876.852090304"
Oct 31 12:41:53.944454 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:53.943814    5060 generic.go:356] "Generic (PLEG): container finished" podID="798f63df-6d7f-442a-b8fc-9611c67040f8" containerID="f168eb5d6e15c60542c75f1826b8ba6c0d5646b25cd64214fd3630a623aa0d0c" exitCode=0
Oct 31 12:41:53.944454 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:53.943960    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26c69f97-npzvq" event={"ID":"798f63df-6d7f-442a-b8fc-9611c67040f8","Type":"ContainerDied","Data":"f168eb5d6e15c60542c75f1826b8ba6c0d5646b25cd64214fd3630a623aa0d0c"}
Oct 31 12:41:53.959980 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:53.959553    5060 generic.go:356] "Generic (PLEG): container finished" podID="96617f2d-e89b-43b1-8b6c-a580a2d372be" containerID="7a043a877747c7ce8331704c988beb5ea63e41a2b8d6ca0fe9460811ec1174e2" exitCode=0
Oct 31 12:41:53.959980 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:53.959778    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f549b09-4lfmf" event={"ID":"96617f2d-e89b-43b1-8b6c-a580a2d372be","Type":"ContainerDied","Data":"7a043a877747c7ce8331704c988beb5ea63e41a2b8d6ca0fe9460811ec1174e2"}
Oct 31 12:41:54.035833 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:54.035776    5060 generic.go:356] "Generic (PLEG): container finished" podID="0a40f13f-5cbe-4cfe-bbf2-d1875c6537c4" containerID="63024a86359dffc58137ad85d25b325d6b23be61a1eebf1de36f0f738c9cb4e9" exitCode=0
Oct 31 12:41:54.036200 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:54.036170    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4505ecc2-xbmxd" event={"ID":"0a40f13f-5cbe-4cfe-bbf2-d1875c6537c4","Type":"ContainerDied","Data":"63024a86359dffc58137ad85d25b325d6b23be61a1eebf1de36f0f738c9cb4e9"}
Oct 31 12:41:54.473570 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:54.463628    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/premium-w2-e6c79d-kserve-6468bf46d4-h66f8" podUID="cd6a266f-8d12-4089-83ee-c75c716bf26c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:41:54.473570 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:54.464436    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e6c79d/premium-w2-e6c79d-kserve-6468bf46d4-h66f8"
Oct 31 12:41:54.473570 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:54.466120    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/simulator-w4-c67de2-kserve-868679cf9b-gqjbx" podUID="e104d7da-1fb2-4c99-a480-b4ae84c4f273" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded"
Oct 31 12:41:54.498515 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:54.487772    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-2-w5-b25d3b-kserve-76f798d59f-vvzhg" podUID="394ed76c-84ac-4612-9acb-a7e36cb68d89" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": context deadline exceeded"
Oct 31 12:41:54.498515 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:54.488610    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c67de2/simulator-w4-c67de2-kserve-868679cf9b-gqjbx"
Oct 31 12:41:54.498515 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:54.488673    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-2-w5-b25d3b-kserve-76f798d59f-vvzhg"
Oct 31 12:41:54.630534 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:54.617502    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x2qzv" podUID="c247ed2a-aa90-424b-9b7e-4f2936d67e3f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:41:54.630534 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:54.618145    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x2qzv"
Oct 31 12:41:55.499513 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:55.483748    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/premium-w2-e6c79d-kserve-6468bf46d4-h66f8" podUID="cd6a266f-8d12-4089-83ee-c75c716bf26c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:41:55.499513 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:55.484767    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/simulator-w4-c67de2-kserve-868679cf9b-gqjbx" podUID="e104d7da-1fb2-4c99-a480-b4ae84c4f273" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded"
Oct 31 12:41:55.499513 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:55.497320    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-2-w5-b25d3b-kserve-76f798d59f-vvzhg" podUID="394ed76c-84ac-4612-9acb-a7e36cb68d89" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": context deadline exceeded"
Oct 31 12:41:55.635516 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:55.619186    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x2qzv" podUID="c247ed2a-aa90-424b-9b7e-4f2936d67e3f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:41:56.146621 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:56.136645    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:41:56.146951 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:56.146911    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:41:56.152931 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:56.152891    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:41:56.546515 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:56.485592    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/simulator-w4-c67de2-kserve-868679cf9b-gqjbx" podUID="e104d7da-1fb2-4c99-a480-b4ae84c4f273" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded"
Oct 31 12:41:56.546515 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:56.486019    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/premium-w2-e6c79d-kserve-6468bf46d4-h66f8" podUID="cd6a266f-8d12-4089-83ee-c75c716bf26c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Oct 31 12:41:56.546515 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:56.501780    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-2-w5-b25d3b-kserve-76f798d59f-vvzhg" podUID="394ed76c-84ac-4612-9acb-a7e36cb68d89" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:41:56.546515 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:56.508623    5060 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-2-w5-b25d3b-kserve-76f798d59f-vvzhg" podUID="394ed76c-84ac-4612-9acb-a7e36cb68d89" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:41:56.599795 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:56.599744    5060 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-c67de2/simulator-w4-c67de2-kserve-868679cf9b-gqjbx" podUID="e104d7da-1fb2-4c99-a480-b4ae84c4f273" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:41:56.626322 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:56.626277    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x2qzv" podUID="c247ed2a-aa90-424b-9b7e-4f2936d67e3f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:41:56.859227 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:41:56.859186    5060 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.731s"
Oct 31 12:41:56.936118 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:56.936078    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-28t2l" event={"ID":"66295870-6cfc-4f48-b353-566fe73a6bf0","Type":"ContainerStarted","Data":"11d442b5712724b7964af9edfab30d6c919ee982ae95393440c8c3ac0df988e0"}
Oct 31 12:41:57.007900 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:56.995638    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-28t2l"]
Oct 31 12:41:57.008191 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:57.008153    5060 prober.go:122] "Probe failed" probeType="Liveness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-kgjh7" podUID="4f8464cd-b924-44f9-821b-f4c683531f45" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.132.0.43:6080/livez\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:41:57.394523 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:57.394338    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-kgjh7" podUID="4f8464cd-b924-44f9-821b-f4c683531f45" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.132.0.43:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:41:57.559517 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:57.554431    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/simulator-w4-c67de2-kserve-868679cf9b-gqjbx" podUID="e104d7da-1fb2-4c99-a480-b4ae84c4f273" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded"
Oct 31 12:41:57.595522 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:57.589357    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/premium-w2-e6c79d-kserve-6468bf46d4-h66f8" podUID="cd6a266f-8d12-4089-83ee-c75c716bf26c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Oct 31 12:41:57.628549 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:57.561130    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-2-w5-b25d3b-kserve-76f798d59f-vvzhg" podUID="394ed76c-84ac-4612-9acb-a7e36cb68d89" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:41:57.634888 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:57.634816    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x2qzv" podUID="c247ed2a-aa90-424b-9b7e-4f2936d67e3f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:41:57.842568 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:57.839085    5060 patch_prober.go:29] interesting pod/e2e-worker-w1-85ce08-openshift-default-77d6458845-58cq4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 12:41:57.930673 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:57.903081    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-58cq4" podUID="c86d1b8b-6bc6-45ff-9412-179f069cf080" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:41:58.424696 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:58.424588    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vmzbr" podUID="7049f597-e139-4f0e-afbd-ec6ea0effab9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:41:58.614719 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:58.592668    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/simulator-w4-c67de2-kserve-868679cf9b-gqjbx" podUID="e104d7da-1fb2-4c99-a480-b4ae84c4f273" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:41:58.614719 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:58.604020    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/premium-w2-e6c79d-kserve-6468bf46d4-h66f8" podUID="cd6a266f-8d12-4089-83ee-c75c716bf26c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:41:58.650619 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:58.632205    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-2-w5-b25d3b-kserve-76f798d59f-vvzhg" podUID="394ed76c-84ac-4612-9acb-a7e36cb68d89" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:41:58.670561 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:58.659410    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x2qzv" podUID="c247ed2a-aa90-424b-9b7e-4f2936d67e3f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:41:59.036658 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:41:59.036617    5060 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.991s"
Oct 31 12:41:59.077078 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:59.036920    5060 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-e6c79d/premium-w2-e6c79d-kserve-6468bf46d4-h66f8" podUID="cd6a266f-8d12-4089-83ee-c75c716bf26c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:41:59.637376 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:59.637322    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-2-w5-b25d3b-kserve-76f798d59f-vvzhg" podUID="394ed76c-84ac-4612-9acb-a7e36cb68d89" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:41:59.743683 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:59.645613    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/simulator-w4-c67de2-kserve-868679cf9b-gqjbx" podUID="e104d7da-1fb2-4c99-a480-b4ae84c4f273" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:41:59.744221 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:41:59.645657    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/premium-w2-e6c79d-kserve-6468bf46d4-h66f8" podUID="cd6a266f-8d12-4089-83ee-c75c716bf26c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:42:00.268414 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:42:00.268377    5060 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.211s"
Oct 31 12:42:00.268706 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:00.268688    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x2qzv"
Oct 31 12:42:00.270200 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:00.270179    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c67de2/simulator-w4-c67de2-kserve-868679cf9b-gqjbx"
Oct 31 12:42:00.315409 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:00.315380    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e6c79d/premium-w2-e6c79d-kserve-6468bf46d4-h66f8"
Oct 31 12:42:00.325721 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:00.325677    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-2-w5-b25d3b-kserve-76f798d59f-vvzhg"
Oct 31 12:42:00.504229 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:00.504197    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4505ecc2-xbmxd"
Oct 31 12:42:00.513725 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:42:00.513691    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7phb" containerName="extractor"
Oct 31 12:42:00.523081 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:00.519698    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f549b09-4lfmf"
Oct 31 12:42:00.523081 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:00.522376    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26c69f97-npzvq"
Oct 31 12:42:00.525157 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:00.525126    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f549b09-4lfmf" event={"ID":"96617f2d-e89b-43b1-8b6c-a580a2d372be","Type":"ContainerDied","Data":"7e8d4ae6dabfa26095030af607b3d695359565bb2940678ded974f929cddfb2b"}
Oct 31 12:42:00.525298 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:00.525196    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7e8d4ae6dabfa26095030af607b3d695359565bb2940678ded974f929cddfb2b"
Oct 31 12:42:00.528246 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:00.528216    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4505ecc2-xbmxd" event={"ID":"0a40f13f-5cbe-4cfe-bbf2-d1875c6537c4","Type":"ContainerDied","Data":"0a4fb273ec1e020c21ada3fa903fb2cd1db5123370bb1e92fb81f9188b74ce7d"}
Oct 31 12:42:00.528348 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:00.528269    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0a4fb273ec1e020c21ada3fa903fb2cd1db5123370bb1e92fb81f9188b74ce7d"
Oct 31 12:42:00.528348 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:00.528324    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4505ecc2-xbmxd"
Oct 31 12:42:00.532977 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:00.532915    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-28t2l" event={"ID":"66295870-6cfc-4f48-b353-566fe73a6bf0","Type":"ContainerStarted","Data":"4b22833c781d46e9c40b427b1113815ec8395e35c2c4ba287f4237abcc240d0b"}
Oct 31 12:42:00.540777 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:00.540664    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26c69f97-npzvq" event={"ID":"798f63df-6d7f-442a-b8fc-9611c67040f8","Type":"ContainerDied","Data":"f0531e1f459a604fdbe4f887578d2416aeae1d56a1e6daf1671eb4145bda140a"}
Oct 31 12:42:00.540777 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:00.540730    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f0531e1f459a604fdbe4f887578d2416aeae1d56a1e6daf1671eb4145bda140a"
Oct 31 12:42:00.540988 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:00.540904    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26c69f97-npzvq"
Oct 31 12:42:00.657562 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:00.656608    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a40f13f-5cbe-4cfe-bbf2-d1875c6537c4-kube-api-access-kfk92\" (UniqueName: \"kubernetes.io/projected/0a40f13f-5cbe-4cfe-bbf2-d1875c6537c4-kube-api-access-kfk92\") pod \"0a40f13f-5cbe-4cfe-bbf2-d1875c6537c4\" (UID: \"0a40f13f-5cbe-4cfe-bbf2-d1875c6537c4\") "
Oct 31 12:42:00.657562 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:00.656665    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/96617f2d-e89b-43b1-8b6c-a580a2d372be-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96617f2d-e89b-43b1-8b6c-a580a2d372be-maas-api-xks-ca\") pod \"96617f2d-e89b-43b1-8b6c-a580a2d372be\" (UID: \"96617f2d-e89b-43b1-8b6c-a580a2d372be\") "
Oct 31 12:42:00.657562 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:00.656839    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/798f63df-6d7f-442a-b8fc-9611c67040f8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/798f63df-6d7f-442a-b8fc-9611c67040f8-maas-api-openshift-ca\") pod \"798f63df-6d7f-442a-b8fc-9611c67040f8\" (UID: \"798f63df-6d7f-442a-b8fc-9611c67040f8\") "
Oct 31 12:42:00.657562 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:00.656906    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a40f13f-5cbe-4cfe-bbf2-d1875c6537c4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a40f13f-5cbe-4cfe-bbf2-d1875c6537c4-maas-api-openshift-ca\") pod \"0a40f13f-5cbe-4cfe-bbf2-d1875c6537c4\" (UID: \"0a40f13f-5cbe-4cfe-bbf2-d1875c6537c4\") "
Oct 31 12:42:00.657562 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:00.656985    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96617f2d-e89b-43b1-8b6c-a580a2d372be-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96617f2d-e89b-43b1-8b6c-a580a2d372be-maas-api-openshift-ca\") pod \"96617f2d-e89b-43b1-8b6c-a580a2d372be\" (UID: \"96617f2d-e89b-43b1-8b6c-a580a2d372be\") "
Oct 31 12:42:00.657562 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:00.657019    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a40f13f-5cbe-4cfe-bbf2-d1875c6537c4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a40f13f-5cbe-4cfe-bbf2-d1875c6537c4-maas-api-xks-ca\") pod \"0a40f13f-5cbe-4cfe-bbf2-d1875c6537c4\" (UID: \"0a40f13f-5cbe-4cfe-bbf2-d1875c6537c4\") "
Oct 31 12:42:00.657562 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:00.657123    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/798f63df-6d7f-442a-b8fc-9611c67040f8-kube-api-access-tnzf4\" (UniqueName: \"kubernetes.io/projected/798f63df-6d7f-442a-b8fc-9611c67040f8-kube-api-access-tnzf4\") pod \"798f63df-6d7f-442a-b8fc-9611c67040f8\" (UID: \"798f63df-6d7f-442a-b8fc-9611c67040f8\") "
Oct 31 12:42:00.657562 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:00.657198    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/798f63df-6d7f-442a-b8fc-9611c67040f8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/798f63df-6d7f-442a-b8fc-9611c67040f8-maas-api-xks-ca\") pod \"798f63df-6d7f-442a-b8fc-9611c67040f8\" (UID: \"798f63df-6d7f-442a-b8fc-9611c67040f8\") "
Oct 31 12:42:00.657562 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:00.657261    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96617f2d-e89b-43b1-8b6c-a580a2d372be-kube-api-access-vvcvv\" (UniqueName: \"kubernetes.io/projected/96617f2d-e89b-43b1-8b6c-a580a2d372be-kube-api-access-vvcvv\") pod \"96617f2d-e89b-43b1-8b6c-a580a2d372be\" (UID: \"96617f2d-e89b-43b1-8b6c-a580a2d372be\") "
Oct 31 12:42:00.665261 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:00.663082    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a40f13f-5cbe-4cfe-bbf2-d1875c6537c4-maas-api-openshift-ca" pod "0a40f13f-5cbe-4cfe-bbf2-d1875c6537c4" (UID: "0a40f13f-5cbe-4cfe-bbf2-d1875c6537c4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:42:00.665261 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:00.664178    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/798f63df-6d7f-442a-b8fc-9611c67040f8-maas-api-openshift-ca" pod "798f63df-6d7f-442a-b8fc-9611c67040f8" (UID: "798f63df-6d7f-442a-b8fc-9611c67040f8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:42:00.673173 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:00.672307    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/798f63df-6d7f-442a-b8fc-9611c67040f8-kube-api-access-tnzf4" pod "798f63df-6d7f-442a-b8fc-9611c67040f8" (UID: "798f63df-6d7f-442a-b8fc-9611c67040f8"). InnerVolumeSpecName "kube-api-access-tnzf4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:42:00.673173 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:00.673110    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96617f2d-e89b-43b1-8b6c-a580a2d372be-maas-api-openshift-ca" pod "96617f2d-e89b-43b1-8b6c-a580a2d372be" (UID: "96617f2d-e89b-43b1-8b6c-a580a2d372be"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:42:00.673560 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:00.673521    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a40f13f-5cbe-4cfe-bbf2-d1875c6537c4-maas-api-xks-ca" pod "0a40f13f-5cbe-4cfe-bbf2-d1875c6537c4" (UID: "0a40f13f-5cbe-4cfe-bbf2-d1875c6537c4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:42:00.673966 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:00.673935    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96617f2d-e89b-43b1-8b6c-a580a2d372be-maas-api-xks-ca" pod "96617f2d-e89b-43b1-8b6c-a580a2d372be" (UID: "96617f2d-e89b-43b1-8b6c-a580a2d372be"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:42:00.675820 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:00.675797    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a40f13f-5cbe-4cfe-bbf2-d1875c6537c4-kube-api-access-kfk92" pod "0a40f13f-5cbe-4cfe-bbf2-d1875c6537c4" (UID: "0a40f13f-5cbe-4cfe-bbf2-d1875c6537c4"). InnerVolumeSpecName "kube-api-access-kfk92". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:42:00.676652 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:00.676477    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/798f63df-6d7f-442a-b8fc-9611c67040f8-maas-api-xks-ca" pod "798f63df-6d7f-442a-b8fc-9611c67040f8" (UID: "798f63df-6d7f-442a-b8fc-9611c67040f8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:42:00.676844 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:00.676830    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96617f2d-e89b-43b1-8b6c-a580a2d372be-kube-api-access-vvcvv" pod "96617f2d-e89b-43b1-8b6c-a580a2d372be" (UID: "96617f2d-e89b-43b1-8b6c-a580a2d372be"). InnerVolumeSpecName "kube-api-access-vvcvv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:42:00.690806 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:00.690742    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-28t2l" podStartSLOduration=8.690721139 podStartE2EDuration="8.690721139s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:41:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:42:00.632331375 +0000 UTC m=+883.735399089" watchObservedRunningTime="2026-10-31 12:42:00.690721139 +0000 UTC m=+883.793788855"
Oct 31 12:42:00.761093 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:00.760660    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvcvv\" (UniqueName: \"kubernetes.io/projected/96617f2d-e89b-43b1-8b6c-a580a2d372be-kube-api-access-vvcvv\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:42:00.761093 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:00.760710    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kfk92\" (UniqueName: \"kubernetes.io/projected/0a40f13f-5cbe-4cfe-bbf2-d1875c6537c4-kube-api-access-kfk92\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:42:00.761093 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:00.760727    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96617f2d-e89b-43b1-8b6c-a580a2d372be-maas-api-xks-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:42:00.761093 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:00.760743    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/798f63df-6d7f-442a-b8fc-9611c67040f8-maas-api-openshift-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:42:00.761093 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:00.760758    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a40f13f-5cbe-4cfe-bbf2-d1875c6537c4-maas-api-openshift-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:42:00.761093 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:00.760773    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96617f2d-e89b-43b1-8b6c-a580a2d372be-maas-api-openshift-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:42:00.761093 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:00.760791    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a40f13f-5cbe-4cfe-bbf2-d1875c6537c4-maas-api-xks-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:42:00.761093 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:00.760805    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tnzf4\" (UniqueName: \"kubernetes.io/projected/798f63df-6d7f-442a-b8fc-9611c67040f8-kube-api-access-tnzf4\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:42:00.761093 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:00.760818    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/798f63df-6d7f-442a-b8fc-9611c67040f8-maas-api-xks-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:42:01.359171 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:01.359032    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-28t2l"
Oct 31 12:42:01.374022 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:01.373659    5060 patch_prober.go:29] interesting pod/e2e-worker-w3-b218d3-openshift-default-67596dd89f-28t2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body=
Oct 31 12:42:01.374022 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:01.373728    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-28t2l" podUID="66295870-6cfc-4f48-b353-566fe73a6bf0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused"
Oct 31 12:42:01.549925 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:01.549893    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f549b09-4lfmf"
Oct 31 12:42:02.359515 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:02.358864    5060 patch_prober.go:29] interesting pod/e2e-worker-w3-b218d3-openshift-default-67596dd89f-28t2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body=
Oct 31 12:42:02.359515 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:02.358937    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-28t2l" podUID="66295870-6cfc-4f48-b353-566fe73a6bf0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused"
Oct 31 12:42:03.363524 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:03.361640    5060 patch_prober.go:29] interesting pod/e2e-worker-w3-b218d3-openshift-default-67596dd89f-28t2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body=
Oct 31 12:42:03.363524 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:03.361710    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-28t2l" podUID="66295870-6cfc-4f48-b353-566fe73a6bf0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused"
Oct 31 12:42:03.631249 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:03.631158    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e6b5b0c1-v9xxw"]
Oct 31 12:42:03.631856 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:03.631841    5060 cpu_manager.go:414] "containerMap: removing container" podUID="798f63df-6d7f-442a-b8fc-9611c67040f8" containerName="revoke-keys"
Oct 31 12:42:03.631918 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:03.631856    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="798f63df-6d7f-442a-b8fc-9611c67040f8" containerName="revoke-keys"
Oct 31 12:42:03.631918 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:03.631873    5060 cpu_manager.go:414] "containerMap: removing container" podUID="96617f2d-e89b-43b1-8b6c-a580a2d372be" containerName="revoke-keys"
Oct 31 12:42:03.631918 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:03.631879    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96617f2d-e89b-43b1-8b6c-a580a2d372be" containerName="revoke-keys"
Oct 31 12:42:03.631918 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:03.631919    5060 cpu_manager.go:414] "containerMap: removing container" podUID="0a40f13f-5cbe-4cfe-bbf2-d1875c6537c4" containerName="revoke-keys"
Oct 31 12:42:03.632129 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:03.631924    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a40f13f-5cbe-4cfe-bbf2-d1875c6537c4" containerName="revoke-keys"
Oct 31 12:42:03.632129 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:03.632017    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="798f63df-6d7f-442a-b8fc-9611c67040f8" containerName="revoke-keys"
Oct 31 12:42:03.632129 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:03.632024    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="96617f2d-e89b-43b1-8b6c-a580a2d372be" containerName="revoke-keys"
Oct 31 12:42:03.632129 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:03.632038    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a40f13f-5cbe-4cfe-bbf2-d1875c6537c4" containerName="revoke-keys"
Oct 31 12:42:03.636143 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:03.636121    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6b5b0c1-v9xxw"
Oct 31 12:42:03.640157 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:03.640132    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 12:42:03.640772 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:03.640742    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 12:42:03.640772 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:03.640771    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7wsmb\""
Oct 31 12:42:03.640956 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:03.640744    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 12:42:03.644344 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:03.644319    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e6b5b0c1-v9xxw"]
Oct 31 12:42:03.746975 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:03.734603    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79lf5\" (UniqueName: \"kubernetes.io/projected/3747a92f-de3a-439a-8e3e-a4b7d2f3b048-kube-api-access-79lf5\") pod \"maas-api-revoke-sub-e6b5b0c1-v9xxw\" (UID: \"3747a92f-de3a-439a-8e3e-a4b7d2f3b048\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6b5b0c1-v9xxw"
Oct 31 12:42:03.747677 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:03.747637    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3747a92f-de3a-439a-8e3e-a4b7d2f3b048-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e6b5b0c1-v9xxw\" (UID: \"3747a92f-de3a-439a-8e3e-a4b7d2f3b048\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6b5b0c1-v9xxw"
Oct 31 12:42:03.747840 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:03.747827    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3747a92f-de3a-439a-8e3e-a4b7d2f3b048-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e6b5b0c1-v9xxw\" (UID: \"3747a92f-de3a-439a-8e3e-a4b7d2f3b048\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6b5b0c1-v9xxw"
Oct 31 12:42:03.851520 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:03.851469    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79lf5\" (UniqueName: \"kubernetes.io/projected/3747a92f-de3a-439a-8e3e-a4b7d2f3b048-kube-api-access-79lf5\") pod \"maas-api-revoke-sub-e6b5b0c1-v9xxw\" (UID: \"3747a92f-de3a-439a-8e3e-a4b7d2f3b048\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6b5b0c1-v9xxw"
Oct 31 12:42:03.851710 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:03.851585    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3747a92f-de3a-439a-8e3e-a4b7d2f3b048-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e6b5b0c1-v9xxw\" (UID: \"3747a92f-de3a-439a-8e3e-a4b7d2f3b048\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6b5b0c1-v9xxw"
Oct 31 12:42:03.851710 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:03.851610    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3747a92f-de3a-439a-8e3e-a4b7d2f3b048-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e6b5b0c1-v9xxw\" (UID: \"3747a92f-de3a-439a-8e3e-a4b7d2f3b048\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6b5b0c1-v9xxw"
Oct 31 12:42:03.852285 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:03.852258    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3747a92f-de3a-439a-8e3e-a4b7d2f3b048-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e6b5b0c1-v9xxw\" (UID: \"3747a92f-de3a-439a-8e3e-a4b7d2f3b048\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6b5b0c1-v9xxw"
Oct 31 12:42:03.854461 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:03.854435    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3747a92f-de3a-439a-8e3e-a4b7d2f3b048-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e6b5b0c1-v9xxw\" (UID: \"3747a92f-de3a-439a-8e3e-a4b7d2f3b048\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6b5b0c1-v9xxw"
Oct 31 12:42:03.893561 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:03.893286    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79lf5\" (UniqueName: \"kubernetes.io/projected/3747a92f-de3a-439a-8e3e-a4b7d2f3b048-kube-api-access-79lf5\") pod \"maas-api-revoke-sub-e6b5b0c1-v9xxw\" (UID: \"3747a92f-de3a-439a-8e3e-a4b7d2f3b048\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6b5b0c1-v9xxw"
Oct 31 12:42:03.966537 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:03.963877    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6b5b0c1-v9xxw"
Oct 31 12:42:04.204196 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:04.204113    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c5e4799-mjqt5"]
Oct 31 12:42:04.211558 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:04.211533    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c5e4799-mjqt5"
Oct 31 12:42:04.213115 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:04.213095    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa778ea2-dsjqx"]
Oct 31 12:42:04.218733 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:04.218709    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c5e4799-mjqt5"]
Oct 31 12:42:04.218863 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:04.218804    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa778ea2-dsjqx"
Oct 31 12:42:04.230394 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:04.230354    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa778ea2-dsjqx"]
Oct 31 12:42:04.274573 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:04.274522    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bx6b\" (UniqueName: \"kubernetes.io/projected/5cb9104d-0b45-4ba5-80ca-bff8d339fd36-kube-api-access-7bx6b\") pod \"maas-api-revoke-sub-fa778ea2-dsjqx\" (UID: \"5cb9104d-0b45-4ba5-80ca-bff8d339fd36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa778ea2-dsjqx"
Oct 31 12:42:04.287833 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:04.287780    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ac6a465-1595-4d84-b230-72b56e67a33e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8c5e4799-mjqt5\" (UID: \"0ac6a465-1595-4d84-b230-72b56e67a33e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c5e4799-mjqt5"
Oct 31 12:42:04.288211 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:04.288176    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ac6a465-1595-4d84-b230-72b56e67a33e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8c5e4799-mjqt5\" (UID: \"0ac6a465-1595-4d84-b230-72b56e67a33e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c5e4799-mjqt5"
Oct 31 12:42:04.288393 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:04.288365    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqzmt\" (UniqueName: \"kubernetes.io/projected/0ac6a465-1595-4d84-b230-72b56e67a33e-kube-api-access-hqzmt\") pod \"maas-api-revoke-sub-8c5e4799-mjqt5\" (UID: \"0ac6a465-1595-4d84-b230-72b56e67a33e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c5e4799-mjqt5"
Oct 31 12:42:04.288559 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:04.288532    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cb9104d-0b45-4ba5-80ca-bff8d339fd36-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fa778ea2-dsjqx\" (UID: \"5cb9104d-0b45-4ba5-80ca-bff8d339fd36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa778ea2-dsjqx"
Oct 31 12:42:04.288770 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:04.288754    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cb9104d-0b45-4ba5-80ca-bff8d339fd36-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fa778ea2-dsjqx\" (UID: \"5cb9104d-0b45-4ba5-80ca-bff8d339fd36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa778ea2-dsjqx"
Oct 31 12:42:04.311998 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:04.311730    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e6b5b0c1-v9xxw"]
Oct 31 12:42:04.385690 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:04.385650    5060 patch_prober.go:29] interesting pod/e2e-worker-w3-b218d3-openshift-default-67596dd89f-28t2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body=
Oct 31 12:42:04.386170 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:04.386131    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-28t2l" podUID="66295870-6cfc-4f48-b353-566fe73a6bf0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused"
Oct 31 12:42:04.393017 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:04.392905    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ac6a465-1595-4d84-b230-72b56e67a33e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8c5e4799-mjqt5\" (UID: \"0ac6a465-1595-4d84-b230-72b56e67a33e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c5e4799-mjqt5"
Oct 31 12:42:04.393163 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:04.393048    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqzmt\" (UniqueName: \"kubernetes.io/projected/0ac6a465-1595-4d84-b230-72b56e67a33e-kube-api-access-hqzmt\") pod \"maas-api-revoke-sub-8c5e4799-mjqt5\" (UID: \"0ac6a465-1595-4d84-b230-72b56e67a33e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c5e4799-mjqt5"
Oct 31 12:42:04.393163 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:04.393076    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cb9104d-0b45-4ba5-80ca-bff8d339fd36-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fa778ea2-dsjqx\" (UID: \"5cb9104d-0b45-4ba5-80ca-bff8d339fd36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa778ea2-dsjqx"
Oct 31 12:42:04.393163 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:04.393131    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cb9104d-0b45-4ba5-80ca-bff8d339fd36-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fa778ea2-dsjqx\" (UID: \"5cb9104d-0b45-4ba5-80ca-bff8d339fd36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa778ea2-dsjqx"
Oct 31 12:42:04.393319 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:04.393270    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bx6b\" (UniqueName: \"kubernetes.io/projected/5cb9104d-0b45-4ba5-80ca-bff8d339fd36-kube-api-access-7bx6b\") pod \"maas-api-revoke-sub-fa778ea2-dsjqx\" (UID: \"5cb9104d-0b45-4ba5-80ca-bff8d339fd36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa778ea2-dsjqx"
Oct 31 12:42:04.393319 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:04.393308    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ac6a465-1595-4d84-b230-72b56e67a33e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8c5e4799-mjqt5\" (UID: \"0ac6a465-1595-4d84-b230-72b56e67a33e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c5e4799-mjqt5"
Oct 31 12:42:04.394236 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:04.394208    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ac6a465-1595-4d84-b230-72b56e67a33e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8c5e4799-mjqt5\" (UID: \"0ac6a465-1595-4d84-b230-72b56e67a33e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c5e4799-mjqt5"
Oct 31 12:42:04.394397 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:04.394375    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cb9104d-0b45-4ba5-80ca-bff8d339fd36-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fa778ea2-dsjqx\" (UID: \"5cb9104d-0b45-4ba5-80ca-bff8d339fd36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa778ea2-dsjqx"
Oct 31 12:42:04.395786 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:04.395759    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ac6a465-1595-4d84-b230-72b56e67a33e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8c5e4799-mjqt5\" (UID: \"0ac6a465-1595-4d84-b230-72b56e67a33e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c5e4799-mjqt5"
Oct 31 12:42:04.396083 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:04.396027    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cb9104d-0b45-4ba5-80ca-bff8d339fd36-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fa778ea2-dsjqx\" (UID: \"5cb9104d-0b45-4ba5-80ca-bff8d339fd36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa778ea2-dsjqx"
Oct 31 12:42:04.409532 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:04.409506    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bx6b\" (UniqueName: \"kubernetes.io/projected/5cb9104d-0b45-4ba5-80ca-bff8d339fd36-kube-api-access-7bx6b\") pod \"maas-api-revoke-sub-fa778ea2-dsjqx\" (UID: \"5cb9104d-0b45-4ba5-80ca-bff8d339fd36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa778ea2-dsjqx"
Oct 31 12:42:04.409650 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:04.409486    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqzmt\" (UniqueName: \"kubernetes.io/projected/0ac6a465-1595-4d84-b230-72b56e67a33e-kube-api-access-hqzmt\") pod \"maas-api-revoke-sub-8c5e4799-mjqt5\" (UID: \"0ac6a465-1595-4d84-b230-72b56e67a33e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c5e4799-mjqt5"
Oct 31 12:42:04.544088 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:04.544057    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c5e4799-mjqt5"
Oct 31 12:42:04.561173 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:04.560086    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa778ea2-dsjqx"
Oct 31 12:42:04.612107 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:04.612075    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6b5b0c1-v9xxw" event={"ID":"3747a92f-de3a-439a-8e3e-a4b7d2f3b048","Type":"ContainerStarted","Data":"cda5c1483f5b647f4a9055751fe067eecfb21865697188f1d2743c79208bd4b3"}
Oct 31 12:42:05.014815 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:42:05.014758    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ac6a465_1595_4d84_b230_72b56e67a33e.slice/crio-03522fa2dbb70299b1e227d8ef42af21b9b8dc7eefec426cf34327d75df4ece1 WatchSource:0}: Error finding container 03522fa2dbb70299b1e227d8ef42af21b9b8dc7eefec426cf34327d75df4ece1: Status 404 returned error can't find the container with id 03522fa2dbb70299b1e227d8ef42af21b9b8dc7eefec426cf34327d75df4ece1
Oct 31 12:42:05.017670 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:05.017647    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c5e4799-mjqt5"]
Oct 31 12:42:05.030807 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:42:05.030778    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cb9104d_0b45_4ba5_80ca_bff8d339fd36.slice/crio-f173e2be5511553321539ff41c4e18b48a6e31b4db22d74142564927251ab61f WatchSource:0}: Error finding container f173e2be5511553321539ff41c4e18b48a6e31b4db22d74142564927251ab61f: Status 404 returned error can't find the container with id f173e2be5511553321539ff41c4e18b48a6e31b4db22d74142564927251ab61f
Oct 31 12:42:05.031958 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:05.031930    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa778ea2-dsjqx"]
Oct 31 12:42:05.364760 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:05.359086    5060 patch_prober.go:29] interesting pod/e2e-worker-w3-b218d3-openshift-default-67596dd89f-28t2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body=
Oct 31 12:42:05.364760 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:05.359146    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-28t2l" podUID="66295870-6cfc-4f48-b353-566fe73a6bf0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused"
Oct 31 12:42:05.651116 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:05.648439    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c5e4799-mjqt5" event={"ID":"0ac6a465-1595-4d84-b230-72b56e67a33e","Type":"ContainerStarted","Data":"03522fa2dbb70299b1e227d8ef42af21b9b8dc7eefec426cf34327d75df4ece1"}
Oct 31 12:42:05.651116 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:05.650621    5060 generic.go:356] "Generic (PLEG): container finished" podID="3747a92f-de3a-439a-8e3e-a4b7d2f3b048" containerID="84545d817f673029f76bdfd04e79ff41953b6497153d89a6be592d06ce23846c" exitCode=0
Oct 31 12:42:05.651116 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:05.650840    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6b5b0c1-v9xxw" event={"ID":"3747a92f-de3a-439a-8e3e-a4b7d2f3b048","Type":"ContainerDied","Data":"84545d817f673029f76bdfd04e79ff41953b6497153d89a6be592d06ce23846c"}
Oct 31 12:42:05.653411 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:05.653373    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa778ea2-dsjqx" event={"ID":"5cb9104d-0b45-4ba5-80ca-bff8d339fd36","Type":"ContainerStarted","Data":"60a0b48f406bbffd3817a09f05f357296de87247b995a466a99345eb83d4ef05"}
Oct 31 12:42:05.653551 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:05.653431    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa778ea2-dsjqx" event={"ID":"5cb9104d-0b45-4ba5-80ca-bff8d339fd36","Type":"ContainerStarted","Data":"f173e2be5511553321539ff41c4e18b48a6e31b4db22d74142564927251ab61f"}
Oct 31 12:42:05.704208 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:05.704142    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa778ea2-dsjqx" podStartSLOduration=1.704122527 podStartE2EDuration="1.704122527s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:42:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:42:05.698064411 +0000 UTC m=+888.801132125" watchObservedRunningTime="2026-10-31 12:42:05.704122527 +0000 UTC m=+888.807190243"
Oct 31 12:42:06.357347 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:06.357307    5060 patch_prober.go:29] interesting pod/e2e-worker-w3-b218d3-openshift-default-67596dd89f-28t2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body=
Oct 31 12:42:06.357539 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:06.357372    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-28t2l" podUID="66295870-6cfc-4f48-b353-566fe73a6bf0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused"
Oct 31 12:42:06.664458 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:06.664422    5060 generic.go:356] "Generic (PLEG): container finished" podID="0ac6a465-1595-4d84-b230-72b56e67a33e" containerID="363780b2bd626836b7147ef4cc43c5ea9d278e3a1a8061ddab1f83c3d3405877" exitCode=0
Oct 31 12:42:06.664948 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:06.664537    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c5e4799-mjqt5" event={"ID":"0ac6a465-1595-4d84-b230-72b56e67a33e","Type":"ContainerDied","Data":"363780b2bd626836b7147ef4cc43c5ea9d278e3a1a8061ddab1f83c3d3405877"}
Oct 31 12:42:06.666635 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:06.666606    5060 generic.go:356] "Generic (PLEG): container finished" podID="5cb9104d-0b45-4ba5-80ca-bff8d339fd36" containerID="60a0b48f406bbffd3817a09f05f357296de87247b995a466a99345eb83d4ef05" exitCode=0
Oct 31 12:42:06.666753 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:06.666686    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa778ea2-dsjqx" event={"ID":"5cb9104d-0b45-4ba5-80ca-bff8d339fd36","Type":"ContainerDied","Data":"60a0b48f406bbffd3817a09f05f357296de87247b995a466a99345eb83d4ef05"}
Oct 31 12:42:06.873622 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:06.873593    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6b5b0c1-v9xxw"
Oct 31 12:42:06.964106 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:06.964019    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3747a92f-de3a-439a-8e3e-a4b7d2f3b048-kube-api-access-79lf5\" (UniqueName: \"kubernetes.io/projected/3747a92f-de3a-439a-8e3e-a4b7d2f3b048-kube-api-access-79lf5\") pod \"3747a92f-de3a-439a-8e3e-a4b7d2f3b048\" (UID: \"3747a92f-de3a-439a-8e3e-a4b7d2f3b048\") "
Oct 31 12:42:06.964257 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:06.964212    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3747a92f-de3a-439a-8e3e-a4b7d2f3b048-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3747a92f-de3a-439a-8e3e-a4b7d2f3b048-maas-api-xks-ca\") pod \"3747a92f-de3a-439a-8e3e-a4b7d2f3b048\" (UID: \"3747a92f-de3a-439a-8e3e-a4b7d2f3b048\") "
Oct 31 12:42:06.964307 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:06.964273    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3747a92f-de3a-439a-8e3e-a4b7d2f3b048-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3747a92f-de3a-439a-8e3e-a4b7d2f3b048-maas-api-openshift-ca\") pod \"3747a92f-de3a-439a-8e3e-a4b7d2f3b048\" (UID: \"3747a92f-de3a-439a-8e3e-a4b7d2f3b048\") "
Oct 31 12:42:06.964732 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:06.964699    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3747a92f-de3a-439a-8e3e-a4b7d2f3b048-maas-api-openshift-ca" pod "3747a92f-de3a-439a-8e3e-a4b7d2f3b048" (UID: "3747a92f-de3a-439a-8e3e-a4b7d2f3b048"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:42:06.965050 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:06.965031    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3747a92f-de3a-439a-8e3e-a4b7d2f3b048-maas-api-openshift-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:42:06.966328 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:06.966300    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3747a92f-de3a-439a-8e3e-a4b7d2f3b048-kube-api-access-79lf5" pod "3747a92f-de3a-439a-8e3e-a4b7d2f3b048" (UID: "3747a92f-de3a-439a-8e3e-a4b7d2f3b048"). InnerVolumeSpecName "kube-api-access-79lf5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:42:06.966776 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:06.966752    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3747a92f-de3a-439a-8e3e-a4b7d2f3b048-maas-api-xks-ca" pod "3747a92f-de3a-439a-8e3e-a4b7d2f3b048" (UID: "3747a92f-de3a-439a-8e3e-a4b7d2f3b048"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:42:07.066821 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:07.066792    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3747a92f-de3a-439a-8e3e-a4b7d2f3b048-maas-api-xks-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:42:07.066821 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:07.066819    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79lf5\" (UniqueName: \"kubernetes.io/projected/3747a92f-de3a-439a-8e3e-a4b7d2f3b048-kube-api-access-79lf5\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:42:07.683180 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:07.683143    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6b5b0c1-v9xxw"
Oct 31 12:42:07.683180 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:07.683162    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6b5b0c1-v9xxw" event={"ID":"3747a92f-de3a-439a-8e3e-a4b7d2f3b048","Type":"ContainerDied","Data":"cda5c1483f5b647f4a9055751fe067eecfb21865697188f1d2743c79208bd4b3"}
Oct 31 12:42:07.683673 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:07.683216    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cda5c1483f5b647f4a9055751fe067eecfb21865697188f1d2743c79208bd4b3"
Oct 31 12:42:07.877089 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:07.877058    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-28t2l"
Oct 31 12:42:07.879267 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:07.878854    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-28t2l"
Oct 31 12:42:07.879941 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:07.879919    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-28t2l"
Oct 31 12:42:07.910679 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:07.910653    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c5e4799-mjqt5"
Oct 31 12:42:07.961190 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:07.961159    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-zswkp"]
Oct 31 12:42:07.962539 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:07.962361    5060 cpu_manager.go:414] "containerMap: removing container" podUID="0ac6a465-1595-4d84-b230-72b56e67a33e" containerName="revoke-keys"
Oct 31 12:42:07.962539 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:07.962385    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ac6a465-1595-4d84-b230-72b56e67a33e" containerName="revoke-keys"
Oct 31 12:42:07.962539 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:07.962430    5060 cpu_manager.go:414] "containerMap: removing container" podUID="3747a92f-de3a-439a-8e3e-a4b7d2f3b048" containerName="revoke-keys"
Oct 31 12:42:07.962539 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:07.962439    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3747a92f-de3a-439a-8e3e-a4b7d2f3b048" containerName="revoke-keys"
Oct 31 12:42:07.962839 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:07.962807    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ac6a465-1595-4d84-b230-72b56e67a33e" containerName="revoke-keys"
Oct 31 12:42:07.962839 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:07.962830    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="3747a92f-de3a-439a-8e3e-a4b7d2f3b048" containerName="revoke-keys"
Oct 31 12:42:07.969780 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:07.969759    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-zswkp"
Oct 31 12:42:07.985867 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:07.982977    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ac6a465-1595-4d84-b230-72b56e67a33e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ac6a465-1595-4d84-b230-72b56e67a33e-maas-api-openshift-ca\") pod \"0ac6a465-1595-4d84-b230-72b56e67a33e\" (UID: \"0ac6a465-1595-4d84-b230-72b56e67a33e\") "
Oct 31 12:42:07.985867 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:07.983021    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0ac6a465-1595-4d84-b230-72b56e67a33e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ac6a465-1595-4d84-b230-72b56e67a33e-maas-api-xks-ca\") pod \"0ac6a465-1595-4d84-b230-72b56e67a33e\" (UID: \"0ac6a465-1595-4d84-b230-72b56e67a33e\") "
Oct 31 12:42:07.985867 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:07.983287    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ac6a465-1595-4d84-b230-72b56e67a33e-kube-api-access-hqzmt\" (UniqueName: \"kubernetes.io/projected/0ac6a465-1595-4d84-b230-72b56e67a33e-kube-api-access-hqzmt\") pod \"0ac6a465-1595-4d84-b230-72b56e67a33e\" (UID: \"0ac6a465-1595-4d84-b230-72b56e67a33e\") "
Oct 31 12:42:07.985867 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:07.984462    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ac6a465-1595-4d84-b230-72b56e67a33e-maas-api-openshift-ca" pod "0ac6a465-1595-4d84-b230-72b56e67a33e" (UID: "0ac6a465-1595-4d84-b230-72b56e67a33e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:42:07.993814 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:07.993788    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-zswkp"]
Oct 31 12:42:07.994044 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:07.994005    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ac6a465-1595-4d84-b230-72b56e67a33e-kube-api-access-hqzmt" pod "0ac6a465-1595-4d84-b230-72b56e67a33e" (UID: "0ac6a465-1595-4d84-b230-72b56e67a33e"). InnerVolumeSpecName "kube-api-access-hqzmt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:42:07.994849 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:07.994777    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa778ea2-dsjqx"
Oct 31 12:42:08.002954 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:08.000654    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ac6a465-1595-4d84-b230-72b56e67a33e-maas-api-xks-ca" pod "0ac6a465-1595-4d84-b230-72b56e67a33e" (UID: "0ac6a465-1595-4d84-b230-72b56e67a33e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:42:08.105089 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:08.105048    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5cb9104d-0b45-4ba5-80ca-bff8d339fd36-kube-api-access-7bx6b\" (UniqueName: \"kubernetes.io/projected/5cb9104d-0b45-4ba5-80ca-bff8d339fd36-kube-api-access-7bx6b\") pod \"5cb9104d-0b45-4ba5-80ca-bff8d339fd36\" (UID: \"5cb9104d-0b45-4ba5-80ca-bff8d339fd36\") "
Oct 31 12:42:08.124899 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:08.124858    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5cb9104d-0b45-4ba5-80ca-bff8d339fd36-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cb9104d-0b45-4ba5-80ca-bff8d339fd36-maas-api-openshift-ca\") pod \"5cb9104d-0b45-4ba5-80ca-bff8d339fd36\" (UID: \"5cb9104d-0b45-4ba5-80ca-bff8d339fd36\") "
Oct 31 12:42:08.125079 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:08.125059    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5cb9104d-0b45-4ba5-80ca-bff8d339fd36-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cb9104d-0b45-4ba5-80ca-bff8d339fd36-maas-api-xks-ca\") pod \"5cb9104d-0b45-4ba5-80ca-bff8d339fd36\" (UID: \"5cb9104d-0b45-4ba5-80ca-bff8d339fd36\") "
Oct 31 12:42:08.125467 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:08.125444    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90387d5f-2bb8-4922-a1ad-9466e0745aaf-credential-socket\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-zswkp\" (UID: \"90387d5f-2bb8-4922-a1ad-9466e0745aaf\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-zswkp"
Oct 31 12:42:08.125761 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:08.125739    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90387d5f-2bb8-4922-a1ad-9466e0745aaf-istio-token\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-zswkp\" (UID: \"90387d5f-2bb8-4922-a1ad-9466e0745aaf\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-zswkp"
Oct 31 12:42:08.125832 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:08.125779    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6m7m\" (UniqueName: \"kubernetes.io/projected/90387d5f-2bb8-4922-a1ad-9466e0745aaf-kube-api-access-v6m7m\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-zswkp\" (UID: \"90387d5f-2bb8-4922-a1ad-9466e0745aaf\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-zswkp"
Oct 31 12:42:08.125932 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:08.125909    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90387d5f-2bb8-4922-a1ad-9466e0745aaf-istio-envoy\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-zswkp\" (UID: \"90387d5f-2bb8-4922-a1ad-9466e0745aaf\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-zswkp"
Oct 31 12:42:08.126169 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:08.126019    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5cb9104d-0b45-4ba5-80ca-bff8d339fd36-maas-api-openshift-ca" pod "5cb9104d-0b45-4ba5-80ca-bff8d339fd36" (UID: "5cb9104d-0b45-4ba5-80ca-bff8d339fd36"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:42:08.126274 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:08.126260    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90387d5f-2bb8-4922-a1ad-9466e0745aaf-istiod-ca-cert\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-zswkp\" (UID: \"90387d5f-2bb8-4922-a1ad-9466e0745aaf\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-zswkp"
Oct 31 12:42:08.126396 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:08.126381    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90387d5f-2bb8-4922-a1ad-9466e0745aaf-workload-certs\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-zswkp\" (UID: \"90387d5f-2bb8-4922-a1ad-9466e0745aaf\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-zswkp"
Oct 31 12:42:08.126700 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:08.126631    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90387d5f-2bb8-4922-a1ad-9466e0745aaf-istio-data\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-zswkp\" (UID: \"90387d5f-2bb8-4922-a1ad-9466e0745aaf\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-zswkp"
Oct 31 12:42:08.126700 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:08.126681    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90387d5f-2bb8-4922-a1ad-9466e0745aaf-workload-socket\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-zswkp\" (UID: \"90387d5f-2bb8-4922-a1ad-9466e0745aaf\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-zswkp"
Oct 31 12:42:08.126846 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:08.126763    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90387d5f-2bb8-4922-a1ad-9466e0745aaf-istio-podinfo\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-zswkp\" (UID: \"90387d5f-2bb8-4922-a1ad-9466e0745aaf\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-zswkp"
Oct 31 12:42:08.126971 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:08.126915    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ac6a465-1595-4d84-b230-72b56e67a33e-maas-api-openshift-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:42:08.127081 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:08.127067    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ac6a465-1595-4d84-b230-72b56e67a33e-maas-api-xks-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:42:08.127164 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:08.127153    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cb9104d-0b45-4ba5-80ca-bff8d339fd36-maas-api-openshift-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:42:08.127218 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:08.127211    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hqzmt\" (UniqueName: \"kubernetes.io/projected/0ac6a465-1595-4d84-b230-72b56e67a33e-kube-api-access-hqzmt\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:42:08.128922 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:08.128904    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5cb9104d-0b45-4ba5-80ca-bff8d339fd36-maas-api-xks-ca" pod "5cb9104d-0b45-4ba5-80ca-bff8d339fd36" (UID: "5cb9104d-0b45-4ba5-80ca-bff8d339fd36"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:42:08.133991 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:08.133965    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cb9104d-0b45-4ba5-80ca-bff8d339fd36-kube-api-access-7bx6b" pod "5cb9104d-0b45-4ba5-80ca-bff8d339fd36" (UID: "5cb9104d-0b45-4ba5-80ca-bff8d339fd36"). InnerVolumeSpecName "kube-api-access-7bx6b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:42:08.228445 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:08.228352    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90387d5f-2bb8-4922-a1ad-9466e0745aaf-istio-data\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-zswkp\" (UID: \"90387d5f-2bb8-4922-a1ad-9466e0745aaf\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-zswkp"
Oct 31 12:42:08.228445 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:08.228401    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90387d5f-2bb8-4922-a1ad-9466e0745aaf-workload-socket\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-zswkp\" (UID: \"90387d5f-2bb8-4922-a1ad-9466e0745aaf\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-zswkp"
Oct 31 12:42:08.228445 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:08.228433    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90387d5f-2bb8-4922-a1ad-9466e0745aaf-istio-podinfo\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-zswkp\" (UID: \"90387d5f-2bb8-4922-a1ad-9466e0745aaf\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-zswkp"
Oct 31 12:42:08.228670 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:08.228570    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90387d5f-2bb8-4922-a1ad-9466e0745aaf-credential-socket\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-zswkp\" (UID: \"90387d5f-2bb8-4922-a1ad-9466e0745aaf\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-zswkp"
Oct 31 12:42:08.228670 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:08.228646    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90387d5f-2bb8-4922-a1ad-9466e0745aaf-istio-token\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-zswkp\" (UID: \"90387d5f-2bb8-4922-a1ad-9466e0745aaf\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-zswkp"
Oct 31 12:42:08.228735 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:08.228671    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6m7m\" (UniqueName: \"kubernetes.io/projected/90387d5f-2bb8-4922-a1ad-9466e0745aaf-kube-api-access-v6m7m\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-zswkp\" (UID: \"90387d5f-2bb8-4922-a1ad-9466e0745aaf\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-zswkp"
Oct 31 12:42:08.228772 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:08.228745    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90387d5f-2bb8-4922-a1ad-9466e0745aaf-istio-envoy\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-zswkp\" (UID: \"90387d5f-2bb8-4922-a1ad-9466e0745aaf\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-zswkp"
Oct 31 12:42:08.228874 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:08.228856    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90387d5f-2bb8-4922-a1ad-9466e0745aaf-istio-data\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-zswkp\" (UID: \"90387d5f-2bb8-4922-a1ad-9466e0745aaf\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-zswkp"
Oct 31 12:42:08.228909 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:08.228865    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90387d5f-2bb8-4922-a1ad-9466e0745aaf-istiod-ca-cert\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-zswkp\" (UID: \"90387d5f-2bb8-4922-a1ad-9466e0745aaf\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-zswkp"
Oct 31 12:42:08.228942 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:08.228915    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90387d5f-2bb8-4922-a1ad-9466e0745aaf-workload-certs\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-zswkp\" (UID: \"90387d5f-2bb8-4922-a1ad-9466e0745aaf\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-zswkp"
Oct 31 12:42:08.229048 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:08.229030    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7bx6b\" (UniqueName: \"kubernetes.io/projected/5cb9104d-0b45-4ba5-80ca-bff8d339fd36-kube-api-access-7bx6b\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:42:08.229091 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:08.229058    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cb9104d-0b45-4ba5-80ca-bff8d339fd36-maas-api-xks-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:42:08.229313 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:08.229292    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90387d5f-2bb8-4922-a1ad-9466e0745aaf-workload-certs\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-zswkp\" (UID: \"90387d5f-2bb8-4922-a1ad-9466e0745aaf\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-zswkp"
Oct 31 12:42:08.229525 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:08.229503    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90387d5f-2bb8-4922-a1ad-9466e0745aaf-istiod-ca-cert\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-zswkp\" (UID: \"90387d5f-2bb8-4922-a1ad-9466e0745aaf\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-zswkp"
Oct 31 12:42:08.229569 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:08.229559    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90387d5f-2bb8-4922-a1ad-9466e0745aaf-credential-socket\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-zswkp\" (UID: \"90387d5f-2bb8-4922-a1ad-9466e0745aaf\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-zswkp"
Oct 31 12:42:08.229794 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:08.229774    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90387d5f-2bb8-4922-a1ad-9466e0745aaf-workload-socket\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-zswkp\" (UID: \"90387d5f-2bb8-4922-a1ad-9466e0745aaf\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-zswkp"
Oct 31 12:42:08.231991 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:08.231964    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90387d5f-2bb8-4922-a1ad-9466e0745aaf-istio-envoy\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-zswkp\" (UID: \"90387d5f-2bb8-4922-a1ad-9466e0745aaf\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-zswkp"
Oct 31 12:42:08.232307 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:08.232287    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90387d5f-2bb8-4922-a1ad-9466e0745aaf-istio-podinfo\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-zswkp\" (UID: \"90387d5f-2bb8-4922-a1ad-9466e0745aaf\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-zswkp"
Oct 31 12:42:08.242481 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:08.242458    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90387d5f-2bb8-4922-a1ad-9466e0745aaf-istio-token\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-zswkp\" (UID: \"90387d5f-2bb8-4922-a1ad-9466e0745aaf\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-zswkp"
Oct 31 12:42:08.242644 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:08.242625    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6m7m\" (UniqueName: \"kubernetes.io/projected/90387d5f-2bb8-4922-a1ad-9466e0745aaf-kube-api-access-v6m7m\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-zswkp\" (UID: \"90387d5f-2bb8-4922-a1ad-9466e0745aaf\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-zswkp"
Oct 31 12:42:08.342722 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:08.342689    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-zswkp"
Oct 31 12:42:08.645003 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:42:08.644955    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90387d5f_2bb8_4922_a1ad_9466e0745aaf.slice/crio-d9df11aeb8265c983c2221e0969335a6b73f2f895ee4add240070d9c77dca3cf WatchSource:0}: Error finding container d9df11aeb8265c983c2221e0969335a6b73f2f895ee4add240070d9c77dca3cf: Status 404 returned error can't find the container with id d9df11aeb8265c983c2221e0969335a6b73f2f895ee4add240070d9c77dca3cf
Oct 31 12:42:08.646542 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:08.646513    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-zswkp"]
Oct 31 12:42:08.648162 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:08.648120    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:42:08.648276 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:08.648189    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:42:08.648276 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:08.648221    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:42:08.694263 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:08.694229    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-zswkp" event={"ID":"90387d5f-2bb8-4922-a1ad-9466e0745aaf","Type":"ContainerStarted","Data":"d9df11aeb8265c983c2221e0969335a6b73f2f895ee4add240070d9c77dca3cf"}
Oct 31 12:42:08.695824 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:08.695800    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c5e4799-mjqt5"
Oct 31 12:42:08.695941 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:08.695802    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c5e4799-mjqt5" event={"ID":"0ac6a465-1595-4d84-b230-72b56e67a33e","Type":"ContainerDied","Data":"03522fa2dbb70299b1e227d8ef42af21b9b8dc7eefec426cf34327d75df4ece1"}
Oct 31 12:42:08.695941 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:08.695935    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="03522fa2dbb70299b1e227d8ef42af21b9b8dc7eefec426cf34327d75df4ece1"
Oct 31 12:42:08.697759 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:08.697733    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa778ea2-dsjqx"
Oct 31 12:42:08.697869 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:08.697751    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa778ea2-dsjqx" event={"ID":"5cb9104d-0b45-4ba5-80ca-bff8d339fd36","Type":"ContainerDied","Data":"f173e2be5511553321539ff41c4e18b48a6e31b4db22d74142564927251ab61f"}
Oct 31 12:42:08.697869 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:08.697805    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f173e2be5511553321539ff41c4e18b48a6e31b4db22d74142564927251ab61f"
Oct 31 12:42:09.710947 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:09.710911    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-zswkp" event={"ID":"90387d5f-2bb8-4922-a1ad-9466e0745aaf","Type":"ContainerStarted","Data":"0b8ba48fe21bde7067b3ae3fffa99971a3a916655997de34860726a10b9a24de"}
Oct 31 12:42:09.743195 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:09.743126    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-zswkp" podStartSLOduration=2.743107097 podStartE2EDuration="2.743107097s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:42:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:42:09.741533106 +0000 UTC m=+892.844600818" watchObservedRunningTime="2026-10-31 12:42:09.743107097 +0000 UTC m=+892.846174812"
Oct 31 12:42:10.342845 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:10.342812    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-zswkp"
Oct 31 12:42:10.344409 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:10.344378    5060 patch_prober.go:29] interesting pod/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-zswkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body=
Oct 31 12:42:10.344520 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:10.344456    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-zswkp" podUID="90387d5f-2bb8-4922-a1ad-9466e0745aaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused"
Oct 31 12:42:11.343822 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:11.343787    5060 patch_prober.go:29] interesting pod/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-zswkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body=
Oct 31 12:42:11.344199 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:11.343846    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-zswkp" podUID="90387d5f-2bb8-4922-a1ad-9466e0745aaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused"
Oct 31 12:42:12.343408 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:12.343374    5060 patch_prober.go:29] interesting pod/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-zswkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body=
Oct 31 12:42:12.343595 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:12.343428    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-zswkp" podUID="90387d5f-2bb8-4922-a1ad-9466e0745aaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused"
Oct 31 12:42:12.607048 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:12.606966    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6adf0260-kcxpx"]
Oct 31 12:42:12.607719 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:12.607702    5060 cpu_manager.go:414] "containerMap: removing container" podUID="5cb9104d-0b45-4ba5-80ca-bff8d339fd36" containerName="revoke-keys"
Oct 31 12:42:12.607719 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:12.607719    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5cb9104d-0b45-4ba5-80ca-bff8d339fd36" containerName="revoke-keys"
Oct 31 12:42:12.607850 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:12.607841    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="5cb9104d-0b45-4ba5-80ca-bff8d339fd36" containerName="revoke-keys"
Oct 31 12:42:12.612931 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:12.612905    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6adf0260-kcxpx"
Oct 31 12:42:12.618274 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:12.618246    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 12:42:12.618476 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:12.618456    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7wsmb\""
Oct 31 12:42:12.618603 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:12.618586    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 12:42:12.618687 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:12.618630    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 12:42:12.621035 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:12.621008    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6adf0260-kcxpx"]
Oct 31 12:42:12.637927 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:12.637900    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d21ba884-a8a6-4197-8b2a-f8373396ec51-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6adf0260-kcxpx\" (UID: \"d21ba884-a8a6-4197-8b2a-f8373396ec51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6adf0260-kcxpx"
Oct 31 12:42:12.638279 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:12.638263    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d21ba884-a8a6-4197-8b2a-f8373396ec51-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6adf0260-kcxpx\" (UID: \"d21ba884-a8a6-4197-8b2a-f8373396ec51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6adf0260-kcxpx"
Oct 31 12:42:12.638536 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:12.638482    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbflw\" (UniqueName: \"kubernetes.io/projected/d21ba884-a8a6-4197-8b2a-f8373396ec51-kube-api-access-tbflw\") pod \"maas-api-revoke-sub-6adf0260-kcxpx\" (UID: \"d21ba884-a8a6-4197-8b2a-f8373396ec51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6adf0260-kcxpx"
Oct 31 12:42:12.739702 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:12.739662    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d21ba884-a8a6-4197-8b2a-f8373396ec51-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6adf0260-kcxpx\" (UID: \"d21ba884-a8a6-4197-8b2a-f8373396ec51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6adf0260-kcxpx"
Oct 31 12:42:12.739849 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:12.739719    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbflw\" (UniqueName: \"kubernetes.io/projected/d21ba884-a8a6-4197-8b2a-f8373396ec51-kube-api-access-tbflw\") pod \"maas-api-revoke-sub-6adf0260-kcxpx\" (UID: \"d21ba884-a8a6-4197-8b2a-f8373396ec51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6adf0260-kcxpx"
Oct 31 12:42:12.739849 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:12.739773    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d21ba884-a8a6-4197-8b2a-f8373396ec51-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6adf0260-kcxpx\" (UID: \"d21ba884-a8a6-4197-8b2a-f8373396ec51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6adf0260-kcxpx"
Oct 31 12:42:12.740522 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:12.740463    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d21ba884-a8a6-4197-8b2a-f8373396ec51-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6adf0260-kcxpx\" (UID: \"d21ba884-a8a6-4197-8b2a-f8373396ec51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6adf0260-kcxpx"
Oct 31 12:42:12.742140 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:12.742114    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d21ba884-a8a6-4197-8b2a-f8373396ec51-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6adf0260-kcxpx\" (UID: \"d21ba884-a8a6-4197-8b2a-f8373396ec51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6adf0260-kcxpx"
Oct 31 12:42:12.762513 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:12.754858    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbflw\" (UniqueName: \"kubernetes.io/projected/d21ba884-a8a6-4197-8b2a-f8373396ec51-kube-api-access-tbflw\") pod \"maas-api-revoke-sub-6adf0260-kcxpx\" (UID: \"d21ba884-a8a6-4197-8b2a-f8373396ec51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6adf0260-kcxpx"
Oct 31 12:42:12.930846 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:12.930768    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6adf0260-kcxpx"
Oct 31 12:42:13.326514 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:13.323943    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6adf0260-kcxpx"]
Oct 31 12:42:13.385719 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:13.366026    5060 patch_prober.go:29] interesting pod/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-zswkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body=
Oct 31 12:42:13.385719 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:13.366084    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-zswkp" podUID="90387d5f-2bb8-4922-a1ad-9466e0745aaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused"
Oct 31 12:42:13.809698 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:13.809661    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6adf0260-kcxpx" event={"ID":"d21ba884-a8a6-4197-8b2a-f8373396ec51","Type":"ContainerStarted","Data":"5a5d97f3ddc9e5389793759f5c298afc8a032f5d7e4b7ac72af2540ecb7348d4"}
Oct 31 12:42:14.343187 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:14.343155    5060 patch_prober.go:29] interesting pod/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-zswkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body=
Oct 31 12:42:14.343360 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:14.343212    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-zswkp" podUID="90387d5f-2bb8-4922-a1ad-9466e0745aaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused"
Oct 31 12:42:14.375558 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:14.375518    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fbc4381f-ctm7g"]
Oct 31 12:42:14.419283 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:14.419241    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbc4381f-ctm7g"
Oct 31 12:42:14.424117 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:14.424082    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fbc4381f-ctm7g"]
Oct 31 12:42:14.600767 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:14.600625    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99d41b7a-82d6-4a10-a059-954635516f68-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fbc4381f-ctm7g\" (UID: \"99d41b7a-82d6-4a10-a059-954635516f68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbc4381f-ctm7g"
Oct 31 12:42:14.600918 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:14.600882    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5tpv\" (UniqueName: \"kubernetes.io/projected/99d41b7a-82d6-4a10-a059-954635516f68-kube-api-access-n5tpv\") pod \"maas-api-revoke-sub-fbc4381f-ctm7g\" (UID: \"99d41b7a-82d6-4a10-a059-954635516f68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbc4381f-ctm7g"
Oct 31 12:42:14.601145 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:14.601116    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99d41b7a-82d6-4a10-a059-954635516f68-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fbc4381f-ctm7g\" (UID: \"99d41b7a-82d6-4a10-a059-954635516f68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbc4381f-ctm7g"
Oct 31 12:42:14.702552 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:14.702517    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99d41b7a-82d6-4a10-a059-954635516f68-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fbc4381f-ctm7g\" (UID: \"99d41b7a-82d6-4a10-a059-954635516f68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbc4381f-ctm7g"
Oct 31 12:42:14.702735 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:14.702685    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99d41b7a-82d6-4a10-a059-954635516f68-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fbc4381f-ctm7g\" (UID: \"99d41b7a-82d6-4a10-a059-954635516f68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbc4381f-ctm7g"
Oct 31 12:42:14.702904 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:14.702882    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5tpv\" (UniqueName: \"kubernetes.io/projected/99d41b7a-82d6-4a10-a059-954635516f68-kube-api-access-n5tpv\") pod \"maas-api-revoke-sub-fbc4381f-ctm7g\" (UID: \"99d41b7a-82d6-4a10-a059-954635516f68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbc4381f-ctm7g"
Oct 31 12:42:14.703641 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:14.703613    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99d41b7a-82d6-4a10-a059-954635516f68-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fbc4381f-ctm7g\" (UID: \"99d41b7a-82d6-4a10-a059-954635516f68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbc4381f-ctm7g"
Oct 31 12:42:14.705808 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:14.705776    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99d41b7a-82d6-4a10-a059-954635516f68-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fbc4381f-ctm7g\" (UID: \"99d41b7a-82d6-4a10-a059-954635516f68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbc4381f-ctm7g"
Oct 31 12:42:14.715182 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:14.715157    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5tpv\" (UniqueName: \"kubernetes.io/projected/99d41b7a-82d6-4a10-a059-954635516f68-kube-api-access-n5tpv\") pod \"maas-api-revoke-sub-fbc4381f-ctm7g\" (UID: \"99d41b7a-82d6-4a10-a059-954635516f68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbc4381f-ctm7g"
Oct 31 12:42:14.735781 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:14.735755    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbc4381f-ctm7g"
Oct 31 12:42:14.830639 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:14.830605    5060 generic.go:356] "Generic (PLEG): container finished" podID="d21ba884-a8a6-4197-8b2a-f8373396ec51" containerID="b877904e4fff437fe599080c1e9653a1423e9b402fc56e4bae1812d795831b8a" exitCode=0
Oct 31 12:42:14.831077 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:14.830849    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6adf0260-kcxpx" event={"ID":"d21ba884-a8a6-4197-8b2a-f8373396ec51","Type":"ContainerDied","Data":"b877904e4fff437fe599080c1e9653a1423e9b402fc56e4bae1812d795831b8a"}
Oct 31 12:42:15.061763 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:15.061734    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fbc4381f-ctm7g"]
Oct 31 12:42:15.343777 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:15.343697    5060 patch_prober.go:29] interesting pod/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-zswkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body=
Oct 31 12:42:15.343777 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:15.343747    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-zswkp" podUID="90387d5f-2bb8-4922-a1ad-9466e0745aaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused"
Oct 31 12:42:15.888915 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:15.885506    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbc4381f-ctm7g" event={"ID":"99d41b7a-82d6-4a10-a059-954635516f68","Type":"ContainerStarted","Data":"6d08814ed4ebc1038e3e29f1caaaf69061150f60d4bbd52a2af4921e29e514a1"}
Oct 31 12:42:15.888915 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:15.885571    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbc4381f-ctm7g" event={"ID":"99d41b7a-82d6-4a10-a059-954635516f68","Type":"ContainerStarted","Data":"9eb399b65d139cab2f36033eea09157b68ac84c0bd4a3849e2e788339871ef55"}
Oct 31 12:42:15.944439 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:15.944375    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbc4381f-ctm7g" podStartSLOduration=1.944353065 podStartE2EDuration="1.944353065s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:42:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:42:15.938265301 +0000 UTC m=+899.041333029" watchObservedRunningTime="2026-10-31 12:42:15.944353065 +0000 UTC m=+899.047420779"
Oct 31 12:42:16.180539 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:16.180513    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6adf0260-kcxpx"
Oct 31 12:42:16.247069 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:16.247034    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d21ba884-a8a6-4197-8b2a-f8373396ec51-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d21ba884-a8a6-4197-8b2a-f8373396ec51-maas-api-xks-ca\") pod \"d21ba884-a8a6-4197-8b2a-f8373396ec51\" (UID: \"d21ba884-a8a6-4197-8b2a-f8373396ec51\") "
Oct 31 12:42:16.247266 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:16.247111    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d21ba884-a8a6-4197-8b2a-f8373396ec51-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d21ba884-a8a6-4197-8b2a-f8373396ec51-maas-api-openshift-ca\") pod \"d21ba884-a8a6-4197-8b2a-f8373396ec51\" (UID: \"d21ba884-a8a6-4197-8b2a-f8373396ec51\") "
Oct 31 12:42:16.247354 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:16.247285    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d21ba884-a8a6-4197-8b2a-f8373396ec51-kube-api-access-tbflw\" (UniqueName: \"kubernetes.io/projected/d21ba884-a8a6-4197-8b2a-f8373396ec51-kube-api-access-tbflw\") pod \"d21ba884-a8a6-4197-8b2a-f8373396ec51\" (UID: \"d21ba884-a8a6-4197-8b2a-f8373396ec51\") "
Oct 31 12:42:16.248012 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:16.247977    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d21ba884-a8a6-4197-8b2a-f8373396ec51-maas-api-openshift-ca" pod "d21ba884-a8a6-4197-8b2a-f8373396ec51" (UID: "d21ba884-a8a6-4197-8b2a-f8373396ec51"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:42:16.249419 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:16.249386    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d21ba884-a8a6-4197-8b2a-f8373396ec51-maas-api-xks-ca" pod "d21ba884-a8a6-4197-8b2a-f8373396ec51" (UID: "d21ba884-a8a6-4197-8b2a-f8373396ec51"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:42:16.250270 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:16.250241    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d21ba884-a8a6-4197-8b2a-f8373396ec51-kube-api-access-tbflw" pod "d21ba884-a8a6-4197-8b2a-f8373396ec51" (UID: "d21ba884-a8a6-4197-8b2a-f8373396ec51"). InnerVolumeSpecName "kube-api-access-tbflw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:42:16.347387 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:16.347353    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-zswkp"
Oct 31 12:42:16.347726 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:16.347705    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-zswkp"
Oct 31 12:42:16.348528 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:16.348475    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-zswkp"
Oct 31 12:42:16.348950 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:16.348929    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d21ba884-a8a6-4197-8b2a-f8373396ec51-maas-api-xks-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:42:16.349021 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:16.348956    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d21ba884-a8a6-4197-8b2a-f8373396ec51-maas-api-openshift-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:42:16.349021 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:16.348973    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbflw\" (UniqueName: \"kubernetes.io/projected/d21ba884-a8a6-4197-8b2a-f8373396ec51-kube-api-access-tbflw\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:42:16.894737 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:16.894695    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6adf0260-kcxpx"
Oct 31 12:42:16.894737 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:16.894718    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6adf0260-kcxpx" event={"ID":"d21ba884-a8a6-4197-8b2a-f8373396ec51","Type":"ContainerDied","Data":"5a5d97f3ddc9e5389793759f5c298afc8a032f5d7e4b7ac72af2540ecb7348d4"}
Oct 31 12:42:16.895183 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:16.894776    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5a5d97f3ddc9e5389793759f5c298afc8a032f5d7e4b7ac72af2540ecb7348d4"
Oct 31 12:42:16.897071 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:16.897043    5060 generic.go:356] "Generic (PLEG): container finished" podID="99d41b7a-82d6-4a10-a059-954635516f68" containerID="6d08814ed4ebc1038e3e29f1caaaf69061150f60d4bbd52a2af4921e29e514a1" exitCode=0
Oct 31 12:42:16.897222 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:16.897111    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbc4381f-ctm7g" event={"ID":"99d41b7a-82d6-4a10-a059-954635516f68","Type":"ContainerDied","Data":"6d08814ed4ebc1038e3e29f1caaaf69061150f60d4bbd52a2af4921e29e514a1"}
Oct 31 12:42:17.117817 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:17.117784    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a060601-ghnt5"]
Oct 31 12:42:17.118545 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:17.118530    5060 cpu_manager.go:414] "containerMap: removing container" podUID="d21ba884-a8a6-4197-8b2a-f8373396ec51" containerName="revoke-keys"
Oct 31 12:42:17.118597 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:17.118546    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d21ba884-a8a6-4197-8b2a-f8373396ec51" containerName="revoke-keys"
Oct 31 12:42:17.118683 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:17.118674    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="d21ba884-a8a6-4197-8b2a-f8373396ec51" containerName="revoke-keys"
Oct 31 12:42:17.126988 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:17.126956    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a060601-ghnt5"
Oct 31 12:42:17.141384 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:17.141349    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a060601-ghnt5"]
Oct 31 12:42:17.161273 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:17.161185    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8c8p8\" (UniqueName: \"kubernetes.io/projected/94144e6d-f757-4ebf-99c1-212dcf43fd9b-kube-api-access-8c8p8\") pod \"maas-api-revoke-sub-7a060601-ghnt5\" (UID: \"94144e6d-f757-4ebf-99c1-212dcf43fd9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a060601-ghnt5"
Oct 31 12:42:17.161433 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:17.161361    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94144e6d-f757-4ebf-99c1-212dcf43fd9b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7a060601-ghnt5\" (UID: \"94144e6d-f757-4ebf-99c1-212dcf43fd9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a060601-ghnt5"
Oct 31 12:42:17.161433 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:17.161392    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94144e6d-f757-4ebf-99c1-212dcf43fd9b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7a060601-ghnt5\" (UID: \"94144e6d-f757-4ebf-99c1-212dcf43fd9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a060601-ghnt5"
Oct 31 12:42:17.262679 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:17.262617    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8c8p8\" (UniqueName: \"kubernetes.io/projected/94144e6d-f757-4ebf-99c1-212dcf43fd9b-kube-api-access-8c8p8\") pod \"maas-api-revoke-sub-7a060601-ghnt5\" (UID: \"94144e6d-f757-4ebf-99c1-212dcf43fd9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a060601-ghnt5"
Oct 31 12:42:17.262885 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:17.262807    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94144e6d-f757-4ebf-99c1-212dcf43fd9b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7a060601-ghnt5\" (UID: \"94144e6d-f757-4ebf-99c1-212dcf43fd9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a060601-ghnt5"
Oct 31 12:42:17.262885 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:17.262837    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94144e6d-f757-4ebf-99c1-212dcf43fd9b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7a060601-ghnt5\" (UID: \"94144e6d-f757-4ebf-99c1-212dcf43fd9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a060601-ghnt5"
Oct 31 12:42:17.263802 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:17.263595    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94144e6d-f757-4ebf-99c1-212dcf43fd9b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7a060601-ghnt5\" (UID: \"94144e6d-f757-4ebf-99c1-212dcf43fd9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a060601-ghnt5"
Oct 31 12:42:17.265474 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:17.265453    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94144e6d-f757-4ebf-99c1-212dcf43fd9b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7a060601-ghnt5\" (UID: \"94144e6d-f757-4ebf-99c1-212dcf43fd9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a060601-ghnt5"
Oct 31 12:42:17.275754 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:17.275665    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8c8p8\" (UniqueName: \"kubernetes.io/projected/94144e6d-f757-4ebf-99c1-212dcf43fd9b-kube-api-access-8c8p8\") pod \"maas-api-revoke-sub-7a060601-ghnt5\" (UID: \"94144e6d-f757-4ebf-99c1-212dcf43fd9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a060601-ghnt5"
Oct 31 12:42:17.326834 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:17.326794    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pcvvm_f032909d-bc39-4c9e-99b7-f3d800842119/console-operator/2.log"
Oct 31 12:42:17.342499 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:17.342464    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pcvvm_f032909d-bc39-4c9e-99b7-f3d800842119/console-operator/2.log"
Oct 31 12:42:17.456416 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:17.456336    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a060601-ghnt5"
Oct 31 12:42:17.767587 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:42:17.767545    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94144e6d_f757_4ebf_99c1_212dcf43fd9b.slice/crio-355581739b6c3a329c18dfa6cdf7b6ab570665b5225b69e55f5101ee95469da0 WatchSource:0}: Error finding container 355581739b6c3a329c18dfa6cdf7b6ab570665b5225b69e55f5101ee95469da0: Status 404 returned error can't find the container with id 355581739b6c3a329c18dfa6cdf7b6ab570665b5225b69e55f5101ee95469da0
Oct 31 12:42:17.769680 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:17.769355    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a060601-ghnt5"]
Oct 31 12:42:17.911867 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:17.911827    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a060601-ghnt5" event={"ID":"94144e6d-f757-4ebf-99c1-212dcf43fd9b","Type":"ContainerStarted","Data":"355581739b6c3a329c18dfa6cdf7b6ab570665b5225b69e55f5101ee95469da0"}
Oct 31 12:42:18.079369 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:18.079340    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbc4381f-ctm7g"
Oct 31 12:42:18.184788 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:18.184757    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a614abb-6dw8n"]
Oct 31 12:42:18.185541 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:18.185520    5060 cpu_manager.go:414] "containerMap: removing container" podUID="99d41b7a-82d6-4a10-a059-954635516f68" containerName="revoke-keys"
Oct 31 12:42:18.185541 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:18.185540    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99d41b7a-82d6-4a10-a059-954635516f68" containerName="revoke-keys"
Oct 31 12:42:18.185750 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:18.185736    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="99d41b7a-82d6-4a10-a059-954635516f68" containerName="revoke-keys"
Oct 31 12:42:18.189925 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:18.189906    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a614abb-6dw8n"
Oct 31 12:42:18.207742 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:18.207525    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a614abb-6dw8n"]
Oct 31 12:42:18.262030 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:18.261984    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99d41b7a-82d6-4a10-a059-954635516f68-kube-api-access-n5tpv\" (UniqueName: \"kubernetes.io/projected/99d41b7a-82d6-4a10-a059-954635516f68-kube-api-access-n5tpv\") pod \"99d41b7a-82d6-4a10-a059-954635516f68\" (UID: \"99d41b7a-82d6-4a10-a059-954635516f68\") "
Oct 31 12:42:18.262216 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:18.262190    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99d41b7a-82d6-4a10-a059-954635516f68-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99d41b7a-82d6-4a10-a059-954635516f68-maas-api-openshift-ca\") pod \"99d41b7a-82d6-4a10-a059-954635516f68\" (UID: \"99d41b7a-82d6-4a10-a059-954635516f68\") "
Oct 31 12:42:18.262846 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:18.262816    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/99d41b7a-82d6-4a10-a059-954635516f68-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99d41b7a-82d6-4a10-a059-954635516f68-maas-api-xks-ca\") pod \"99d41b7a-82d6-4a10-a059-954635516f68\" (UID: \"99d41b7a-82d6-4a10-a059-954635516f68\") "
Oct 31 12:42:18.268836 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:18.263303    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99d41b7a-82d6-4a10-a059-954635516f68-maas-api-openshift-ca" pod "99d41b7a-82d6-4a10-a059-954635516f68" (UID: "99d41b7a-82d6-4a10-a059-954635516f68"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:42:18.268836 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:18.263875    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99d41b7a-82d6-4a10-a059-954635516f68-maas-api-openshift-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:42:18.268836 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:18.267210    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99d41b7a-82d6-4a10-a059-954635516f68-maas-api-xks-ca" pod "99d41b7a-82d6-4a10-a059-954635516f68" (UID: "99d41b7a-82d6-4a10-a059-954635516f68"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:42:18.268836 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:18.268695    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99d41b7a-82d6-4a10-a059-954635516f68-kube-api-access-n5tpv" pod "99d41b7a-82d6-4a10-a059-954635516f68" (UID: "99d41b7a-82d6-4a10-a059-954635516f68"). InnerVolumeSpecName "kube-api-access-n5tpv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:42:18.365670 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:18.365571    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/622ca715-506b-4684-96c1-a59c9ed0fe77-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8a614abb-6dw8n\" (UID: \"622ca715-506b-4684-96c1-a59c9ed0fe77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a614abb-6dw8n"
Oct 31 12:42:18.365835 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:18.365723    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrwzs\" (UniqueName: \"kubernetes.io/projected/622ca715-506b-4684-96c1-a59c9ed0fe77-kube-api-access-vrwzs\") pod \"maas-api-revoke-sub-8a614abb-6dw8n\" (UID: \"622ca715-506b-4684-96c1-a59c9ed0fe77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a614abb-6dw8n"
Oct 31 12:42:18.365904 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:18.365888    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/622ca715-506b-4684-96c1-a59c9ed0fe77-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8a614abb-6dw8n\" (UID: \"622ca715-506b-4684-96c1-a59c9ed0fe77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a614abb-6dw8n"
Oct 31 12:42:18.366015 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:18.366003    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5tpv\" (UniqueName: \"kubernetes.io/projected/99d41b7a-82d6-4a10-a059-954635516f68-kube-api-access-n5tpv\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:42:18.366072 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:18.366018    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99d41b7a-82d6-4a10-a059-954635516f68-maas-api-xks-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:42:18.466902 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:18.466864    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/622ca715-506b-4684-96c1-a59c9ed0fe77-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8a614abb-6dw8n\" (UID: \"622ca715-506b-4684-96c1-a59c9ed0fe77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a614abb-6dw8n"
Oct 31 12:42:18.467105 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:18.467036    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrwzs\" (UniqueName: \"kubernetes.io/projected/622ca715-506b-4684-96c1-a59c9ed0fe77-kube-api-access-vrwzs\") pod \"maas-api-revoke-sub-8a614abb-6dw8n\" (UID: \"622ca715-506b-4684-96c1-a59c9ed0fe77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a614abb-6dw8n"
Oct 31 12:42:18.467228 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:18.467207    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/622ca715-506b-4684-96c1-a59c9ed0fe77-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8a614abb-6dw8n\" (UID: \"622ca715-506b-4684-96c1-a59c9ed0fe77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a614abb-6dw8n"
Oct 31 12:42:18.467815 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:18.467784    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/622ca715-506b-4684-96c1-a59c9ed0fe77-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8a614abb-6dw8n\" (UID: \"622ca715-506b-4684-96c1-a59c9ed0fe77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a614abb-6dw8n"
Oct 31 12:42:18.473558 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:18.470058    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/622ca715-506b-4684-96c1-a59c9ed0fe77-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8a614abb-6dw8n\" (UID: \"622ca715-506b-4684-96c1-a59c9ed0fe77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a614abb-6dw8n"
Oct 31 12:42:18.479873 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:18.479842    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrwzs\" (UniqueName: \"kubernetes.io/projected/622ca715-506b-4684-96c1-a59c9ed0fe77-kube-api-access-vrwzs\") pod \"maas-api-revoke-sub-8a614abb-6dw8n\" (UID: \"622ca715-506b-4684-96c1-a59c9ed0fe77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a614abb-6dw8n"
Oct 31 12:42:18.516871 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:18.516847    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a614abb-6dw8n"
Oct 31 12:42:18.809062 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:18.809032    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a614abb-6dw8n"]
Oct 31 12:42:18.848373 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:42:18.848343    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod622ca715_506b_4684_96c1_a59c9ed0fe77.slice/crio-6620a842c65d8c3ef832b133ecf03dd0b3818b45ba819df602a8c03e6a75a50b WatchSource:0}: Error finding container 6620a842c65d8c3ef832b133ecf03dd0b3818b45ba819df602a8c03e6a75a50b: Status 404 returned error can't find the container with id 6620a842c65d8c3ef832b133ecf03dd0b3818b45ba819df602a8c03e6a75a50b
Oct 31 12:42:18.922406 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:18.922367    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbc4381f-ctm7g" event={"ID":"99d41b7a-82d6-4a10-a059-954635516f68","Type":"ContainerDied","Data":"9eb399b65d139cab2f36033eea09157b68ac84c0bd4a3849e2e788339871ef55"}
Oct 31 12:42:18.922406 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:18.922394    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbc4381f-ctm7g"
Oct 31 12:42:18.922827 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:18.922426    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9eb399b65d139cab2f36033eea09157b68ac84c0bd4a3849e2e788339871ef55"
Oct 31 12:42:18.923918 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:18.923892    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a614abb-6dw8n" event={"ID":"622ca715-506b-4684-96c1-a59c9ed0fe77","Type":"ContainerStarted","Data":"6620a842c65d8c3ef832b133ecf03dd0b3818b45ba819df602a8c03e6a75a50b"}
Oct 31 12:42:18.925884 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:18.925862    5060 generic.go:356] "Generic (PLEG): container finished" podID="94144e6d-f757-4ebf-99c1-212dcf43fd9b" containerID="ace316680fc1ffa9d4eee2000889ae0154c1fe4e42e7eac3928cbabd872d31c3" exitCode=0
Oct 31 12:42:18.925976 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:18.925904    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a060601-ghnt5" event={"ID":"94144e6d-f757-4ebf-99c1-212dcf43fd9b","Type":"ContainerDied","Data":"ace316680fc1ffa9d4eee2000889ae0154c1fe4e42e7eac3928cbabd872d31c3"}
Oct 31 12:42:19.939868 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:19.939834    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a614abb-6dw8n" event={"ID":"622ca715-506b-4684-96c1-a59c9ed0fe77","Type":"ContainerStarted","Data":"f77ceda4cc9823ffcb2589768f5fc6160d621266ba1e4eca73b51850f51a0f7e"}
Oct 31 12:42:19.968696 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:19.968633    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a614abb-6dw8n" podStartSLOduration=1.968612177 podStartE2EDuration="1.968612177s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:42:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:42:19.96123167 +0000 UTC m=+903.064299383" watchObservedRunningTime="2026-10-31 12:42:19.968612177 +0000 UTC m=+903.071679890"
Oct 31 12:42:20.217923 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:20.217896    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a060601-ghnt5"
Oct 31 12:42:20.316736 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:20.316706    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94144e6d-f757-4ebf-99c1-212dcf43fd9b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94144e6d-f757-4ebf-99c1-212dcf43fd9b-maas-api-openshift-ca\") pod \"94144e6d-f757-4ebf-99c1-212dcf43fd9b\" (UID: \"94144e6d-f757-4ebf-99c1-212dcf43fd9b\") "
Oct 31 12:42:20.316907 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:20.316868    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/94144e6d-f757-4ebf-99c1-212dcf43fd9b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94144e6d-f757-4ebf-99c1-212dcf43fd9b-maas-api-xks-ca\") pod \"94144e6d-f757-4ebf-99c1-212dcf43fd9b\" (UID: \"94144e6d-f757-4ebf-99c1-212dcf43fd9b\") "
Oct 31 12:42:20.317065 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:20.317048    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94144e6d-f757-4ebf-99c1-212dcf43fd9b-kube-api-access-8c8p8\" (UniqueName: \"kubernetes.io/projected/94144e6d-f757-4ebf-99c1-212dcf43fd9b-kube-api-access-8c8p8\") pod \"94144e6d-f757-4ebf-99c1-212dcf43fd9b\" (UID: \"94144e6d-f757-4ebf-99c1-212dcf43fd9b\") "
Oct 31 12:42:20.317117 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:20.317092    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94144e6d-f757-4ebf-99c1-212dcf43fd9b-maas-api-openshift-ca" pod "94144e6d-f757-4ebf-99c1-212dcf43fd9b" (UID: "94144e6d-f757-4ebf-99c1-212dcf43fd9b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:42:20.317736 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:20.317698    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94144e6d-f757-4ebf-99c1-212dcf43fd9b-maas-api-openshift-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:42:20.319196 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:20.319163    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94144e6d-f757-4ebf-99c1-212dcf43fd9b-maas-api-xks-ca" pod "94144e6d-f757-4ebf-99c1-212dcf43fd9b" (UID: "94144e6d-f757-4ebf-99c1-212dcf43fd9b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:42:20.319322 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:20.319297    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94144e6d-f757-4ebf-99c1-212dcf43fd9b-kube-api-access-8c8p8" pod "94144e6d-f757-4ebf-99c1-212dcf43fd9b" (UID: "94144e6d-f757-4ebf-99c1-212dcf43fd9b"). InnerVolumeSpecName "kube-api-access-8c8p8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:42:20.418776 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:20.418734    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8c8p8\" (UniqueName: \"kubernetes.io/projected/94144e6d-f757-4ebf-99c1-212dcf43fd9b-kube-api-access-8c8p8\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:42:20.418776 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:20.418769    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94144e6d-f757-4ebf-99c1-212dcf43fd9b-maas-api-xks-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:42:20.952869 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:20.952833    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a060601-ghnt5"
Oct 31 12:42:20.952869 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:20.952855    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a060601-ghnt5" event={"ID":"94144e6d-f757-4ebf-99c1-212dcf43fd9b","Type":"ContainerDied","Data":"355581739b6c3a329c18dfa6cdf7b6ab570665b5225b69e55f5101ee95469da0"}
Oct 31 12:42:20.953327 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:20.952912    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="355581739b6c3a329c18dfa6cdf7b6ab570665b5225b69e55f5101ee95469da0"
Oct 31 12:42:20.954793 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:20.954767    5060 generic.go:356] "Generic (PLEG): container finished" podID="622ca715-506b-4684-96c1-a59c9ed0fe77" containerID="f77ceda4cc9823ffcb2589768f5fc6160d621266ba1e4eca73b51850f51a0f7e" exitCode=0
Oct 31 12:42:20.954911 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:20.954848    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a614abb-6dw8n" event={"ID":"622ca715-506b-4684-96c1-a59c9ed0fe77","Type":"ContainerDied","Data":"f77ceda4cc9823ffcb2589768f5fc6160d621266ba1e4eca73b51850f51a0f7e"}
Oct 31 12:42:22.105515 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:22.093091    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-qqhsw"]
Oct 31 12:42:22.117573 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:22.107165    5060 cpu_manager.go:414] "containerMap: removing container" podUID="94144e6d-f757-4ebf-99c1-212dcf43fd9b" containerName="revoke-keys"
Oct 31 12:42:22.117573 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:22.107198    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94144e6d-f757-4ebf-99c1-212dcf43fd9b" containerName="revoke-keys"
Oct 31 12:42:22.117573 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:22.107441    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="94144e6d-f757-4ebf-99c1-212dcf43fd9b" containerName="revoke-keys"
Oct 31 12:42:22.146517 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:22.141774    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-qqhsw"
Oct 31 12:42:22.320581 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:22.315629    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9b854bc-d670-414d-8e11-dd7bd401a59f-istio-token\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-qqhsw\" (UID: \"d9b854bc-d670-414d-8e11-dd7bd401a59f\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-qqhsw"
Oct 31 12:42:22.320581 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:22.315721    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9b854bc-d670-414d-8e11-dd7bd401a59f-workload-socket\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-qqhsw\" (UID: \"d9b854bc-d670-414d-8e11-dd7bd401a59f\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-qqhsw"
Oct 31 12:42:22.320581 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:22.315800    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9b854bc-d670-414d-8e11-dd7bd401a59f-credential-socket\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-qqhsw\" (UID: \"d9b854bc-d670-414d-8e11-dd7bd401a59f\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-qqhsw"
Oct 31 12:42:22.320581 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:22.315825    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9b854bc-d670-414d-8e11-dd7bd401a59f-istio-data\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-qqhsw\" (UID: \"d9b854bc-d670-414d-8e11-dd7bd401a59f\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-qqhsw"
Oct 31 12:42:22.320581 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:22.315856    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9b854bc-d670-414d-8e11-dd7bd401a59f-workload-certs\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-qqhsw\" (UID: \"d9b854bc-d670-414d-8e11-dd7bd401a59f\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-qqhsw"
Oct 31 12:42:22.320581 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:22.316091    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9b854bc-d670-414d-8e11-dd7bd401a59f-istiod-ca-cert\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-qqhsw\" (UID: \"d9b854bc-d670-414d-8e11-dd7bd401a59f\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-qqhsw"
Oct 31 12:42:22.320581 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:22.316144    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9b854bc-d670-414d-8e11-dd7bd401a59f-istio-podinfo\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-qqhsw\" (UID: \"d9b854bc-d670-414d-8e11-dd7bd401a59f\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-qqhsw"
Oct 31 12:42:22.320581 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:22.316191    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9b854bc-d670-414d-8e11-dd7bd401a59f-istio-envoy\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-qqhsw\" (UID: \"d9b854bc-d670-414d-8e11-dd7bd401a59f\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-qqhsw"
Oct 31 12:42:22.320581 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:22.316222    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5hcq\" (UniqueName: \"kubernetes.io/projected/d9b854bc-d670-414d-8e11-dd7bd401a59f-kube-api-access-r5hcq\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-qqhsw\" (UID: \"d9b854bc-d670-414d-8e11-dd7bd401a59f\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-qqhsw"
Oct 31 12:42:22.404518 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:22.401340    5060 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x2qzv" podUID="c247ed2a-aa90-424b-9b7e-4f2936d67e3f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:42:22.445604 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:22.432366    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9b854bc-d670-414d-8e11-dd7bd401a59f-workload-socket\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-qqhsw\" (UID: \"d9b854bc-d670-414d-8e11-dd7bd401a59f\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-qqhsw"
Oct 31 12:42:22.445604 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:22.432469    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9b854bc-d670-414d-8e11-dd7bd401a59f-credential-socket\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-qqhsw\" (UID: \"d9b854bc-d670-414d-8e11-dd7bd401a59f\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-qqhsw"
Oct 31 12:42:22.445604 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:22.432526    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9b854bc-d670-414d-8e11-dd7bd401a59f-istio-data\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-qqhsw\" (UID: \"d9b854bc-d670-414d-8e11-dd7bd401a59f\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-qqhsw"
Oct 31 12:42:22.445604 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:22.432559    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9b854bc-d670-414d-8e11-dd7bd401a59f-workload-certs\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-qqhsw\" (UID: \"d9b854bc-d670-414d-8e11-dd7bd401a59f\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-qqhsw"
Oct 31 12:42:22.445604 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:22.432788    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9b854bc-d670-414d-8e11-dd7bd401a59f-istiod-ca-cert\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-qqhsw\" (UID: \"d9b854bc-d670-414d-8e11-dd7bd401a59f\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-qqhsw"
Oct 31 12:42:22.445604 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:22.432845    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9b854bc-d670-414d-8e11-dd7bd401a59f-istio-podinfo\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-qqhsw\" (UID: \"d9b854bc-d670-414d-8e11-dd7bd401a59f\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-qqhsw"
Oct 31 12:42:22.445604 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:22.432889    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9b854bc-d670-414d-8e11-dd7bd401a59f-istio-envoy\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-qqhsw\" (UID: \"d9b854bc-d670-414d-8e11-dd7bd401a59f\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-qqhsw"
Oct 31 12:42:22.445604 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:22.432922    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5hcq\" (UniqueName: \"kubernetes.io/projected/d9b854bc-d670-414d-8e11-dd7bd401a59f-kube-api-access-r5hcq\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-qqhsw\" (UID: \"d9b854bc-d670-414d-8e11-dd7bd401a59f\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-qqhsw"
Oct 31 12:42:22.445604 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:22.432960    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9b854bc-d670-414d-8e11-dd7bd401a59f-istio-token\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-qqhsw\" (UID: \"d9b854bc-d670-414d-8e11-dd7bd401a59f\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-qqhsw"
Oct 31 12:42:22.445604 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:22.433136    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9b854bc-d670-414d-8e11-dd7bd401a59f-credential-socket\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-qqhsw\" (UID: \"d9b854bc-d670-414d-8e11-dd7bd401a59f\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-qqhsw"
Oct 31 12:42:22.445604 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:22.433427    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9b854bc-d670-414d-8e11-dd7bd401a59f-workload-socket\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-qqhsw\" (UID: \"d9b854bc-d670-414d-8e11-dd7bd401a59f\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-qqhsw"
Oct 31 12:42:22.445604 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:22.433666    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9b854bc-d670-414d-8e11-dd7bd401a59f-istio-data\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-qqhsw\" (UID: \"d9b854bc-d670-414d-8e11-dd7bd401a59f\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-qqhsw"
Oct 31 12:42:22.445604 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:22.438189    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9b854bc-d670-414d-8e11-dd7bd401a59f-istiod-ca-cert\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-qqhsw\" (UID: \"d9b854bc-d670-414d-8e11-dd7bd401a59f\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-qqhsw"
Oct 31 12:42:22.465206 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:22.457407    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9b854bc-d670-414d-8e11-dd7bd401a59f-workload-certs\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-qqhsw\" (UID: \"d9b854bc-d670-414d-8e11-dd7bd401a59f\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-qqhsw"
Oct 31 12:42:22.562883 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:22.553170    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-2-w5-b25d3b-kserve-76f798d59f-vvzhg" podUID="394ed76c-84ac-4612-9acb-a7e36cb68d89" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": context deadline exceeded"
Oct 31 12:42:22.562883 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:22.555420    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/simulator-w4-c67de2-kserve-868679cf9b-gqjbx" podUID="e104d7da-1fb2-4c99-a480-b4ae84c4f273" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:42:22.562883 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:22.555470    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/premium-w2-e6c79d-kserve-6468bf46d4-h66f8" podUID="cd6a266f-8d12-4089-83ee-c75c716bf26c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Oct 31 12:42:22.575728 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:22.575552    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-qqhsw"]
Oct 31 12:42:22.638526 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:22.631768    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x2qzv" podUID="c247ed2a-aa90-424b-9b7e-4f2936d67e3f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:42:22.662452 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:22.644133    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9b854bc-d670-414d-8e11-dd7bd401a59f-istio-podinfo\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-qqhsw\" (UID: \"d9b854bc-d670-414d-8e11-dd7bd401a59f\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-qqhsw"
Oct 31 12:42:22.664085 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:22.664052    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5hcq\" (UniqueName: \"kubernetes.io/projected/d9b854bc-d670-414d-8e11-dd7bd401a59f-kube-api-access-r5hcq\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-qqhsw\" (UID: \"d9b854bc-d670-414d-8e11-dd7bd401a59f\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-qqhsw"
Oct 31 12:42:22.743162 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:22.743123    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9b854bc-d670-414d-8e11-dd7bd401a59f-istio-envoy\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-qqhsw\" (UID: \"d9b854bc-d670-414d-8e11-dd7bd401a59f\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-qqhsw"
Oct 31 12:42:22.751513 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:22.743880    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9b854bc-d670-414d-8e11-dd7bd401a59f-istio-token\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-qqhsw\" (UID: \"d9b854bc-d670-414d-8e11-dd7bd401a59f\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-qqhsw"
Oct 31 12:42:22.873187 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:22.872783    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-qqhsw"
Oct 31 12:42:23.049213 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:23.049167    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-jnpqm"]
Oct 31 12:42:23.092531 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:23.089408    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-jnpqm"
Oct 31 12:42:23.181520 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:23.181469    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-pz8cf"]
Oct 31 12:42:23.198809 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:23.198764    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe33e47d-4eb8-411b-8320-21135401155b-istio-envoy\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-jnpqm\" (UID: \"fe33e47d-4eb8-411b-8320-21135401155b\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-jnpqm"
Oct 31 12:42:23.199092 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:23.199067    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe33e47d-4eb8-411b-8320-21135401155b-workload-certs\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-jnpqm\" (UID: \"fe33e47d-4eb8-411b-8320-21135401155b\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-jnpqm"
Oct 31 12:42:23.199235 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:23.199212    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqg8z\" (UniqueName: \"kubernetes.io/projected/fe33e47d-4eb8-411b-8320-21135401155b-kube-api-access-pqg8z\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-jnpqm\" (UID: \"fe33e47d-4eb8-411b-8320-21135401155b\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-jnpqm"
Oct 31 12:42:23.199429 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:23.199410    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe33e47d-4eb8-411b-8320-21135401155b-istio-data\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-jnpqm\" (UID: \"fe33e47d-4eb8-411b-8320-21135401155b\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-jnpqm"
Oct 31 12:42:23.199586 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:23.199568    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe33e47d-4eb8-411b-8320-21135401155b-istiod-ca-cert\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-jnpqm\" (UID: \"fe33e47d-4eb8-411b-8320-21135401155b\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-jnpqm"
Oct 31 12:42:23.199729 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:23.199713    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe33e47d-4eb8-411b-8320-21135401155b-credential-socket\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-jnpqm\" (UID: \"fe33e47d-4eb8-411b-8320-21135401155b\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-jnpqm"
Oct 31 12:42:23.199843 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:23.199827    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe33e47d-4eb8-411b-8320-21135401155b-istio-podinfo\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-jnpqm\" (UID: \"fe33e47d-4eb8-411b-8320-21135401155b\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-jnpqm"
Oct 31 12:42:23.199968 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:23.199952    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe33e47d-4eb8-411b-8320-21135401155b-workload-socket\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-jnpqm\" (UID: \"fe33e47d-4eb8-411b-8320-21135401155b\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-jnpqm"
Oct 31 12:42:23.200201 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:23.200183    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe33e47d-4eb8-411b-8320-21135401155b-istio-token\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-jnpqm\" (UID: \"fe33e47d-4eb8-411b-8320-21135401155b\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-jnpqm"
Oct 31 12:42:23.268554 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:23.248570    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-jnpqm"]
Oct 31 12:42:23.269271 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:23.257345    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-pz8cf"
Oct 31 12:42:23.351016 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:23.335304    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe33e47d-4eb8-411b-8320-21135401155b-istio-data\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-jnpqm\" (UID: \"fe33e47d-4eb8-411b-8320-21135401155b\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-jnpqm"
Oct 31 12:42:23.351016 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:23.335364    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe33e47d-4eb8-411b-8320-21135401155b-istiod-ca-cert\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-jnpqm\" (UID: \"fe33e47d-4eb8-411b-8320-21135401155b\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-jnpqm"
Oct 31 12:42:23.351016 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:23.335404    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe33e47d-4eb8-411b-8320-21135401155b-credential-socket\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-jnpqm\" (UID: \"fe33e47d-4eb8-411b-8320-21135401155b\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-jnpqm"
Oct 31 12:42:23.351016 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:23.335439    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe33e47d-4eb8-411b-8320-21135401155b-istio-podinfo\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-jnpqm\" (UID: \"fe33e47d-4eb8-411b-8320-21135401155b\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-jnpqm"
Oct 31 12:42:23.351016 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:23.335515    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe33e47d-4eb8-411b-8320-21135401155b-workload-socket\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-jnpqm\" (UID: \"fe33e47d-4eb8-411b-8320-21135401155b\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-jnpqm"
Oct 31 12:42:23.351016 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:23.335679    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe33e47d-4eb8-411b-8320-21135401155b-istio-token\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-jnpqm\" (UID: \"fe33e47d-4eb8-411b-8320-21135401155b\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-jnpqm"
Oct 31 12:42:23.351016 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:23.335817    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe33e47d-4eb8-411b-8320-21135401155b-istio-data\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-jnpqm\" (UID: \"fe33e47d-4eb8-411b-8320-21135401155b\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-jnpqm"
Oct 31 12:42:23.351016 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:23.335849    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe33e47d-4eb8-411b-8320-21135401155b-istio-envoy\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-jnpqm\" (UID: \"fe33e47d-4eb8-411b-8320-21135401155b\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-jnpqm"
Oct 31 12:42:23.351016 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:23.335901    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe33e47d-4eb8-411b-8320-21135401155b-workload-certs\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-jnpqm\" (UID: \"fe33e47d-4eb8-411b-8320-21135401155b\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-jnpqm"
Oct 31 12:42:23.351016 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:23.335935    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqg8z\" (UniqueName: \"kubernetes.io/projected/fe33e47d-4eb8-411b-8320-21135401155b-kube-api-access-pqg8z\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-jnpqm\" (UID: \"fe33e47d-4eb8-411b-8320-21135401155b\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-jnpqm"
Oct 31 12:42:23.351016 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:23.336845    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe33e47d-4eb8-411b-8320-21135401155b-istiod-ca-cert\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-jnpqm\" (UID: \"fe33e47d-4eb8-411b-8320-21135401155b\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-jnpqm"
Oct 31 12:42:23.351016 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:23.337105    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe33e47d-4eb8-411b-8320-21135401155b-credential-socket\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-jnpqm\" (UID: \"fe33e47d-4eb8-411b-8320-21135401155b\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-jnpqm"
Oct 31 12:42:23.390569 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:23.363749    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe33e47d-4eb8-411b-8320-21135401155b-workload-socket\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-jnpqm\" (UID: \"fe33e47d-4eb8-411b-8320-21135401155b\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-jnpqm"
Oct 31 12:42:23.390569 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:23.364060    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe33e47d-4eb8-411b-8320-21135401155b-workload-certs\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-jnpqm\" (UID: \"fe33e47d-4eb8-411b-8320-21135401155b\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-jnpqm"
Oct 31 12:42:23.470885 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:23.466336    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/326d88b8-e041-4b4f-ab99-f1d2622d76b4-istio-podinfo\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-pz8cf\" (UID: \"326d88b8-e041-4b4f-ab99-f1d2622d76b4\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-pz8cf"
Oct 31 12:42:23.470885 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:23.466390    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/326d88b8-e041-4b4f-ab99-f1d2622d76b4-istio-envoy\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-pz8cf\" (UID: \"326d88b8-e041-4b4f-ab99-f1d2622d76b4\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-pz8cf"
Oct 31 12:42:23.470885 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:23.466511    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/326d88b8-e041-4b4f-ab99-f1d2622d76b4-credential-socket\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-pz8cf\" (UID: \"326d88b8-e041-4b4f-ab99-f1d2622d76b4\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-pz8cf"
Oct 31 12:42:23.470885 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:23.466591    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/326d88b8-e041-4b4f-ab99-f1d2622d76b4-workload-socket\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-pz8cf\" (UID: \"326d88b8-e041-4b4f-ab99-f1d2622d76b4\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-pz8cf"
Oct 31 12:42:23.470885 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:23.466650    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rx486\" (UniqueName: \"kubernetes.io/projected/326d88b8-e041-4b4f-ab99-f1d2622d76b4-kube-api-access-rx486\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-pz8cf\" (UID: \"326d88b8-e041-4b4f-ab99-f1d2622d76b4\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-pz8cf"
Oct 31 12:42:23.470885 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:23.466717    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/326d88b8-e041-4b4f-ab99-f1d2622d76b4-istio-token\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-pz8cf\" (UID: \"326d88b8-e041-4b4f-ab99-f1d2622d76b4\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-pz8cf"
Oct 31 12:42:23.470885 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:23.466753    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/326d88b8-e041-4b4f-ab99-f1d2622d76b4-workload-certs\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-pz8cf\" (UID: \"326d88b8-e041-4b4f-ab99-f1d2622d76b4\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-pz8cf"
Oct 31 12:42:23.470885 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:23.466845    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/326d88b8-e041-4b4f-ab99-f1d2622d76b4-istiod-ca-cert\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-pz8cf\" (UID: \"326d88b8-e041-4b4f-ab99-f1d2622d76b4\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-pz8cf"
Oct 31 12:42:23.470885 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:23.466880    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/326d88b8-e041-4b4f-ab99-f1d2622d76b4-istio-data\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-pz8cf\" (UID: \"326d88b8-e041-4b4f-ab99-f1d2622d76b4\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-pz8cf"
Oct 31 12:42:23.560572 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:23.556232    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe33e47d-4eb8-411b-8320-21135401155b-istio-envoy\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-jnpqm\" (UID: \"fe33e47d-4eb8-411b-8320-21135401155b\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-jnpqm"
Oct 31 12:42:23.560572 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:23.556330    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/premium-w2-e6c79d-kserve-6468bf46d4-h66f8" podUID="cd6a266f-8d12-4089-83ee-c75c716bf26c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:42:23.560572 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:23.556923    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e6c79d/premium-w2-e6c79d-kserve-6468bf46d4-h66f8"
Oct 31 12:42:23.560572 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:23.558184    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/simulator-w4-c67de2-kserve-868679cf9b-gqjbx" podUID="e104d7da-1fb2-4c99-a480-b4ae84c4f273" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:42:23.560572 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:23.558505    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-2-w5-b25d3b-kserve-76f798d59f-vvzhg" podUID="394ed76c-84ac-4612-9acb-a7e36cb68d89" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": context deadline exceeded"
Oct 31 12:42:23.578115 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:23.578064    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c67de2/simulator-w4-c67de2-kserve-868679cf9b-gqjbx"
Oct 31 12:42:23.603875 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:23.603847    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-2-w5-b25d3b-kserve-76f798d59f-vvzhg"
Oct 31 12:42:23.608828 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:23.584219    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/326d88b8-e041-4b4f-ab99-f1d2622d76b4-istio-podinfo\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-pz8cf\" (UID: \"326d88b8-e041-4b4f-ab99-f1d2622d76b4\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-pz8cf"
Oct 31 12:42:23.609071 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:23.609039    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/326d88b8-e041-4b4f-ab99-f1d2622d76b4-istio-envoy\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-pz8cf\" (UID: \"326d88b8-e041-4b4f-ab99-f1d2622d76b4\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-pz8cf"
Oct 31 12:42:23.616737 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:23.616695    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/326d88b8-e041-4b4f-ab99-f1d2622d76b4-credential-socket\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-pz8cf\" (UID: \"326d88b8-e041-4b4f-ab99-f1d2622d76b4\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-pz8cf"
Oct 31 12:42:23.617064 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:23.617031    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/326d88b8-e041-4b4f-ab99-f1d2622d76b4-workload-socket\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-pz8cf\" (UID: \"326d88b8-e041-4b4f-ab99-f1d2622d76b4\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-pz8cf"
Oct 31 12:42:23.617309 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:23.617289    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rx486\" (UniqueName: \"kubernetes.io/projected/326d88b8-e041-4b4f-ab99-f1d2622d76b4-kube-api-access-rx486\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-pz8cf\" (UID: \"326d88b8-e041-4b4f-ab99-f1d2622d76b4\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-pz8cf"
Oct 31 12:42:23.617547 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:23.617529    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/326d88b8-e041-4b4f-ab99-f1d2622d76b4-istio-token\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-pz8cf\" (UID: \"326d88b8-e041-4b4f-ab99-f1d2622d76b4\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-pz8cf"
Oct 31 12:42:23.648607 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:23.644563    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/326d88b8-e041-4b4f-ab99-f1d2622d76b4-workload-socket\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-pz8cf\" (UID: \"326d88b8-e041-4b4f-ab99-f1d2622d76b4\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-pz8cf"
Oct 31 12:42:23.660927 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:23.650404    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/326d88b8-e041-4b4f-ab99-f1d2622d76b4-credential-socket\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-pz8cf\" (UID: \"326d88b8-e041-4b4f-ab99-f1d2622d76b4\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-pz8cf"
Oct 31 12:42:23.660927 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:23.650476    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x2qzv" podUID="c247ed2a-aa90-424b-9b7e-4f2936d67e3f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:42:23.660927 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:23.650849    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x2qzv"
Oct 31 12:42:23.687921 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:23.617712    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/326d88b8-e041-4b4f-ab99-f1d2622d76b4-workload-certs\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-pz8cf\" (UID: \"326d88b8-e041-4b4f-ab99-f1d2622d76b4\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-pz8cf"
Oct 31 12:42:23.687921 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:23.670720    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/326d88b8-e041-4b4f-ab99-f1d2622d76b4-istiod-ca-cert\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-pz8cf\" (UID: \"326d88b8-e041-4b4f-ab99-f1d2622d76b4\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-pz8cf"
Oct 31 12:42:23.687921 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:23.670765    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/326d88b8-e041-4b4f-ab99-f1d2622d76b4-istio-data\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-pz8cf\" (UID: \"326d88b8-e041-4b4f-ab99-f1d2622d76b4\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-pz8cf"
Oct 31 12:42:23.687921 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:23.671412    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/326d88b8-e041-4b4f-ab99-f1d2622d76b4-istio-data\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-pz8cf\" (UID: \"326d88b8-e041-4b4f-ab99-f1d2622d76b4\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-pz8cf"
Oct 31 12:42:23.687921 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:23.672440    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/326d88b8-e041-4b4f-ab99-f1d2622d76b4-istiod-ca-cert\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-pz8cf\" (UID: \"326d88b8-e041-4b4f-ab99-f1d2622d76b4\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-pz8cf"
Oct 31 12:42:23.743797 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:23.734733    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/326d88b8-e041-4b4f-ab99-f1d2622d76b4-workload-certs\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-pz8cf\" (UID: \"326d88b8-e041-4b4f-ab99-f1d2622d76b4\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-pz8cf"
Oct 31 12:42:23.992573 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:23.987331    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe33e47d-4eb8-411b-8320-21135401155b-istio-podinfo\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-jnpqm\" (UID: \"fe33e47d-4eb8-411b-8320-21135401155b\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-jnpqm"
Oct 31 12:42:23.994580 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:23.993067    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe33e47d-4eb8-411b-8320-21135401155b-istio-token\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-jnpqm\" (UID: \"fe33e47d-4eb8-411b-8320-21135401155b\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-jnpqm"
Oct 31 12:42:24.320524 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:24.295424    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqg8z\" (UniqueName: \"kubernetes.io/projected/fe33e47d-4eb8-411b-8320-21135401155b-kube-api-access-pqg8z\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-jnpqm\" (UID: \"fe33e47d-4eb8-411b-8320-21135401155b\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-jnpqm"
Oct 31 12:42:24.320524 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:24.298715    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/326d88b8-e041-4b4f-ab99-f1d2622d76b4-istio-podinfo\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-pz8cf\" (UID: \"326d88b8-e041-4b4f-ab99-f1d2622d76b4\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-pz8cf"
Oct 31 12:42:24.348017 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:24.347970    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-pz8cf"]
Oct 31 12:42:24.492771 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:24.452771    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/326d88b8-e041-4b4f-ab99-f1d2622d76b4-istio-token\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-pz8cf\" (UID: \"326d88b8-e041-4b4f-ab99-f1d2622d76b4\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-pz8cf"
Oct 31 12:42:24.492771 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:24.453087    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-jnpqm"
Oct 31 12:42:24.535718 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:24.533903    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/326d88b8-e041-4b4f-ab99-f1d2622d76b4-istio-envoy\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-pz8cf\" (UID: \"326d88b8-e041-4b4f-ab99-f1d2622d76b4\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-pz8cf"
Oct 31 12:42:24.577575 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:24.566790    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-2-w5-b25d3b-kserve-76f798d59f-vvzhg" podUID="394ed76c-84ac-4612-9acb-a7e36cb68d89" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:42:24.992607 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:24.881271    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rx486\" (UniqueName: \"kubernetes.io/projected/326d88b8-e041-4b4f-ab99-f1d2622d76b4-kube-api-access-rx486\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-pz8cf\" (UID: \"326d88b8-e041-4b4f-ab99-f1d2622d76b4\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-pz8cf"
Oct 31 12:42:25.021960 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:24.604450    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/premium-w2-e6c79d-kserve-6468bf46d4-h66f8" podUID="cd6a266f-8d12-4089-83ee-c75c716bf26c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:42:25.021960 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:24.660058    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x2qzv" podUID="c247ed2a-aa90-424b-9b7e-4f2936d67e3f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:42:25.057579 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:24.583374    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/simulator-w4-c67de2-kserve-868679cf9b-gqjbx" podUID="e104d7da-1fb2-4c99-a480-b4ae84c4f273" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:42:25.437524 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:25.432249    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-pz8cf"
Oct 31 12:42:25.611518 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:25.585527    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a614abb-6dw8n"
Oct 31 12:42:26.206211 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:26.206169    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/simulator-w4-c67de2-kserve-868679cf9b-gqjbx" podUID="e104d7da-1fb2-4c99-a480-b4ae84c4f273" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded"
Oct 31 12:42:26.487516 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:26.459207    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/premium-w2-e6c79d-kserve-6468bf46d4-h66f8" podUID="cd6a266f-8d12-4089-83ee-c75c716bf26c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Oct 31 12:42:26.489217 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:26.489172    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x2qzv" podUID="c247ed2a-aa90-424b-9b7e-4f2936d67e3f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded"
Oct 31 12:42:26.568523 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:26.542893    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-2-w5-b25d3b-kserve-76f798d59f-vvzhg" podUID="394ed76c-84ac-4612-9acb-a7e36cb68d89" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:42:26.705638 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:26.562857    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-rbn8q" podUID="dea929f7-61c3-4921-9bdb-106f763e067e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:42:26.942532 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:26.939710    5060 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-2-w5-b25d3b-kserve-76f798d59f-vvzhg" podUID="394ed76c-84ac-4612-9acb-a7e36cb68d89" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:42:27.017566 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:27.013773    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/622ca715-506b-4684-96c1-a59c9ed0fe77-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/622ca715-506b-4684-96c1-a59c9ed0fe77-maas-api-openshift-ca\") pod \"622ca715-506b-4684-96c1-a59c9ed0fe77\" (UID: \"622ca715-506b-4684-96c1-a59c9ed0fe77\") "
Oct 31 12:42:27.120563 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:42:27.093802    5060 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="2.052s"
Oct 31 12:42:27.143550 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:27.027118    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/622ca715-506b-4684-96c1-a59c9ed0fe77-maas-api-openshift-ca" pod "622ca715-506b-4684-96c1-a59c9ed0fe77" (UID: "622ca715-506b-4684-96c1-a59c9ed0fe77"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:42:27.213217 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:27.213173    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/simulator-w4-c67de2-kserve-868679cf9b-gqjbx" podUID="e104d7da-1fb2-4c99-a480-b4ae84c4f273" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:42:27.266593 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:27.017013    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:42:27.266942 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:27.266899    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:42:27.267121 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:27.267100    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:42:27.303736 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:27.272249    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/622ca715-506b-4684-96c1-a59c9ed0fe77-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/622ca715-506b-4684-96c1-a59c9ed0fe77-maas-api-xks-ca\") pod \"622ca715-506b-4684-96c1-a59c9ed0fe77\" (UID: \"622ca715-506b-4684-96c1-a59c9ed0fe77\") "
Oct 31 12:42:27.303736 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:27.272361    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/622ca715-506b-4684-96c1-a59c9ed0fe77-kube-api-access-vrwzs\" (UniqueName: \"kubernetes.io/projected/622ca715-506b-4684-96c1-a59c9ed0fe77-kube-api-access-vrwzs\") pod \"622ca715-506b-4684-96c1-a59c9ed0fe77\" (UID: \"622ca715-506b-4684-96c1-a59c9ed0fe77\") "
Oct 31 12:42:27.323843 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:27.323803    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/622ca715-506b-4684-96c1-a59c9ed0fe77-maas-api-xks-ca" pod "622ca715-506b-4684-96c1-a59c9ed0fe77" (UID: "622ca715-506b-4684-96c1-a59c9ed0fe77"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:42:27.337856 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:27.328547    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/622ca715-506b-4684-96c1-a59c9ed0fe77-maas-api-openshift-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:42:27.538360 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:27.521457    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x2qzv" podUID="c247ed2a-aa90-424b-9b7e-4f2936d67e3f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:42:27.564572 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:27.530714    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/premium-w2-e6c79d-kserve-6468bf46d4-h66f8" podUID="cd6a266f-8d12-4089-83ee-c75c716bf26c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:42:27.599990 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:27.599942    5060 prober.go:122] "Probe failed" probeType="Liveness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-kgjh7" podUID="4f8464cd-b924-44f9-821b-f4c683531f45" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.132.0.43:6080/livez\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:42:27.600716 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:27.600684    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-kgjh7" podUID="4f8464cd-b924-44f9-821b-f4c683531f45" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.132.0.43:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:42:27.620565 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:27.620512    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-2-w5-b25d3b-kserve-76f798d59f-vvzhg" podUID="394ed76c-84ac-4612-9acb-a7e36cb68d89" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:42:27.727621 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:27.631586    5060 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-c67de2/simulator-w4-c67de2-kserve-868679cf9b-gqjbx" podUID="e104d7da-1fb2-4c99-a480-b4ae84c4f273" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": context deadline exceeded"
Oct 31 12:42:27.727621 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:27.631719    5060 patch_prober.go:29] interesting pod/e2e-worker-w1-85ce08-openshift-default-77d6458845-58cq4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 12:42:27.727621 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:27.631753    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-58cq4" podUID="c86d1b8b-6bc6-45ff-9412-179f069cf080" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:42:27.727621 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:27.679786    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/622ca715-506b-4684-96c1-a59c9ed0fe77-maas-api-xks-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:42:28.056520 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:28.055774    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/622ca715-506b-4684-96c1-a59c9ed0fe77-kube-api-access-vrwzs" pod "622ca715-506b-4684-96c1-a59c9ed0fe77" (UID: "622ca715-506b-4684-96c1-a59c9ed0fe77"). InnerVolumeSpecName "kube-api-access-vrwzs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:42:28.085943 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:28.085909    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/622ca715-506b-4684-96c1-a59c9ed0fe77-kube-api-access-vrwzs\" (UniqueName: \"kubernetes.io/projected/622ca715-506b-4684-96c1-a59c9ed0fe77-kube-api-access-vrwzs\") pod \"622ca715-506b-4684-96c1-a59c9ed0fe77\" (UID: \"622ca715-506b-4684-96c1-a59c9ed0fe77\") "
Oct 31 12:42:28.087291 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:42:28.087256    5060 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/622ca715-506b-4684-96c1-a59c9ed0fe77/volumes/kubernetes.io~projected/kube-api-access-vrwzs
Oct 31 12:42:28.087525 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:28.087506    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/622ca715-506b-4684-96c1-a59c9ed0fe77-kube-api-access-vrwzs" pod "622ca715-506b-4684-96c1-a59c9ed0fe77" (UID: "622ca715-506b-4684-96c1-a59c9ed0fe77"). InnerVolumeSpecName "kube-api-access-vrwzs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:42:28.179559 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:42:28.144914    5060 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.051s"
Oct 31 12:42:28.180745 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:28.180694    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a614abb-6dw8n" event={"ID":"622ca715-506b-4684-96c1-a59c9ed0fe77","Type":"ContainerDied","Data":"6620a842c65d8c3ef832b133ecf03dd0b3818b45ba819df602a8c03e6a75a50b"}
Oct 31 12:42:28.181017 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:28.181000    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6620a842c65d8c3ef832b133ecf03dd0b3818b45ba819df602a8c03e6a75a50b"
Oct 31 12:42:28.283645 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:28.174989    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a614abb-6dw8n"
Oct 31 12:42:28.331515 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:28.320029    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrwzs\" (UniqueName: \"kubernetes.io/projected/622ca715-506b-4684-96c1-a59c9ed0fe77-kube-api-access-vrwzs\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:42:28.650516 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:28.625088    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x2qzv" podUID="c247ed2a-aa90-424b-9b7e-4f2936d67e3f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded"
Oct 31 12:42:28.733695 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:28.658571    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-2-w5-b25d3b-kserve-76f798d59f-vvzhg" podUID="394ed76c-84ac-4612-9acb-a7e36cb68d89" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:42:28.733695 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:28.697556    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/premium-w2-e6c79d-kserve-6468bf46d4-h66f8" podUID="cd6a266f-8d12-4089-83ee-c75c716bf26c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Oct 31 12:42:28.733695 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:28.699286    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/simulator-w4-c67de2-kserve-868679cf9b-gqjbx" podUID="e104d7da-1fb2-4c99-a480-b4ae84c4f273" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded"
Oct 31 12:42:28.981421 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:28.981366    5060 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-e6c79d/premium-w2-e6c79d-kserve-6468bf46d4-h66f8" podUID="cd6a266f-8d12-4089-83ee-c75c716bf26c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:42:29.059789 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:42:29.056636    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe33e47d_4eb8_411b_8320_21135401155b.slice/crio-e38284c36244ff49f621a9313984ef52ae5792dd3c02262a1f061a2cdb46a5c0 WatchSource:0}: Error finding container e38284c36244ff49f621a9313984ef52ae5792dd3c02262a1f061a2cdb46a5c0: Status 404 returned error can't find the container with id e38284c36244ff49f621a9313984ef52ae5792dd3c02262a1f061a2cdb46a5c0
Oct 31 12:42:29.216233 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:29.216185    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:42:29.216606 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:29.216554    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:42:29.216847 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:29.216809    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:42:29.720834 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:29.707235    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/simulator-w4-c67de2-kserve-868679cf9b-gqjbx" podUID="e104d7da-1fb2-4c99-a480-b4ae84c4f273" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:42:29.720834 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:29.715159    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/premium-w2-e6c79d-kserve-6468bf46d4-h66f8" podUID="cd6a266f-8d12-4089-83ee-c75c716bf26c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:42:29.720834 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:29.715418    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x2qzv" podUID="c247ed2a-aa90-424b-9b7e-4f2936d67e3f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded"
Oct 31 12:42:29.720834 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:29.715629    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-2-w5-b25d3b-kserve-76f798d59f-vvzhg" podUID="394ed76c-84ac-4612-9acb-a7e36cb68d89" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": context deadline exceeded"
Oct 31 12:42:30.002573 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:30.001634    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-jnpqm" event={"ID":"fe33e47d-4eb8-411b-8320-21135401155b","Type":"ContainerStarted","Data":"e38284c36244ff49f621a9313984ef52ae5792dd3c02262a1f061a2cdb46a5c0"}
Oct 31 12:42:30.002573 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:30.001713    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-qqhsw" event={"ID":"d9b854bc-d670-414d-8e11-dd7bd401a59f","Type":"ContainerStarted","Data":"43a74d71decd2d2e41ae7eaeebf21ddd7236647369dae8ac32329bf45591a915"}
Oct 31 12:42:30.643911 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:42:30.636423    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7phb" containerName="extractor"
Oct 31 12:42:30.764525 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:30.754015    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/simulator-w4-c67de2-kserve-868679cf9b-gqjbx" podUID="e104d7da-1fb2-4c99-a480-b4ae84c4f273" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded"
Oct 31 12:42:30.801599 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:30.800791    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/premium-w2-e6c79d-kserve-6468bf46d4-h66f8" podUID="cd6a266f-8d12-4089-83ee-c75c716bf26c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Oct 31 12:42:30.827513 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:30.802675    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x2qzv" podUID="c247ed2a-aa90-424b-9b7e-4f2936d67e3f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded"
Oct 31 12:42:31.005130 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:30.880239    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-2-w5-b25d3b-kserve-76f798d59f-vvzhg" podUID="394ed76c-84ac-4612-9acb-a7e36cb68d89" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": context deadline exceeded"
Oct 31 12:42:31.025540 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:31.025465    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e6c79d/premium-w2-e6c79d-kserve-6468bf46d4-h66f8"
Oct 31 12:42:31.050774 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:31.050714    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:42:31.051129 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:31.051094    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:42:31.051373 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:31.051350    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:42:31.241757 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:31.234373    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-qqhsw"]
Oct 31 12:42:31.241757 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:31.234458    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-2-w5-b25d3b-kserve-76f798d59f-vvzhg"
Oct 31 12:42:31.241757 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:31.234553    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c67de2/simulator-w4-c67de2-kserve-868679cf9b-gqjbx"
Oct 31 12:42:31.241757 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:31.234574    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-pz8cf" event={"ID":"326d88b8-e041-4b4f-ab99-f1d2622d76b4","Type":"ContainerStarted","Data":"f13c80dabdb5f38aa5f229bae3faf663bfb114ed7cac8ec872711cbc3ec11537"}
Oct 31 12:42:31.245642 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:31.245618    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x2qzv"
Oct 31 12:42:31.248599 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:31.248576    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-jnpqm"]
Oct 31 12:42:31.248770 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:31.248757    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0258f09-9x2p4"]
Oct 31 12:42:31.249919 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:31.249896    5060 cpu_manager.go:414] "containerMap: removing container" podUID="622ca715-506b-4684-96c1-a59c9ed0fe77" containerName="revoke-keys"
Oct 31 12:42:31.251378 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:31.251356    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="622ca715-506b-4684-96c1-a59c9ed0fe77" containerName="revoke-keys"
Oct 31 12:42:31.251768 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:31.251751    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="622ca715-506b-4684-96c1-a59c9ed0fe77" containerName="revoke-keys"
Oct 31 12:42:31.256790 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:31.256768    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0258f09-9x2p4"]
Oct 31 12:42:31.256929 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:31.256917    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-873f5159-ldjxk"]
Oct 31 12:42:31.270300 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:31.257403    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0258f09-9x2p4"
Oct 31 12:42:31.272651 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:31.272628    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-873f5159-ldjxk"]
Oct 31 12:42:31.272764 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:31.272692    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-pz8cf"]
Oct 31 12:42:31.272827 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:31.272801    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-873f5159-ldjxk"
Oct 31 12:42:31.275231 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:31.275211    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 12:42:31.320529 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:31.275613    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 12:42:31.336030 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:31.281319    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7wsmb\""
Oct 31 12:42:31.336177 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:31.281364    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 12:42:31.344576 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:31.316102    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/930c4fa6-67c2-45f8-ad39-08235a3027ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d0258f09-9x2p4\" (UID: \"930c4fa6-67c2-45f8-ad39-08235a3027ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0258f09-9x2p4"
Oct 31 12:42:31.344898 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:31.344869    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/930c4fa6-67c2-45f8-ad39-08235a3027ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d0258f09-9x2p4\" (UID: \"930c4fa6-67c2-45f8-ad39-08235a3027ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0258f09-9x2p4"
Oct 31 12:42:31.347781 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:31.347756    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wdt8\" (UniqueName: \"kubernetes.io/projected/930c4fa6-67c2-45f8-ad39-08235a3027ef-kube-api-access-9wdt8\") pod \"maas-api-revoke-sub-d0258f09-9x2p4\" (UID: \"930c4fa6-67c2-45f8-ad39-08235a3027ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0258f09-9x2p4"
Oct 31 12:42:31.454440 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:31.453999    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/930c4fa6-67c2-45f8-ad39-08235a3027ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d0258f09-9x2p4\" (UID: \"930c4fa6-67c2-45f8-ad39-08235a3027ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0258f09-9x2p4"
Oct 31 12:42:31.454440 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:31.454041    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e695a11f-7212-46ab-b537-a40a639e7a71-maas-api-xks-ca\") pod \"maas-api-revoke-sub-873f5159-ldjxk\" (UID: \"e695a11f-7212-46ab-b537-a40a639e7a71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-873f5159-ldjxk"
Oct 31 12:42:31.454440 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:31.454262    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/930c4fa6-67c2-45f8-ad39-08235a3027ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d0258f09-9x2p4\" (UID: \"930c4fa6-67c2-45f8-ad39-08235a3027ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0258f09-9x2p4"
Oct 31 12:42:31.454440 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:31.454357    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wdt8\" (UniqueName: \"kubernetes.io/projected/930c4fa6-67c2-45f8-ad39-08235a3027ef-kube-api-access-9wdt8\") pod \"maas-api-revoke-sub-d0258f09-9x2p4\" (UID: \"930c4fa6-67c2-45f8-ad39-08235a3027ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0258f09-9x2p4"
Oct 31 12:42:31.454440 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:31.454426    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e695a11f-7212-46ab-b537-a40a639e7a71-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-873f5159-ldjxk\" (UID: \"e695a11f-7212-46ab-b537-a40a639e7a71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-873f5159-ldjxk"
Oct 31 12:42:31.455745 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:31.454839    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clqt8\" (UniqueName: \"kubernetes.io/projected/e695a11f-7212-46ab-b537-a40a639e7a71-kube-api-access-clqt8\") pod \"maas-api-revoke-sub-873f5159-ldjxk\" (UID: \"e695a11f-7212-46ab-b537-a40a639e7a71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-873f5159-ldjxk"
Oct 31 12:42:31.456239 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:31.455917    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/930c4fa6-67c2-45f8-ad39-08235a3027ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d0258f09-9x2p4\" (UID: \"930c4fa6-67c2-45f8-ad39-08235a3027ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0258f09-9x2p4"
Oct 31 12:42:31.494778 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:31.478266    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/930c4fa6-67c2-45f8-ad39-08235a3027ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d0258f09-9x2p4\" (UID: \"930c4fa6-67c2-45f8-ad39-08235a3027ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0258f09-9x2p4"
Oct 31 12:42:31.494778 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:31.483597    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wdt8\" (UniqueName: \"kubernetes.io/projected/930c4fa6-67c2-45f8-ad39-08235a3027ef-kube-api-access-9wdt8\") pod \"maas-api-revoke-sub-d0258f09-9x2p4\" (UID: \"930c4fa6-67c2-45f8-ad39-08235a3027ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0258f09-9x2p4"
Oct 31 12:42:31.520232 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:31.519747    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0258f09-9x2p4"
Oct 31 12:42:31.644612 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:31.644561    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e695a11f-7212-46ab-b537-a40a639e7a71-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-873f5159-ldjxk\" (UID: \"e695a11f-7212-46ab-b537-a40a639e7a71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-873f5159-ldjxk"
Oct 31 12:42:31.646920 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:31.646888    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clqt8\" (UniqueName: \"kubernetes.io/projected/e695a11f-7212-46ab-b537-a40a639e7a71-kube-api-access-clqt8\") pod \"maas-api-revoke-sub-873f5159-ldjxk\" (UID: \"e695a11f-7212-46ab-b537-a40a639e7a71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-873f5159-ldjxk"
Oct 31 12:42:31.647482 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:31.647449    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e695a11f-7212-46ab-b537-a40a639e7a71-maas-api-xks-ca\") pod \"maas-api-revoke-sub-873f5159-ldjxk\" (UID: \"e695a11f-7212-46ab-b537-a40a639e7a71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-873f5159-ldjxk"
Oct 31 12:42:31.650789 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:31.650045    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e695a11f-7212-46ab-b537-a40a639e7a71-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-873f5159-ldjxk\" (UID: \"e695a11f-7212-46ab-b537-a40a639e7a71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-873f5159-ldjxk"
Oct 31 12:42:31.653920 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:31.653840    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e695a11f-7212-46ab-b537-a40a639e7a71-maas-api-xks-ca\") pod \"maas-api-revoke-sub-873f5159-ldjxk\" (UID: \"e695a11f-7212-46ab-b537-a40a639e7a71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-873f5159-ldjxk"
Oct 31 12:42:31.669068 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:31.669032    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-clqt8\" (UniqueName: \"kubernetes.io/projected/e695a11f-7212-46ab-b537-a40a639e7a71-kube-api-access-clqt8\") pod \"maas-api-revoke-sub-873f5159-ldjxk\" (UID: \"e695a11f-7212-46ab-b537-a40a639e7a71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-873f5159-ldjxk"
Oct 31 12:42:31.887512 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:31.880742    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-873f5159-ldjxk"
Oct 31 12:42:32.034558 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:32.028417    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0258f09-9x2p4"]
Oct 31 12:42:32.128692 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:32.119796    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6526221-b77v7"]
Oct 31 12:42:32.213701 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:32.199697    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6526221-b77v7"]
Oct 31 12:42:32.213701 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:32.199830    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6526221-b77v7"
Oct 31 12:42:32.307319 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:32.306182    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2264162-d632-485f-89c6-286c99b30f62-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d6526221-b77v7\" (UID: \"a2264162-d632-485f-89c6-286c99b30f62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6526221-b77v7"
Oct 31 12:42:32.307319 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:32.306320    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2264162-d632-485f-89c6-286c99b30f62-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d6526221-b77v7\" (UID: \"a2264162-d632-485f-89c6-286c99b30f62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6526221-b77v7"
Oct 31 12:42:32.307319 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:32.306465    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jz7lz\" (UniqueName: \"kubernetes.io/projected/a2264162-d632-485f-89c6-286c99b30f62-kube-api-access-jz7lz\") pod \"maas-api-revoke-sub-d6526221-b77v7\" (UID: \"a2264162-d632-485f-89c6-286c99b30f62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6526221-b77v7"
Oct 31 12:42:32.414517 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:32.413642    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jz7lz\" (UniqueName: \"kubernetes.io/projected/a2264162-d632-485f-89c6-286c99b30f62-kube-api-access-jz7lz\") pod \"maas-api-revoke-sub-d6526221-b77v7\" (UID: \"a2264162-d632-485f-89c6-286c99b30f62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6526221-b77v7"
Oct 31 12:42:32.414517 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:32.413840    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2264162-d632-485f-89c6-286c99b30f62-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d6526221-b77v7\" (UID: \"a2264162-d632-485f-89c6-286c99b30f62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6526221-b77v7"
Oct 31 12:42:32.414517 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:32.413942    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2264162-d632-485f-89c6-286c99b30f62-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d6526221-b77v7\" (UID: \"a2264162-d632-485f-89c6-286c99b30f62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6526221-b77v7"
Oct 31 12:42:32.478995 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:32.419973    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2264162-d632-485f-89c6-286c99b30f62-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d6526221-b77v7\" (UID: \"a2264162-d632-485f-89c6-286c99b30f62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6526221-b77v7"
Oct 31 12:42:32.479961 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:42:32.479620    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode695a11f_7212_46ab_b537_a40a639e7a71.slice/crio-638df74890d6d07dc94a3741b65a8f29de4f2aa7e1141acfbf3b03f2ec538819 WatchSource:0}: Error finding container 638df74890d6d07dc94a3741b65a8f29de4f2aa7e1141acfbf3b03f2ec538819: Status 404 returned error can't find the container with id 638df74890d6d07dc94a3741b65a8f29de4f2aa7e1141acfbf3b03f2ec538819
Oct 31 12:42:32.592630 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:42:32.576046    5060 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod930c4fa6_67c2_45f8_ad39_08235a3027ef.slice/crio-b48ef5839b9faff8f6c6de6f61b9c9421b1605aa356e1fe3fa9982a5cfb308fb.scope\": RecentStats: unable to find data in memory cache]"
Oct 31 12:42:32.592630 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:32.591380    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-873f5159-ldjxk"]
Oct 31 12:42:32.594665 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:32.594602    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jz7lz\" (UniqueName: \"kubernetes.io/projected/a2264162-d632-485f-89c6-286c99b30f62-kube-api-access-jz7lz\") pod \"maas-api-revoke-sub-d6526221-b77v7\" (UID: \"a2264162-d632-485f-89c6-286c99b30f62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6526221-b77v7"
Oct 31 12:42:32.603741 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:32.603704    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-qqhsw" event={"ID":"d9b854bc-d670-414d-8e11-dd7bd401a59f","Type":"ContainerStarted","Data":"6cd106dfdae1207201a8ba030192c7573beb0d6a4448189ba398e915b4a179c3"}
Oct 31 12:42:32.620968 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:32.620931    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-jnpqm" event={"ID":"fe33e47d-4eb8-411b-8320-21135401155b","Type":"ContainerStarted","Data":"225fbe10c1ebd91d1bcec7e34a4d214f560578e77150fd8df8a40f988588486c"}
Oct 31 12:42:32.624516 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:32.624452    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-pz8cf" event={"ID":"326d88b8-e041-4b4f-ab99-f1d2622d76b4","Type":"ContainerStarted","Data":"adf65d533598e86aae889d3f711cf879259eb58f2663d14215861777e4ff8ad3"}
Oct 31 12:42:32.632641 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:32.632603    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0258f09-9x2p4" event={"ID":"930c4fa6-67c2-45f8-ad39-08235a3027ef","Type":"ContainerStarted","Data":"9013a225bf90e7dcc8c9f864cec8a5d77fa74e070be1842a92a84ead432d2388"}
Oct 31 12:42:32.679722 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:32.679687    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2264162-d632-485f-89c6-286c99b30f62-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d6526221-b77v7\" (UID: \"a2264162-d632-485f-89c6-286c99b30f62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6526221-b77v7"
Oct 31 12:42:32.714555 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:32.707366    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-qqhsw" podStartSLOduration=10.707334033 podStartE2EDuration="10.707334033s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:42:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:42:32.641104504 +0000 UTC m=+915.744172218" watchObservedRunningTime="2026-10-31 12:42:32.707334033 +0000 UTC m=+915.810401746"
Oct 31 12:42:32.761876 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:32.761790    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-pz8cf" podStartSLOduration=10.76176864 podStartE2EDuration="10.76176864s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:42:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:42:32.668794863 +0000 UTC m=+915.771862576" watchObservedRunningTime="2026-10-31 12:42:32.76176864 +0000 UTC m=+915.864836353"
Oct 31 12:42:32.785423 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:32.785352    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-jnpqm" podStartSLOduration=10.785329406 podStartE2EDuration="10.785329406s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:42:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:42:32.745046156 +0000 UTC m=+915.848113871" watchObservedRunningTime="2026-10-31 12:42:32.785329406 +0000 UTC m=+915.888397121"
Oct 31 12:42:32.853252 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:32.850835    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6526221-b77v7"
Oct 31 12:42:32.929039 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:32.877073    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-qqhsw"
Oct 31 12:42:32.964579 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:32.948455    5060 patch_prober.go:29] interesting pod/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-qqhsw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body=
Oct 31 12:42:32.964579 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:32.948540    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-qqhsw" podUID="d9b854bc-d670-414d-8e11-dd7bd401a59f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused"
Oct 31 12:42:33.512532 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:33.508604    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-jnpqm"
Oct 31 12:42:33.516614 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:33.516571    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-pz8cf"
Oct 31 12:42:33.845537 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:33.843149    5060 patch_prober.go:29] interesting pod/e2e-worker-w6-67d410-openshift-default-676b978549-jnpqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body=
Oct 31 12:42:33.845537 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:33.843212    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-jnpqm" podUID="fe33e47d-4eb8-411b-8320-21135401155b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused"
Oct 31 12:42:33.846460 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:33.845956    5060 patch_prober.go:29] interesting pod/e2e-worker-w6-67d410-openshift-default-676b978549-pz8cf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body=
Oct 31 12:42:33.846460 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:33.846020    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-pz8cf" podUID="326d88b8-e041-4b4f-ab99-f1d2622d76b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused"
Oct 31 12:42:33.931558 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:33.918125    5060 patch_prober.go:29] interesting pod/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-qqhsw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body=
Oct 31 12:42:33.931558 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:33.918190    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-qqhsw" podUID="d9b854bc-d670-414d-8e11-dd7bd401a59f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused"
Oct 31 12:42:34.241749 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:34.241700    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/simulator-w4-c67de2-kserve-868679cf9b-gqjbx" podUID="e104d7da-1fb2-4c99-a480-b4ae84c4f273" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:42:34.613550 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:34.607638    5060 patch_prober.go:29] interesting pod/e2e-worker-w6-67d410-openshift-default-676b978549-pz8cf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body=
Oct 31 12:42:34.613550 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:34.607700    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-pz8cf" podUID="326d88b8-e041-4b4f-ab99-f1d2622d76b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused"
Oct 31 12:42:34.613550 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:34.607785    5060 patch_prober.go:29] interesting pod/e2e-worker-w6-67d410-openshift-default-676b978549-jnpqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body=
Oct 31 12:42:34.613550 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:34.607806    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-jnpqm" podUID="fe33e47d-4eb8-411b-8320-21135401155b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused"
Oct 31 12:42:34.613550 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:34.608163    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6526221-b77v7"]
Oct 31 12:42:34.642699 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:34.642645    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/premium-w2-e6c79d-kserve-6468bf46d4-h66f8" podUID="cd6a266f-8d12-4089-83ee-c75c716bf26c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Oct 31 12:42:34.748033 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:34.747981    5060 generic.go:356] "Generic (PLEG): container finished" podID="930c4fa6-67c2-45f8-ad39-08235a3027ef" containerID="b48ef5839b9faff8f6c6de6f61b9c9421b1605aa356e1fe3fa9982a5cfb308fb" exitCode=0
Oct 31 12:42:34.749237 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:34.748259    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0258f09-9x2p4" event={"ID":"930c4fa6-67c2-45f8-ad39-08235a3027ef","Type":"ContainerDied","Data":"b48ef5839b9faff8f6c6de6f61b9c9421b1605aa356e1fe3fa9982a5cfb308fb"}
Oct 31 12:42:34.770020 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:34.769984    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-873f5159-ldjxk" event={"ID":"e695a11f-7212-46ab-b537-a40a639e7a71","Type":"ContainerStarted","Data":"638df74890d6d07dc94a3741b65a8f29de4f2aa7e1141acfbf3b03f2ec538819"}
Oct 31 12:42:34.891607 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:34.890695    5060 patch_prober.go:29] interesting pod/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-qqhsw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body=
Oct 31 12:42:34.891607 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:34.890765    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-qqhsw" podUID="d9b854bc-d670-414d-8e11-dd7bd401a59f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused"
Oct 31 12:42:34.970888 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:34.967105    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-873f5159-ldjxk" podStartSLOduration=5.9670817 podStartE2EDuration="5.9670817s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:42:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:42:34.802526928 +0000 UTC m=+917.905594641" watchObservedRunningTime="2026-10-31 12:42:34.9670817 +0000 UTC m=+918.070149413"
Oct 31 12:42:35.471873 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:35.471834    5060 patch_prober.go:29] interesting pod/e2e-worker-w6-67d410-openshift-default-676b978549-jnpqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body=
Oct 31 12:42:35.495632 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:35.495584    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-jnpqm" podUID="fe33e47d-4eb8-411b-8320-21135401155b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused"
Oct 31 12:42:35.527971 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:35.527239    5060 patch_prober.go:29] interesting pod/e2e-worker-w6-67d410-openshift-default-676b978549-pz8cf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body=
Oct 31 12:42:35.527971 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:35.527292    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-pz8cf" podUID="326d88b8-e041-4b4f-ab99-f1d2622d76b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused"
Oct 31 12:42:35.894013 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:35.893975    5060 patch_prober.go:29] interesting pod/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-qqhsw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body=
Oct 31 12:42:35.894264 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:35.894236    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-qqhsw" podUID="d9b854bc-d670-414d-8e11-dd7bd401a59f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused"
Oct 31 12:42:35.928526 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:35.928428    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6526221-b77v7" event={"ID":"a2264162-d632-485f-89c6-286c99b30f62","Type":"ContainerStarted","Data":"4f49adcc5a41287c8dc8ab5079f906b7d61a1ca0199d915da1d31e3c4424108e"}
Oct 31 12:42:35.928793 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:35.928769    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6526221-b77v7" event={"ID":"a2264162-d632-485f-89c6-286c99b30f62","Type":"ContainerStarted","Data":"e9f2d9cc4c3003bcc22a861cd996e08373b4016c4af95e60013262046131d308"}
Oct 31 12:42:36.042685 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:36.026035    5060 generic.go:356] "Generic (PLEG): container finished" podID="e695a11f-7212-46ab-b537-a40a639e7a71" containerID="28fb4ff8de3ac2703156a593eaff912df36502c48a7a3dae70c37cad22b9f927" exitCode=0
Oct 31 12:42:36.042685 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:36.026801    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-873f5159-ldjxk" event={"ID":"e695a11f-7212-46ab-b537-a40a639e7a71","Type":"ContainerDied","Data":"28fb4ff8de3ac2703156a593eaff912df36502c48a7a3dae70c37cad22b9f927"}
Oct 31 12:42:36.236377 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:36.236288    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6526221-b77v7" podStartSLOduration=4.236264932 podStartE2EDuration="4.236264932s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:42:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:42:36.009254198 +0000 UTC m=+919.112321912" watchObservedRunningTime="2026-10-31 12:42:36.236264932 +0000 UTC m=+919.339332643"
Oct 31 12:42:36.458389 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:36.457904    5060 patch_prober.go:29] interesting pod/e2e-worker-w6-67d410-openshift-default-676b978549-jnpqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body=
Oct 31 12:42:36.458389 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:36.457968    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-jnpqm" podUID="fe33e47d-4eb8-411b-8320-21135401155b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused"
Oct 31 12:42:36.511873 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:36.511812    5060 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-2-w5-b25d3b-kserve-76f798d59f-vvzhg" podUID="394ed76c-84ac-4612-9acb-a7e36cb68d89" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:42:36.516053 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:36.516005    5060 patch_prober.go:29] interesting pod/e2e-worker-w6-67d410-openshift-default-676b978549-pz8cf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body=
Oct 31 12:42:36.516663 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:36.516631    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-pz8cf" podUID="326d88b8-e041-4b4f-ab99-f1d2622d76b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused"
Oct 31 12:42:36.926702 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:36.897901    5060 patch_prober.go:29] interesting pod/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-qqhsw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body=
Oct 31 12:42:36.926995 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:36.926952    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-qqhsw" podUID="d9b854bc-d670-414d-8e11-dd7bd401a59f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused"
Oct 31 12:42:37.200231 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:37.199630    5060 generic.go:356] "Generic (PLEG): container finished" podID="a2264162-d632-485f-89c6-286c99b30f62" containerID="4f49adcc5a41287c8dc8ab5079f906b7d61a1ca0199d915da1d31e3c4424108e" exitCode=0
Oct 31 12:42:37.200231 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:37.199747    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6526221-b77v7" event={"ID":"a2264162-d632-485f-89c6-286c99b30f62","Type":"ContainerDied","Data":"4f49adcc5a41287c8dc8ab5079f906b7d61a1ca0199d915da1d31e3c4424108e"}
Oct 31 12:42:37.202991 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:37.202909    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0258f09-9x2p4" event={"ID":"930c4fa6-67c2-45f8-ad39-08235a3027ef","Type":"ContainerDied","Data":"9013a225bf90e7dcc8c9f864cec8a5d77fa74e070be1842a92a84ead432d2388"}
Oct 31 12:42:37.202991 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:37.202958    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9013a225bf90e7dcc8c9f864cec8a5d77fa74e070be1842a92a84ead432d2388"
Oct 31 12:42:37.456570 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:37.449465    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0258f09-9x2p4"
Oct 31 12:42:37.476317 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:37.476285    5060 patch_prober.go:29] interesting pod/e2e-worker-w6-67d410-openshift-default-676b978549-jnpqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body=
Oct 31 12:42:37.476564 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:37.476539    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-jnpqm" podUID="fe33e47d-4eb8-411b-8320-21135401155b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused"
Oct 31 12:42:37.516563 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:37.516528    5060 patch_prober.go:29] interesting pod/e2e-worker-w6-67d410-openshift-default-676b978549-pz8cf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body=
Oct 31 12:42:37.516815 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:37.516788    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-pz8cf" podUID="326d88b8-e041-4b4f-ab99-f1d2622d76b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused"
Oct 31 12:42:37.595525 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:37.580602    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/premium-w2-e6c79d-kserve-6468bf46d4-h66f8" podUID="cd6a266f-8d12-4089-83ee-c75c716bf26c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:42:37.681509 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:37.681347    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/930c4fa6-67c2-45f8-ad39-08235a3027ef-kube-api-access-9wdt8\" (UniqueName: \"kubernetes.io/projected/930c4fa6-67c2-45f8-ad39-08235a3027ef-kube-api-access-9wdt8\") pod \"930c4fa6-67c2-45f8-ad39-08235a3027ef\" (UID: \"930c4fa6-67c2-45f8-ad39-08235a3027ef\") "
Oct 31 12:42:37.681942 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:37.681916    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/930c4fa6-67c2-45f8-ad39-08235a3027ef-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/930c4fa6-67c2-45f8-ad39-08235a3027ef-maas-api-openshift-ca\") pod \"930c4fa6-67c2-45f8-ad39-08235a3027ef\" (UID: \"930c4fa6-67c2-45f8-ad39-08235a3027ef\") "
Oct 31 12:42:37.682173 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:37.682156    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/930c4fa6-67c2-45f8-ad39-08235a3027ef-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/930c4fa6-67c2-45f8-ad39-08235a3027ef-maas-api-xks-ca\") pod \"930c4fa6-67c2-45f8-ad39-08235a3027ef\" (UID: \"930c4fa6-67c2-45f8-ad39-08235a3027ef\") "
Oct 31 12:42:37.701437 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:37.701401    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/930c4fa6-67c2-45f8-ad39-08235a3027ef-maas-api-openshift-ca" pod "930c4fa6-67c2-45f8-ad39-08235a3027ef" (UID: "930c4fa6-67c2-45f8-ad39-08235a3027ef"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:42:37.764521 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:37.738176    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/930c4fa6-67c2-45f8-ad39-08235a3027ef-kube-api-access-9wdt8" pod "930c4fa6-67c2-45f8-ad39-08235a3027ef" (UID: "930c4fa6-67c2-45f8-ad39-08235a3027ef"). InnerVolumeSpecName "kube-api-access-9wdt8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:42:37.851513 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:37.774083    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/930c4fa6-67c2-45f8-ad39-08235a3027ef-maas-api-xks-ca" pod "930c4fa6-67c2-45f8-ad39-08235a3027ef" (UID: "930c4fa6-67c2-45f8-ad39-08235a3027ef"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:42:37.851513 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:37.791004    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/930c4fa6-67c2-45f8-ad39-08235a3027ef-maas-api-openshift-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:42:37.851513 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:37.838483    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-2-w5-b25d3b-kserve-76f798d59f-vvzhg" podUID="394ed76c-84ac-4612-9acb-a7e36cb68d89" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:42:37.986565 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:37.983036    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-qqhsw"
Oct 31 12:42:38.005511 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:38.004455    5060 patch_prober.go:29] interesting pod/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-qqhsw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body=
Oct 31 12:42:38.005511 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:38.004543    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-qqhsw" podUID="d9b854bc-d670-414d-8e11-dd7bd401a59f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused"
Oct 31 12:42:38.124689 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:38.098084    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/930c4fa6-67c2-45f8-ad39-08235a3027ef-maas-api-xks-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:42:38.124689 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:38.098120    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9wdt8\" (UniqueName: \"kubernetes.io/projected/930c4fa6-67c2-45f8-ad39-08235a3027ef-kube-api-access-9wdt8\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:42:38.155734 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:38.148567    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x2qzv" podUID="c247ed2a-aa90-424b-9b7e-4f2936d67e3f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:42:38.480567 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:38.454060    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7017-21ac45"]
Oct 31 12:42:38.480567 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:38.455259    5060 cpu_manager.go:414] "containerMap: removing container" podUID="930c4fa6-67c2-45f8-ad39-08235a3027ef" containerName="revoke-keys"
Oct 31 12:42:38.480567 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:38.455282    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="930c4fa6-67c2-45f8-ad39-08235a3027ef" containerName="revoke-keys"
Oct 31 12:42:38.480567 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:38.462464    5060 patch_prober.go:29] interesting pod/e2e-worker-w6-67d410-openshift-default-676b978549-jnpqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body=
Oct 31 12:42:38.480567 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:38.462532    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-jnpqm" podUID="fe33e47d-4eb8-411b-8320-21135401155b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused"
Oct 31 12:42:38.511967 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:38.481642    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="930c4fa6-67c2-45f8-ad39-08235a3027ef" containerName="revoke-keys"
Oct 31 12:42:38.529769 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:38.529734    5060 patch_prober.go:29] interesting pod/e2e-worker-w6-67d410-openshift-default-676b978549-pz8cf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body=
Oct 31 12:42:38.530018 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:38.529989    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-pz8cf" podUID="326d88b8-e041-4b4f-ab99-f1d2622d76b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused"
Oct 31 12:42:38.547590 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:38.541663    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7017-21ac45"
Oct 31 12:42:38.548089 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:38.548061    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-klvx2\""
Oct 31 12:42:38.671106 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:38.643829    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jktd6\" (UniqueName: \"kubernetes.io/projected/f8b8b447-7ff1-4387-bf6c-e5d61a6da00f-kube-api-access-jktd6\") pod \"test-curl-7017-21ac45\" (UID: \"f8b8b447-7ff1-4387-bf6c-e5d61a6da00f\") " pod="opendatahub/test-curl-7017-21ac45"
Oct 31 12:42:38.780956 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:38.754403    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jktd6\" (UniqueName: \"kubernetes.io/projected/f8b8b447-7ff1-4387-bf6c-e5d61a6da00f-kube-api-access-jktd6\") pod \"test-curl-7017-21ac45\" (UID: \"f8b8b447-7ff1-4387-bf6c-e5d61a6da00f\") " pod="opendatahub/test-curl-7017-21ac45"
Oct 31 12:42:38.834003 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:38.833959    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0258f09-9x2p4"
Oct 31 12:42:38.858841 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:38.854262    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jktd6\" (UniqueName: \"kubernetes.io/projected/f8b8b447-7ff1-4387-bf6c-e5d61a6da00f-kube-api-access-jktd6\") pod \"test-curl-7017-21ac45\" (UID: \"f8b8b447-7ff1-4387-bf6c-e5d61a6da00f\") " pod="opendatahub/test-curl-7017-21ac45"
Oct 31 12:42:38.858841 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:38.836947    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7017-21ac45"]
Oct 31 12:42:38.896528 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:38.892311    5060 patch_prober.go:29] interesting pod/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-qqhsw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body=
Oct 31 12:42:38.896528 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:38.892374    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-qqhsw" podUID="d9b854bc-d670-414d-8e11-dd7bd401a59f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused"
Oct 31 12:42:38.918385 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:38.918180    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7017-21ac45"
Oct 31 12:42:39.038077 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:39.037615    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-873f5159-ldjxk"
Oct 31 12:42:39.120024 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:39.116180    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e695a11f-7212-46ab-b537-a40a639e7a71-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e695a11f-7212-46ab-b537-a40a639e7a71-maas-api-openshift-ca\") pod \"e695a11f-7212-46ab-b537-a40a639e7a71\" (UID: \"e695a11f-7212-46ab-b537-a40a639e7a71\") "
Oct 31 12:42:39.120024 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:39.116315    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e695a11f-7212-46ab-b537-a40a639e7a71-kube-api-access-clqt8\" (UniqueName: \"kubernetes.io/projected/e695a11f-7212-46ab-b537-a40a639e7a71-kube-api-access-clqt8\") pod \"e695a11f-7212-46ab-b537-a40a639e7a71\" (UID: \"e695a11f-7212-46ab-b537-a40a639e7a71\") "
Oct 31 12:42:39.120024 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:39.116392    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e695a11f-7212-46ab-b537-a40a639e7a71-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e695a11f-7212-46ab-b537-a40a639e7a71-maas-api-xks-ca\") pod \"e695a11f-7212-46ab-b537-a40a639e7a71\" (UID: \"e695a11f-7212-46ab-b537-a40a639e7a71\") "
Oct 31 12:42:39.120770 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:39.120725    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e695a11f-7212-46ab-b537-a40a639e7a71-maas-api-openshift-ca" pod "e695a11f-7212-46ab-b537-a40a639e7a71" (UID: "e695a11f-7212-46ab-b537-a40a639e7a71"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:42:39.242826 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:39.242769    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e695a11f-7212-46ab-b537-a40a639e7a71-maas-api-openshift-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:42:39.249065 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:39.249027    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e695a11f-7212-46ab-b537-a40a639e7a71-kube-api-access-clqt8" pod "e695a11f-7212-46ab-b537-a40a639e7a71" (UID: "e695a11f-7212-46ab-b537-a40a639e7a71"). InnerVolumeSpecName "kube-api-access-clqt8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:42:39.328890 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:39.328846    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e695a11f-7212-46ab-b537-a40a639e7a71-maas-api-xks-ca" pod "e695a11f-7212-46ab-b537-a40a639e7a71" (UID: "e695a11f-7212-46ab-b537-a40a639e7a71"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:42:39.359518 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:39.349449    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6526221-b77v7"
Oct 31 12:42:39.359518 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:39.355183    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-clqt8\" (UniqueName: \"kubernetes.io/projected/e695a11f-7212-46ab-b537-a40a639e7a71-kube-api-access-clqt8\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:42:39.359518 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:39.355215    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e695a11f-7212-46ab-b537-a40a639e7a71-maas-api-xks-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:42:39.518601 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:39.501811    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2264162-d632-485f-89c6-286c99b30f62-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2264162-d632-485f-89c6-286c99b30f62-maas-api-openshift-ca\") pod \"a2264162-d632-485f-89c6-286c99b30f62\" (UID: \"a2264162-d632-485f-89c6-286c99b30f62\") "
Oct 31 12:42:39.518601 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:39.502092    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a2264162-d632-485f-89c6-286c99b30f62-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2264162-d632-485f-89c6-286c99b30f62-maas-api-xks-ca\") pod \"a2264162-d632-485f-89c6-286c99b30f62\" (UID: \"a2264162-d632-485f-89c6-286c99b30f62\") "
Oct 31 12:42:39.518601 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:39.502169    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2264162-d632-485f-89c6-286c99b30f62-kube-api-access-jz7lz\" (UniqueName: \"kubernetes.io/projected/a2264162-d632-485f-89c6-286c99b30f62-kube-api-access-jz7lz\") pod \"a2264162-d632-485f-89c6-286c99b30f62\" (UID: \"a2264162-d632-485f-89c6-286c99b30f62\") "
Oct 31 12:42:39.572515 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:39.552284    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2264162-d632-485f-89c6-286c99b30f62-maas-api-openshift-ca" pod "a2264162-d632-485f-89c6-286c99b30f62" (UID: "a2264162-d632-485f-89c6-286c99b30f62"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:42:39.576561 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:39.573062    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2264162-d632-485f-89c6-286c99b30f62-maas-api-openshift-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:42:39.590558 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:39.579643    5060 patch_prober.go:29] interesting pod/e2e-worker-w6-67d410-openshift-default-676b978549-pz8cf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body=
Oct 31 12:42:39.590840 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:39.590810    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-pz8cf" podUID="326d88b8-e041-4b4f-ab99-f1d2622d76b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused"
Oct 31 12:42:39.627346 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:39.580197    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-jnpqm"
Oct 31 12:42:39.627346 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:39.580441    5060 patch_prober.go:29] interesting pod/e2e-worker-w6-67d410-openshift-default-676b978549-jnpqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body=
Oct 31 12:42:39.627346 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:39.617969    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-jnpqm" podUID="fe33e47d-4eb8-411b-8320-21135401155b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused"
Oct 31 12:42:39.696901 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:39.665836    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2264162-d632-485f-89c6-286c99b30f62-kube-api-access-jz7lz" pod "a2264162-d632-485f-89c6-286c99b30f62" (UID: "a2264162-d632-485f-89c6-286c99b30f62"). InnerVolumeSpecName "kube-api-access-jz7lz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:42:39.696901 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:39.669872    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2264162-d632-485f-89c6-286c99b30f62-maas-api-xks-ca" pod "a2264162-d632-485f-89c6-286c99b30f62" (UID: "a2264162-d632-485f-89c6-286c99b30f62"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:42:39.696901 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:39.692026    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2264162-d632-485f-89c6-286c99b30f62-maas-api-xks-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:42:39.696901 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:39.692061    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jz7lz\" (UniqueName: \"kubernetes.io/projected/a2264162-d632-485f-89c6-286c99b30f62-kube-api-access-jz7lz\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:42:39.853779 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:39.853756    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7017-21ac45"]
Oct 31 12:42:39.862974 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:39.862905    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6526221-b77v7" event={"ID":"a2264162-d632-485f-89c6-286c99b30f62","Type":"ContainerDied","Data":"e9f2d9cc4c3003bcc22a861cd996e08373b4016c4af95e60013262046131d308"}
Oct 31 12:42:39.863204 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:39.863142    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e9f2d9cc4c3003bcc22a861cd996e08373b4016c4af95e60013262046131d308"
Oct 31 12:42:39.863382 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:39.863352    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6526221-b77v7"
Oct 31 12:42:39.870357 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:39.868707    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-873f5159-ldjxk" event={"ID":"e695a11f-7212-46ab-b537-a40a639e7a71","Type":"ContainerDied","Data":"638df74890d6d07dc94a3741b65a8f29de4f2aa7e1141acfbf3b03f2ec538819"}
Oct 31 12:42:39.870357 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:39.868752    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="638df74890d6d07dc94a3741b65a8f29de4f2aa7e1141acfbf3b03f2ec538819"
Oct 31 12:42:39.870357 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:39.868812    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-873f5159-ldjxk"
Oct 31 12:42:39.875732 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:39.875343    5060 patch_prober.go:29] interesting pod/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-qqhsw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body=
Oct 31 12:42:39.875732 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:39.875389    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-qqhsw" podUID="d9b854bc-d670-414d-8e11-dd7bd401a59f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused"
Oct 31 12:42:40.432825 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:40.432794    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-pz8cf"
Oct 31 12:42:40.458478 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:40.458440    5060 patch_prober.go:29] interesting pod/e2e-worker-w6-67d410-openshift-default-676b978549-jnpqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body=
Oct 31 12:42:40.458787 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:40.458742    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-jnpqm" podUID="fe33e47d-4eb8-411b-8320-21135401155b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused"
Oct 31 12:42:40.881163 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:40.881131    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7017-21ac45" event={"ID":"f8b8b447-7ff1-4387-bf6c-e5d61a6da00f","Type":"ContainerStarted","Data":"5875ff59581f6e4be2950784993964400dd4c7a8a57dedcdb12b12378907c67a"}
Oct 31 12:42:40.940943 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:40.940906    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-qqhsw"
Oct 31 12:42:40.943221 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:40.943180    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-qqhsw"
Oct 31 12:42:41.096942 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:41.096900    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-pz8cf"
Oct 31 12:42:41.100858 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:41.100835    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-pz8cf"
Oct 31 12:42:41.526482 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:41.526448    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-jnpqm"
Oct 31 12:42:41.527951 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:41.527926    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-jnpqm"
Oct 31 12:42:43.192774 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:43.192737    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f97bf1ca-swdmw"]
Oct 31 12:42:43.193874 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:43.193849    5060 cpu_manager.go:414] "containerMap: removing container" podUID="a2264162-d632-485f-89c6-286c99b30f62" containerName="revoke-keys"
Oct 31 12:42:43.193874 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:43.193873    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2264162-d632-485f-89c6-286c99b30f62" containerName="revoke-keys"
Oct 31 12:42:43.194019 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:43.193906    5060 cpu_manager.go:414] "containerMap: removing container" podUID="e695a11f-7212-46ab-b537-a40a639e7a71" containerName="revoke-keys"
Oct 31 12:42:43.194019 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:43.193914    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e695a11f-7212-46ab-b537-a40a639e7a71" containerName="revoke-keys"
Oct 31 12:42:43.194116 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:43.194104    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2264162-d632-485f-89c6-286c99b30f62" containerName="revoke-keys"
Oct 31 12:42:43.194160 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:43.194124    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="e695a11f-7212-46ab-b537-a40a639e7a71" containerName="revoke-keys"
Oct 31 12:42:43.204832 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:43.204806    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f97bf1ca-swdmw"
Oct 31 12:42:43.210089 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:43.210061    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 12:42:43.211255 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:43.211234    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 12:42:43.211351 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:43.211311    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7wsmb\""
Oct 31 12:42:43.211405 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:43.211365    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 12:42:43.212246 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:43.212215    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f97bf1ca-swdmw"]
Oct 31 12:42:43.292426 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:43.292185    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sv6vr\" (UniqueName: \"kubernetes.io/projected/c4db4fbd-149c-4d21-b64e-efcf12e2bc05-kube-api-access-sv6vr\") pod \"maas-api-revoke-sub-f97bf1ca-swdmw\" (UID: \"c4db4fbd-149c-4d21-b64e-efcf12e2bc05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f97bf1ca-swdmw"
Oct 31 12:42:43.292426 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:43.292348    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4db4fbd-149c-4d21-b64e-efcf12e2bc05-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f97bf1ca-swdmw\" (UID: \"c4db4fbd-149c-4d21-b64e-efcf12e2bc05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f97bf1ca-swdmw"
Oct 31 12:42:43.292426 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:43.292391    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4db4fbd-149c-4d21-b64e-efcf12e2bc05-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f97bf1ca-swdmw\" (UID: \"c4db4fbd-149c-4d21-b64e-efcf12e2bc05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f97bf1ca-swdmw"
Oct 31 12:42:43.408814 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:43.408774    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sv6vr\" (UniqueName: \"kubernetes.io/projected/c4db4fbd-149c-4d21-b64e-efcf12e2bc05-kube-api-access-sv6vr\") pod \"maas-api-revoke-sub-f97bf1ca-swdmw\" (UID: \"c4db4fbd-149c-4d21-b64e-efcf12e2bc05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f97bf1ca-swdmw"
Oct 31 12:42:43.409014 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:43.408877    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4db4fbd-149c-4d21-b64e-efcf12e2bc05-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f97bf1ca-swdmw\" (UID: \"c4db4fbd-149c-4d21-b64e-efcf12e2bc05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f97bf1ca-swdmw"
Oct 31 12:42:43.409014 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:43.408926    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4db4fbd-149c-4d21-b64e-efcf12e2bc05-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f97bf1ca-swdmw\" (UID: \"c4db4fbd-149c-4d21-b64e-efcf12e2bc05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f97bf1ca-swdmw"
Oct 31 12:42:43.411342 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:43.411252    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4db4fbd-149c-4d21-b64e-efcf12e2bc05-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f97bf1ca-swdmw\" (UID: \"c4db4fbd-149c-4d21-b64e-efcf12e2bc05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f97bf1ca-swdmw"
Oct 31 12:42:43.414467 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:43.414408    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4db4fbd-149c-4d21-b64e-efcf12e2bc05-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f97bf1ca-swdmw\" (UID: \"c4db4fbd-149c-4d21-b64e-efcf12e2bc05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f97bf1ca-swdmw"
Oct 31 12:42:43.432438 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:43.432360    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sv6vr\" (UniqueName: \"kubernetes.io/projected/c4db4fbd-149c-4d21-b64e-efcf12e2bc05-kube-api-access-sv6vr\") pod \"maas-api-revoke-sub-f97bf1ca-swdmw\" (UID: \"c4db4fbd-149c-4d21-b64e-efcf12e2bc05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f97bf1ca-swdmw"
Oct 31 12:42:43.531727 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:43.531687    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f97bf1ca-swdmw"
Oct 31 12:42:43.854754 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:42:43.854712    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4db4fbd_149c_4d21_b64e_efcf12e2bc05.slice/crio-96a462f12c32a9bdb62c2616440015ec6e9f90a76ceef16971c816cc9ee81b66 WatchSource:0}: Error finding container 96a462f12c32a9bdb62c2616440015ec6e9f90a76ceef16971c816cc9ee81b66: Status 404 returned error can't find the container with id 96a462f12c32a9bdb62c2616440015ec6e9f90a76ceef16971c816cc9ee81b66
Oct 31 12:42:43.858432 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:43.858399    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f97bf1ca-swdmw"]
Oct 31 12:42:43.928165 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:43.928134    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f97bf1ca-swdmw" event={"ID":"c4db4fbd-149c-4d21-b64e-efcf12e2bc05","Type":"ContainerStarted","Data":"96a462f12c32a9bdb62c2616440015ec6e9f90a76ceef16971c816cc9ee81b66"}
Oct 31 12:42:44.948756 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:44.948705    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7017-21ac45" event={"ID":"f8b8b447-7ff1-4387-bf6c-e5d61a6da00f","Type":"ContainerStarted","Data":"3dd31712197133f0a2ba09b600d8435cf511c4068b7daedc3eb24bb1a2bbec70"}
Oct 31 12:42:44.951175 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:44.951149    5060 generic.go:356] "Generic (PLEG): container finished" podID="c4db4fbd-149c-4d21-b64e-efcf12e2bc05" containerID="8f78780b79621f1fbfd5d8a445fe54669d3176a2cebcb190cbd516f66ef034e9" exitCode=0
Oct 31 12:42:44.951303 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:44.951228    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f97bf1ca-swdmw" event={"ID":"c4db4fbd-149c-4d21-b64e-efcf12e2bc05","Type":"ContainerDied","Data":"8f78780b79621f1fbfd5d8a445fe54669d3176a2cebcb190cbd516f66ef034e9"}
Oct 31 12:42:44.971340 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:44.971294    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7017-21ac45" podStartSLOduration=2.955454589 podStartE2EDuration="6.971279486s" totalImagesPullingTime="4.015824897s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:42:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:42:44.966477046 +0000 UTC m=+928.069544772" watchObservedRunningTime="2026-10-31 12:42:44.971279486 +0000 UTC m=+928.074347198"
Oct 31 12:42:45.086243 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:45.086203    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06ffe023-2fswn"]
Oct 31 12:42:45.092778 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:45.092576    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06ffe023-2fswn"
Oct 31 12:42:45.099262 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:45.099227    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06ffe023-2fswn"]
Oct 31 12:42:45.317083 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:45.316715    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da0cf5fb-f52a-4cff-98b8-75ce9fcf48b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-06ffe023-2fswn\" (UID: \"da0cf5fb-f52a-4cff-98b8-75ce9fcf48b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06ffe023-2fswn"
Oct 31 12:42:45.317083 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:45.316936    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da0cf5fb-f52a-4cff-98b8-75ce9fcf48b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-06ffe023-2fswn\" (UID: \"da0cf5fb-f52a-4cff-98b8-75ce9fcf48b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06ffe023-2fswn"
Oct 31 12:42:45.317083 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:45.317019    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtxkd\" (UniqueName: \"kubernetes.io/projected/da0cf5fb-f52a-4cff-98b8-75ce9fcf48b9-kube-api-access-jtxkd\") pod \"maas-api-revoke-sub-06ffe023-2fswn\" (UID: \"da0cf5fb-f52a-4cff-98b8-75ce9fcf48b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06ffe023-2fswn"
Oct 31 12:42:45.418076 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:45.418044    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da0cf5fb-f52a-4cff-98b8-75ce9fcf48b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-06ffe023-2fswn\" (UID: \"da0cf5fb-f52a-4cff-98b8-75ce9fcf48b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06ffe023-2fswn"
Oct 31 12:42:45.418271 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:45.418081    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtxkd\" (UniqueName: \"kubernetes.io/projected/da0cf5fb-f52a-4cff-98b8-75ce9fcf48b9-kube-api-access-jtxkd\") pod \"maas-api-revoke-sub-06ffe023-2fswn\" (UID: \"da0cf5fb-f52a-4cff-98b8-75ce9fcf48b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06ffe023-2fswn"
Oct 31 12:42:45.418347 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:45.418290    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da0cf5fb-f52a-4cff-98b8-75ce9fcf48b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-06ffe023-2fswn\" (UID: \"da0cf5fb-f52a-4cff-98b8-75ce9fcf48b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06ffe023-2fswn"
Oct 31 12:42:45.418976 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:45.418946    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da0cf5fb-f52a-4cff-98b8-75ce9fcf48b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-06ffe023-2fswn\" (UID: \"da0cf5fb-f52a-4cff-98b8-75ce9fcf48b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06ffe023-2fswn"
Oct 31 12:42:45.420936 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:45.420912    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da0cf5fb-f52a-4cff-98b8-75ce9fcf48b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-06ffe023-2fswn\" (UID: \"da0cf5fb-f52a-4cff-98b8-75ce9fcf48b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06ffe023-2fswn"
Oct 31 12:42:45.430726 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:45.430699    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtxkd\" (UniqueName: \"kubernetes.io/projected/da0cf5fb-f52a-4cff-98b8-75ce9fcf48b9-kube-api-access-jtxkd\") pod \"maas-api-revoke-sub-06ffe023-2fswn\" (UID: \"da0cf5fb-f52a-4cff-98b8-75ce9fcf48b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06ffe023-2fswn"
Oct 31 12:42:45.720044 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:45.719954    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06ffe023-2fswn"
Oct 31 12:42:45.791279 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:45.791199    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7017-21ac45"]
Oct 31 12:42:45.801029 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:45.800753    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7017-21ac45"]
Oct 31 12:42:45.828548 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:45.820629    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-341d3105-c772h"]
Oct 31 12:42:45.828548 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:45.821947    5060 cpu_manager.go:414] "containerMap: removing container" podUID="f8b8b447-7ff1-4387-bf6c-e5d61a6da00f" containerName="test-curl-7017-21ac45"
Oct 31 12:42:45.828548 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:45.821967    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8b8b447-7ff1-4387-bf6c-e5d61a6da00f" containerName="test-curl-7017-21ac45"
Oct 31 12:42:45.828548 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:45.822206    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8b8b447-7ff1-4387-bf6c-e5d61a6da00f" containerName="test-curl-7017-21ac45"
Oct 31 12:42:45.836445 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:45.836422    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-341d3105-c772h"
Oct 31 12:42:45.839132 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:45.839104    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-341d3105-c772h"]
Oct 31 12:42:45.927565 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:45.927527    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d243b8a-d515-4b0e-ae52-d74de6e56550-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-341d3105-c772h\" (UID: \"7d243b8a-d515-4b0e-ae52-d74de6e56550\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-341d3105-c772h"
Oct 31 12:42:45.927775 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:45.927696    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d243b8a-d515-4b0e-ae52-d74de6e56550-maas-api-xks-ca\") pod \"maas-api-revoke-sub-341d3105-c772h\" (UID: \"7d243b8a-d515-4b0e-ae52-d74de6e56550\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-341d3105-c772h"
Oct 31 12:42:45.927874 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:45.927844    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqmpk\" (UniqueName: \"kubernetes.io/projected/7d243b8a-d515-4b0e-ae52-d74de6e56550-kube-api-access-bqmpk\") pod \"maas-api-revoke-sub-341d3105-c772h\" (UID: \"7d243b8a-d515-4b0e-ae52-d74de6e56550\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-341d3105-c772h"
Oct 31 12:42:45.940842 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:45.940814    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7017-30a720"]
Oct 31 12:42:45.946825 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:45.946803    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7017-30a720"
Oct 31 12:42:45.953750 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:45.953712    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7017-30a720"]
Oct 31 12:42:46.038139 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:46.037553    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06ffe023-2fswn"]
Oct 31 12:42:46.038561 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:46.038523    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d243b8a-d515-4b0e-ae52-d74de6e56550-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-341d3105-c772h\" (UID: \"7d243b8a-d515-4b0e-ae52-d74de6e56550\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-341d3105-c772h"
Oct 31 12:42:46.038700 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:46.038631    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vplk9\" (UniqueName: \"kubernetes.io/projected/731c6831-ebf8-4055-a5bc-875a4af865ce-kube-api-access-vplk9\") pod \"test-curl-7017-30a720\" (UID: \"731c6831-ebf8-4055-a5bc-875a4af865ce\") " pod="opendatahub/test-curl-7017-30a720"
Oct 31 12:42:46.038700 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:46.038683    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d243b8a-d515-4b0e-ae52-d74de6e56550-maas-api-xks-ca\") pod \"maas-api-revoke-sub-341d3105-c772h\" (UID: \"7d243b8a-d515-4b0e-ae52-d74de6e56550\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-341d3105-c772h"
Oct 31 12:42:46.039173 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:46.039130    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqmpk\" (UniqueName: \"kubernetes.io/projected/7d243b8a-d515-4b0e-ae52-d74de6e56550-kube-api-access-bqmpk\") pod \"maas-api-revoke-sub-341d3105-c772h\" (UID: \"7d243b8a-d515-4b0e-ae52-d74de6e56550\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-341d3105-c772h"
Oct 31 12:42:46.039730 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:46.039576    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d243b8a-d515-4b0e-ae52-d74de6e56550-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-341d3105-c772h\" (UID: \"7d243b8a-d515-4b0e-ae52-d74de6e56550\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-341d3105-c772h"
Oct 31 12:42:46.042940 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:46.042913    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d243b8a-d515-4b0e-ae52-d74de6e56550-maas-api-xks-ca\") pod \"maas-api-revoke-sub-341d3105-c772h\" (UID: \"7d243b8a-d515-4b0e-ae52-d74de6e56550\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-341d3105-c772h"
Oct 31 12:42:46.152231 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:46.152189    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vplk9\" (UniqueName: \"kubernetes.io/projected/731c6831-ebf8-4055-a5bc-875a4af865ce-kube-api-access-vplk9\") pod \"test-curl-7017-30a720\" (UID: \"731c6831-ebf8-4055-a5bc-875a4af865ce\") " pod="opendatahub/test-curl-7017-30a720"
Oct 31 12:42:46.187563 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:46.183660    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqmpk\" (UniqueName: \"kubernetes.io/projected/7d243b8a-d515-4b0e-ae52-d74de6e56550-kube-api-access-bqmpk\") pod \"maas-api-revoke-sub-341d3105-c772h\" (UID: \"7d243b8a-d515-4b0e-ae52-d74de6e56550\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-341d3105-c772h"
Oct 31 12:42:46.196175 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:46.196130    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vplk9\" (UniqueName: \"kubernetes.io/projected/731c6831-ebf8-4055-a5bc-875a4af865ce-kube-api-access-vplk9\") pod \"test-curl-7017-30a720\" (UID: \"731c6831-ebf8-4055-a5bc-875a4af865ce\") " pod="opendatahub/test-curl-7017-30a720"
Oct 31 12:42:46.241736 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:46.241703    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f97bf1ca-swdmw"
Oct 31 12:42:46.282257 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:46.282228    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7017-30a720"
Oct 31 12:42:46.357310 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:46.356322    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4db4fbd-149c-4d21-b64e-efcf12e2bc05-kube-api-access-sv6vr\" (UniqueName: \"kubernetes.io/projected/c4db4fbd-149c-4d21-b64e-efcf12e2bc05-kube-api-access-sv6vr\") pod \"c4db4fbd-149c-4d21-b64e-efcf12e2bc05\" (UID: \"c4db4fbd-149c-4d21-b64e-efcf12e2bc05\") "
Oct 31 12:42:46.357310 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:46.356522    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c4db4fbd-149c-4d21-b64e-efcf12e2bc05-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4db4fbd-149c-4d21-b64e-efcf12e2bc05-maas-api-xks-ca\") pod \"c4db4fbd-149c-4d21-b64e-efcf12e2bc05\" (UID: \"c4db4fbd-149c-4d21-b64e-efcf12e2bc05\") "
Oct 31 12:42:46.357310 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:46.356581    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4db4fbd-149c-4d21-b64e-efcf12e2bc05-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4db4fbd-149c-4d21-b64e-efcf12e2bc05-maas-api-openshift-ca\") pod \"c4db4fbd-149c-4d21-b64e-efcf12e2bc05\" (UID: \"c4db4fbd-149c-4d21-b64e-efcf12e2bc05\") "
Oct 31 12:42:46.358009 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:46.357361    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4db4fbd-149c-4d21-b64e-efcf12e2bc05-maas-api-openshift-ca" pod "c4db4fbd-149c-4d21-b64e-efcf12e2bc05" (UID: "c4db4fbd-149c-4d21-b64e-efcf12e2bc05"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:42:46.360194 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:46.360170    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4db4fbd-149c-4d21-b64e-efcf12e2bc05-kube-api-access-sv6vr" pod "c4db4fbd-149c-4d21-b64e-efcf12e2bc05" (UID: "c4db4fbd-149c-4d21-b64e-efcf12e2bc05"). InnerVolumeSpecName "kube-api-access-sv6vr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:42:46.363807 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:46.363777    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4db4fbd-149c-4d21-b64e-efcf12e2bc05-maas-api-xks-ca" pod "c4db4fbd-149c-4d21-b64e-efcf12e2bc05" (UID: "c4db4fbd-149c-4d21-b64e-efcf12e2bc05"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:42:46.457781 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:46.457745    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-341d3105-c772h"
Oct 31 12:42:46.458812 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:46.458781    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sv6vr\" (UniqueName: \"kubernetes.io/projected/c4db4fbd-149c-4d21-b64e-efcf12e2bc05-kube-api-access-sv6vr\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:42:46.458908 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:46.458819    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4db4fbd-149c-4d21-b64e-efcf12e2bc05-maas-api-xks-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:42:46.458908 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:46.458834    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4db4fbd-149c-4d21-b64e-efcf12e2bc05-maas-api-openshift-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:42:46.612438 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:42:46.612398    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod731c6831_ebf8_4055_a5bc_875a4af865ce.slice/crio-7e692e76c4de949ec353df7e426467ba6aac740787cfd9209f23f881c0e338a0 WatchSource:0}: Error finding container 7e692e76c4de949ec353df7e426467ba6aac740787cfd9209f23f881c0e338a0: Status 404 returned error can't find the container with id 7e692e76c4de949ec353df7e426467ba6aac740787cfd9209f23f881c0e338a0
Oct 31 12:42:46.615912 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:46.615869    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7017-30a720"]
Oct 31 12:42:46.772466 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:46.772382    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-341d3105-c772h"]
Oct 31 12:42:46.781134 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:42:46.781099    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d243b8a_d515_4b0e_ae52_d74de6e56550.slice/crio-e0dad3cd46299ccd75d696b547ba3ea081eb9b64bcdd36c73b1266e51ae77310 WatchSource:0}: Error finding container e0dad3cd46299ccd75d696b547ba3ea081eb9b64bcdd36c73b1266e51ae77310: Status 404 returned error can't find the container with id e0dad3cd46299ccd75d696b547ba3ea081eb9b64bcdd36c73b1266e51ae77310
Oct 31 12:42:47.219185 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:47.219145    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7017-30a720" event={"ID":"731c6831-ebf8-4055-a5bc-875a4af865ce","Type":"ContainerStarted","Data":"40a9c5b5b642065c5f382c5d1152c0e1c055d47d3d9159f91dbc09a547f8d1e9"}
Oct 31 12:42:47.219185 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:47.219197    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7017-30a720" event={"ID":"731c6831-ebf8-4055-a5bc-875a4af865ce","Type":"ContainerStarted","Data":"7e692e76c4de949ec353df7e426467ba6aac740787cfd9209f23f881c0e338a0"}
Oct 31 12:42:47.247995 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:47.242939    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7017-30a720" podStartSLOduration=2.242916281 podStartE2EDuration="2.242916281s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:42:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:42:47.207071131 +0000 UTC m=+930.310138845" watchObservedRunningTime="2026-10-31 12:42:47.242916281 +0000 UTC m=+930.345983994"
Oct 31 12:42:47.267521 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:47.266349    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06ffe023-2fswn" event={"ID":"da0cf5fb-f52a-4cff-98b8-75ce9fcf48b9","Type":"ContainerStarted","Data":"95b47a2c3acfe251a1d9c24318203c56441584e4f340befe96c8d9550a184157"}
Oct 31 12:42:47.267521 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:47.266420    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06ffe023-2fswn" event={"ID":"da0cf5fb-f52a-4cff-98b8-75ce9fcf48b9","Type":"ContainerStarted","Data":"7a88f64fb5efd1e776e251c8cbda3e10698a37d493d6af9debe73ca9516c276e"}
Oct 31 12:42:47.293773 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:47.288435    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f97bf1ca-swdmw" event={"ID":"c4db4fbd-149c-4d21-b64e-efcf12e2bc05","Type":"ContainerDied","Data":"96a462f12c32a9bdb62c2616440015ec6e9f90a76ceef16971c816cc9ee81b66"}
Oct 31 12:42:47.304002 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:47.301805    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f97bf1ca-swdmw"
Oct 31 12:42:47.309536 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:47.308863    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="96a462f12c32a9bdb62c2616440015ec6e9f90a76ceef16971c816cc9ee81b66"
Oct 31 12:42:47.340240 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:47.339884    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7017-21ac45" podUID="f8b8b447-7ff1-4387-bf6c-e5d61a6da00f" containerName="test-curl-7017-21ac45" containerID="cri-o://3dd31712197133f0a2ba09b600d8435cf511c4068b7daedc3eb24bb1a2bbec70" gracePeriod=2
Oct 31 12:42:47.340240 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:47.340026    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-341d3105-c772h" event={"ID":"7d243b8a-d515-4b0e-ae52-d74de6e56550","Type":"ContainerStarted","Data":"e0dad3cd46299ccd75d696b547ba3ea081eb9b64bcdd36c73b1266e51ae77310"}
Oct 31 12:42:47.348863 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:47.348799    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06ffe023-2fswn" podStartSLOduration=2.348779059 podStartE2EDuration="2.348779059s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:42:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:42:47.302928351 +0000 UTC m=+930.405996064" watchObservedRunningTime="2026-10-31 12:42:47.348779059 +0000 UTC m=+930.451846772"
Oct 31 12:42:48.406821 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:48.406780    5060 generic.go:356] "Generic (PLEG): container finished" podID="da0cf5fb-f52a-4cff-98b8-75ce9fcf48b9" containerID="95b47a2c3acfe251a1d9c24318203c56441584e4f340befe96c8d9550a184157" exitCode=0
Oct 31 12:42:48.412321 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:48.408547    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06ffe023-2fswn" event={"ID":"da0cf5fb-f52a-4cff-98b8-75ce9fcf48b9","Type":"ContainerDied","Data":"95b47a2c3acfe251a1d9c24318203c56441584e4f340befe96c8d9550a184157"}
Oct 31 12:42:48.415506 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:48.415429    5060 generic.go:356] "Generic (PLEG): container finished" podID="7d243b8a-d515-4b0e-ae52-d74de6e56550" containerID="aac66dda09e0dcdfc5516de8930a983fc1ccbf16cf7d2218e2bd210bc99a204d" exitCode=0
Oct 31 12:42:48.422474 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:48.417674    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-341d3105-c772h" event={"ID":"7d243b8a-d515-4b0e-ae52-d74de6e56550","Type":"ContainerDied","Data":"aac66dda09e0dcdfc5516de8930a983fc1ccbf16cf7d2218e2bd210bc99a204d"}
Oct 31 12:42:49.342816 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:49.338466    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7017-30a720"]
Oct 31 12:42:49.529168 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:49.529119    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7017-30a720" podUID="731c6831-ebf8-4055-a5bc-875a4af865ce" containerName="test-curl-7017-30a720" containerID="cri-o://40a9c5b5b642065c5f382c5d1152c0e1c055d47d3d9159f91dbc09a547f8d1e9" gracePeriod=2
Oct 31 12:42:49.542621 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:49.531465    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7017-30a720"]
Oct 31 12:42:49.542621 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:42:49.531685    5060 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7017-30a720\" is forbidden: User \"system:node:ip-10-0-130-190.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-190.us-east-2.compute.internal' and this object" podUID="731c6831-ebf8-4055-a5bc-875a4af865ce" pod="opendatahub/test-curl-7017-30a720"
Oct 31 12:42:49.872061 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:49.871950    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-2-w5-b25d3b-kserve-76f798d59f-vvzhg" podUID="394ed76c-84ac-4612-9acb-a7e36cb68d89" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:42:50.108538 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:50.097575    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x2qzv" podUID="c247ed2a-aa90-424b-9b7e-4f2936d67e3f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded"
Oct 31 12:42:50.167775 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:50.167681    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-d5e8c3f2-openshift-default-5dc86f657c-j4wrx"]
Oct 31 12:42:50.169464 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:50.169438    5060 cpu_manager.go:414] "containerMap: removing container" podUID="731c6831-ebf8-4055-a5bc-875a4af865ce" containerName="test-curl-7017-30a720"
Oct 31 12:42:50.169663 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:50.169648    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="731c6831-ebf8-4055-a5bc-875a4af865ce" containerName="test-curl-7017-30a720"
Oct 31 12:42:50.169841 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:50.169813    5060 cpu_manager.go:414] "containerMap: removing container" podUID="c4db4fbd-149c-4d21-b64e-efcf12e2bc05" containerName="revoke-keys"
Oct 31 12:42:50.169958 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:50.169945    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4db4fbd-149c-4d21-b64e-efcf12e2bc05" containerName="revoke-keys"
Oct 31 12:42:50.170290 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:50.170277    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="731c6831-ebf8-4055-a5bc-875a4af865ce" containerName="test-curl-7017-30a720"
Oct 31 12:42:50.170420 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:50.170409    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4db4fbd-149c-4d21-b64e-efcf12e2bc05" containerName="revoke-keys"
Oct 31 12:42:50.253514 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:50.232836    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-d5e8c3f2-openshift-default-5dc86f657c-j4wrx"
Oct 31 12:42:50.276520 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:50.261763    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-d5e8c3f2-openshift-default-5dc86f657c-j4wrx"]
Oct 31 12:42:50.276520 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:50.265805    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/simulator-w4-c67de2-kserve-868679cf9b-gqjbx" podUID="e104d7da-1fb2-4c99-a480-b4ae84c4f273" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:42:50.367511 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:50.365664    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-d5e8c3f2-openshift-default-dockercfg-nxf5l\""
Oct 31 12:42:50.367511 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:50.366604    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f135ca99-10b3-4413-8d5f-1541b2aae5cf-workload-certs\") pod \"e2e-derive-d5e8c3f2-openshift-default-5dc86f657c-j4wrx\" (UID: \"f135ca99-10b3-4413-8d5f-1541b2aae5cf\") " pod="openshift-ingress/e2e-derive-d5e8c3f2-openshift-default-5dc86f657c-j4wrx"
Oct 31 12:42:50.367511 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:50.366661    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f135ca99-10b3-4413-8d5f-1541b2aae5cf-workload-socket\") pod \"e2e-derive-d5e8c3f2-openshift-default-5dc86f657c-j4wrx\" (UID: \"f135ca99-10b3-4413-8d5f-1541b2aae5cf\") " pod="openshift-ingress/e2e-derive-d5e8c3f2-openshift-default-5dc86f657c-j4wrx"
Oct 31 12:42:50.367511 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:50.366714    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f135ca99-10b3-4413-8d5f-1541b2aae5cf-istiod-ca-cert\") pod \"e2e-derive-d5e8c3f2-openshift-default-5dc86f657c-j4wrx\" (UID: \"f135ca99-10b3-4413-8d5f-1541b2aae5cf\") " pod="openshift-ingress/e2e-derive-d5e8c3f2-openshift-default-5dc86f657c-j4wrx"
Oct 31 12:42:50.367511 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:50.366774    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5j6b\" (UniqueName: \"kubernetes.io/projected/f135ca99-10b3-4413-8d5f-1541b2aae5cf-kube-api-access-v5j6b\") pod \"e2e-derive-d5e8c3f2-openshift-default-5dc86f657c-j4wrx\" (UID: \"f135ca99-10b3-4413-8d5f-1541b2aae5cf\") " pod="openshift-ingress/e2e-derive-d5e8c3f2-openshift-default-5dc86f657c-j4wrx"
Oct 31 12:42:50.367511 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:50.366867    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f135ca99-10b3-4413-8d5f-1541b2aae5cf-istio-data\") pod \"e2e-derive-d5e8c3f2-openshift-default-5dc86f657c-j4wrx\" (UID: \"f135ca99-10b3-4413-8d5f-1541b2aae5cf\") " pod="openshift-ingress/e2e-derive-d5e8c3f2-openshift-default-5dc86f657c-j4wrx"
Oct 31 12:42:50.367511 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:50.366906    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f135ca99-10b3-4413-8d5f-1541b2aae5cf-istio-envoy\") pod \"e2e-derive-d5e8c3f2-openshift-default-5dc86f657c-j4wrx\" (UID: \"f135ca99-10b3-4413-8d5f-1541b2aae5cf\") " pod="openshift-ingress/e2e-derive-d5e8c3f2-openshift-default-5dc86f657c-j4wrx"
Oct 31 12:42:50.367511 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:50.366951    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f135ca99-10b3-4413-8d5f-1541b2aae5cf-istio-podinfo\") pod \"e2e-derive-d5e8c3f2-openshift-default-5dc86f657c-j4wrx\" (UID: \"f135ca99-10b3-4413-8d5f-1541b2aae5cf\") " pod="openshift-ingress/e2e-derive-d5e8c3f2-openshift-default-5dc86f657c-j4wrx"
Oct 31 12:42:50.367511 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:50.366999    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f135ca99-10b3-4413-8d5f-1541b2aae5cf-istio-token\") pod \"e2e-derive-d5e8c3f2-openshift-default-5dc86f657c-j4wrx\" (UID: \"f135ca99-10b3-4413-8d5f-1541b2aae5cf\") " pod="openshift-ingress/e2e-derive-d5e8c3f2-openshift-default-5dc86f657c-j4wrx"
Oct 31 12:42:50.367511 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:50.367122    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f135ca99-10b3-4413-8d5f-1541b2aae5cf-credential-socket\") pod \"e2e-derive-d5e8c3f2-openshift-default-5dc86f657c-j4wrx\" (UID: \"f135ca99-10b3-4413-8d5f-1541b2aae5cf\") " pod="openshift-ingress/e2e-derive-d5e8c3f2-openshift-default-5dc86f657c-j4wrx"
Oct 31 12:42:50.507568 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:42:50.412468    5060 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7017-30a720\" is forbidden: User \"system:node:ip-10-0-130-190.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-190.us-east-2.compute.internal' and this object" podUID="731c6831-ebf8-4055-a5bc-875a4af865ce" pod="opendatahub/test-curl-7017-30a720"
Oct 31 12:42:50.579887 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:50.511080    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f135ca99-10b3-4413-8d5f-1541b2aae5cf-istio-podinfo\") pod \"e2e-derive-d5e8c3f2-openshift-default-5dc86f657c-j4wrx\" (UID: \"f135ca99-10b3-4413-8d5f-1541b2aae5cf\") " pod="openshift-ingress/e2e-derive-d5e8c3f2-openshift-default-5dc86f657c-j4wrx"
Oct 31 12:42:50.675040 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:50.619602    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f135ca99-10b3-4413-8d5f-1541b2aae5cf-istio-token\") pod \"e2e-derive-d5e8c3f2-openshift-default-5dc86f657c-j4wrx\" (UID: \"f135ca99-10b3-4413-8d5f-1541b2aae5cf\") " pod="openshift-ingress/e2e-derive-d5e8c3f2-openshift-default-5dc86f657c-j4wrx"
Oct 31 12:42:50.675040 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:50.619789    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f135ca99-10b3-4413-8d5f-1541b2aae5cf-credential-socket\") pod \"e2e-derive-d5e8c3f2-openshift-default-5dc86f657c-j4wrx\" (UID: \"f135ca99-10b3-4413-8d5f-1541b2aae5cf\") " pod="openshift-ingress/e2e-derive-d5e8c3f2-openshift-default-5dc86f657c-j4wrx"
Oct 31 12:42:50.675040 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:50.619858    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f135ca99-10b3-4413-8d5f-1541b2aae5cf-workload-certs\") pod \"e2e-derive-d5e8c3f2-openshift-default-5dc86f657c-j4wrx\" (UID: \"f135ca99-10b3-4413-8d5f-1541b2aae5cf\") " pod="openshift-ingress/e2e-derive-d5e8c3f2-openshift-default-5dc86f657c-j4wrx"
Oct 31 12:42:50.675040 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:50.619918    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f135ca99-10b3-4413-8d5f-1541b2aae5cf-workload-socket\") pod \"e2e-derive-d5e8c3f2-openshift-default-5dc86f657c-j4wrx\" (UID: \"f135ca99-10b3-4413-8d5f-1541b2aae5cf\") " pod="openshift-ingress/e2e-derive-d5e8c3f2-openshift-default-5dc86f657c-j4wrx"
Oct 31 12:42:50.675040 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:50.619985    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f135ca99-10b3-4413-8d5f-1541b2aae5cf-istiod-ca-cert\") pod \"e2e-derive-d5e8c3f2-openshift-default-5dc86f657c-j4wrx\" (UID: \"f135ca99-10b3-4413-8d5f-1541b2aae5cf\") " pod="openshift-ingress/e2e-derive-d5e8c3f2-openshift-default-5dc86f657c-j4wrx"
Oct 31 12:42:50.675040 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:50.620062    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5j6b\" (UniqueName: \"kubernetes.io/projected/f135ca99-10b3-4413-8d5f-1541b2aae5cf-kube-api-access-v5j6b\") pod \"e2e-derive-d5e8c3f2-openshift-default-5dc86f657c-j4wrx\" (UID: \"f135ca99-10b3-4413-8d5f-1541b2aae5cf\") " pod="openshift-ingress/e2e-derive-d5e8c3f2-openshift-default-5dc86f657c-j4wrx"
Oct 31 12:42:50.675040 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:50.620203    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f135ca99-10b3-4413-8d5f-1541b2aae5cf-istio-data\") pod \"e2e-derive-d5e8c3f2-openshift-default-5dc86f657c-j4wrx\" (UID: \"f135ca99-10b3-4413-8d5f-1541b2aae5cf\") " pod="openshift-ingress/e2e-derive-d5e8c3f2-openshift-default-5dc86f657c-j4wrx"
Oct 31 12:42:50.675040 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:50.620250    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f135ca99-10b3-4413-8d5f-1541b2aae5cf-istio-envoy\") pod \"e2e-derive-d5e8c3f2-openshift-default-5dc86f657c-j4wrx\" (UID: \"f135ca99-10b3-4413-8d5f-1541b2aae5cf\") " pod="openshift-ingress/e2e-derive-d5e8c3f2-openshift-default-5dc86f657c-j4wrx"
Oct 31 12:42:50.675040 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:50.621975    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f135ca99-10b3-4413-8d5f-1541b2aae5cf-workload-certs\") pod \"e2e-derive-d5e8c3f2-openshift-default-5dc86f657c-j4wrx\" (UID: \"f135ca99-10b3-4413-8d5f-1541b2aae5cf\") " pod="openshift-ingress/e2e-derive-d5e8c3f2-openshift-default-5dc86f657c-j4wrx"
Oct 31 12:42:50.675040 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:50.591755    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/premium-w2-e6c79d-kserve-6468bf46d4-h66f8" podUID="cd6a266f-8d12-4089-83ee-c75c716bf26c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:42:50.675040 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:50.622919    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f135ca99-10b3-4413-8d5f-1541b2aae5cf-credential-socket\") pod \"e2e-derive-d5e8c3f2-openshift-default-5dc86f657c-j4wrx\" (UID: \"f135ca99-10b3-4413-8d5f-1541b2aae5cf\") " pod="openshift-ingress/e2e-derive-d5e8c3f2-openshift-default-5dc86f657c-j4wrx"
Oct 31 12:42:50.675040 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:50.623692    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f135ca99-10b3-4413-8d5f-1541b2aae5cf-istiod-ca-cert\") pod \"e2e-derive-d5e8c3f2-openshift-default-5dc86f657c-j4wrx\" (UID: \"f135ca99-10b3-4413-8d5f-1541b2aae5cf\") " pod="openshift-ingress/e2e-derive-d5e8c3f2-openshift-default-5dc86f657c-j4wrx"
Oct 31 12:42:50.675040 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:50.623978    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f135ca99-10b3-4413-8d5f-1541b2aae5cf-workload-socket\") pod \"e2e-derive-d5e8c3f2-openshift-default-5dc86f657c-j4wrx\" (UID: \"f135ca99-10b3-4413-8d5f-1541b2aae5cf\") " pod="openshift-ingress/e2e-derive-d5e8c3f2-openshift-default-5dc86f657c-j4wrx"
Oct 31 12:42:50.676208 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:50.675825    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f135ca99-10b3-4413-8d5f-1541b2aae5cf-istio-envoy\") pod \"e2e-derive-d5e8c3f2-openshift-default-5dc86f657c-j4wrx\" (UID: \"f135ca99-10b3-4413-8d5f-1541b2aae5cf\") " pod="openshift-ingress/e2e-derive-d5e8c3f2-openshift-default-5dc86f657c-j4wrx"
Oct 31 12:42:50.789068 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:50.687875    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f135ca99-10b3-4413-8d5f-1541b2aae5cf-istio-data\") pod \"e2e-derive-d5e8c3f2-openshift-default-5dc86f657c-j4wrx\" (UID: \"f135ca99-10b3-4413-8d5f-1541b2aae5cf\") " pod="openshift-ingress/e2e-derive-d5e8c3f2-openshift-default-5dc86f657c-j4wrx"
Oct 31 12:42:50.932524 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:50.919299    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5j6b\" (UniqueName: \"kubernetes.io/projected/f135ca99-10b3-4413-8d5f-1541b2aae5cf-kube-api-access-v5j6b\") pod \"e2e-derive-d5e8c3f2-openshift-default-5dc86f657c-j4wrx\" (UID: \"f135ca99-10b3-4413-8d5f-1541b2aae5cf\") " pod="openshift-ingress/e2e-derive-d5e8c3f2-openshift-default-5dc86f657c-j4wrx"
Oct 31 12:42:51.038606 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:50.933528    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f135ca99-10b3-4413-8d5f-1541b2aae5cf-istio-podinfo\") pod \"e2e-derive-d5e8c3f2-openshift-default-5dc86f657c-j4wrx\" (UID: \"f135ca99-10b3-4413-8d5f-1541b2aae5cf\") " pod="openshift-ingress/e2e-derive-d5e8c3f2-openshift-default-5dc86f657c-j4wrx"
Oct 31 12:42:51.240582 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:51.186299    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f135ca99-10b3-4413-8d5f-1541b2aae5cf-istio-token\") pod \"e2e-derive-d5e8c3f2-openshift-default-5dc86f657c-j4wrx\" (UID: \"f135ca99-10b3-4413-8d5f-1541b2aae5cf\") " pod="openshift-ingress/e2e-derive-d5e8c3f2-openshift-default-5dc86f657c-j4wrx"
Oct 31 12:42:51.727570 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:51.006749    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-2-w5-b25d3b-kserve-76f798d59f-vvzhg" podUID="394ed76c-84ac-4612-9acb-a7e36cb68d89" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:42:51.727570 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:51.564852    5060 generic.go:356] "Generic (PLEG): container finished" podID="f8b8b447-7ff1-4387-bf6c-e5d61a6da00f" containerID="3dd31712197133f0a2ba09b600d8435cf511c4068b7daedc3eb24bb1a2bbec70" exitCode=137
Oct 31 12:42:51.727570 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:51.565183    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/simulator-w4-c67de2-kserve-868679cf9b-gqjbx" podUID="e104d7da-1fb2-4c99-a480-b4ae84c4f273" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded"
Oct 31 12:42:51.799519 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:51.763379    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/premium-w2-e6c79d-kserve-6468bf46d4-h66f8" podUID="cd6a266f-8d12-4089-83ee-c75c716bf26c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Oct 31 12:42:52.004666 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:51.802674    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x2qzv" podUID="c247ed2a-aa90-424b-9b7e-4f2936d67e3f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:42:52.552646 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:52.552595    5060 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x2qzv" podUID="c247ed2a-aa90-424b-9b7e-4f2936d67e3f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": context deadline exceeded"
Oct 31 12:42:52.594955 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:52.591667    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/simulator-w4-c67de2-kserve-868679cf9b-gqjbx" podUID="e104d7da-1fb2-4c99-a480-b4ae84c4f273" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded"
Oct 31 12:42:53.348611 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:52.591739    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-2-w5-b25d3b-kserve-76f798d59f-vvzhg" podUID="394ed76c-84ac-4612-9acb-a7e36cb68d89" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": context deadline exceeded"
Oct 31 12:42:53.348611 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:52.961054    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/premium-w2-e6c79d-kserve-6468bf46d4-h66f8" podUID="cd6a266f-8d12-4089-83ee-c75c716bf26c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:42:53.348611 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:53.076870    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x2qzv" podUID="c247ed2a-aa90-424b-9b7e-4f2936d67e3f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:42:53.348611 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:53.151745    5060 patch_prober.go:29] interesting pod/e2e-worker-w3-b218d3-openshift-default-67596dd89f-2fn6t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 12:42:53.493877 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:42:53.410716    5060 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="2.367s"
Oct 31 12:42:53.523516 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:53.523014    5060 patch_prober.go:29] interesting pod/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wnh97 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 12:42:53.523516 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:53.523096    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wnh97" podUID="330523c8-06c9-42a2-ac3f-9dd4ad62c392" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:42:53.927398 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:53.927362    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-d5e8c3f2-openshift-default-5dc86f657c-j4wrx"
Oct 31 12:42:54.053586 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:53.927677    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-2fn6t" podUID="60d395e9-e4e9-4c68-ab10-9ef5371df1b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:42:54.266531 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:54.238459    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-2-w5-b25d3b-kserve-76f798d59f-vvzhg"
Oct 31 12:42:54.266531 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:54.238652    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-b849a4c4-openshift-default-76d4f9446c-blrcp"]
Oct 31 12:42:54.569991 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:54.569562    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7017-21ac45"
Oct 31 12:42:54.570752 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:54.570625    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/premium-w2-e6c79d-kserve-6468bf46d4-h66f8" podUID="cd6a266f-8d12-4089-83ee-c75c716bf26c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:42:54.574119 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:54.571000    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-2-w5-b25d3b-kserve-76f798d59f-vvzhg" podUID="394ed76c-84ac-4612-9acb-a7e36cb68d89" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": context deadline exceeded"
Oct 31 12:42:54.587970 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:54.572019    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-341d3105-c772h"
Oct 31 12:42:54.867517 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:54.822710    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06ffe023-2fswn"
Oct 31 12:42:54.867517 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:54.863985    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8b8b447-7ff1-4387-bf6c-e5d61a6da00f-kube-api-access-jktd6\" (UniqueName: \"kubernetes.io/projected/f8b8b447-7ff1-4387-bf6c-e5d61a6da00f-kube-api-access-jktd6\") pod \"f8b8b447-7ff1-4387-bf6c-e5d61a6da00f\" (UID: \"f8b8b447-7ff1-4387-bf6c-e5d61a6da00f\") "
Oct 31 12:42:54.867517 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:54.864094    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d243b8a-d515-4b0e-ae52-d74de6e56550-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d243b8a-d515-4b0e-ae52-d74de6e56550-maas-api-openshift-ca\") pod \"7d243b8a-d515-4b0e-ae52-d74de6e56550\" (UID: \"7d243b8a-d515-4b0e-ae52-d74de6e56550\") "
Oct 31 12:42:54.867517 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:54.864205    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7d243b8a-d515-4b0e-ae52-d74de6e56550-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d243b8a-d515-4b0e-ae52-d74de6e56550-maas-api-xks-ca\") pod \"7d243b8a-d515-4b0e-ae52-d74de6e56550\" (UID: \"7d243b8a-d515-4b0e-ae52-d74de6e56550\") "
Oct 31 12:42:54.867517 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:54.864257    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d243b8a-d515-4b0e-ae52-d74de6e56550-kube-api-access-bqmpk\" (UniqueName: \"kubernetes.io/projected/7d243b8a-d515-4b0e-ae52-d74de6e56550-kube-api-access-bqmpk\") pod \"7d243b8a-d515-4b0e-ae52-d74de6e56550\" (UID: \"7d243b8a-d515-4b0e-ae52-d74de6e56550\") "
Oct 31 12:42:54.910525 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:54.895746    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x2qzv" podUID="c247ed2a-aa90-424b-9b7e-4f2936d67e3f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:42:54.972481 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:54.969363    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da0cf5fb-f52a-4cff-98b8-75ce9fcf48b9-kube-api-access-jtxkd\" (UniqueName: \"kubernetes.io/projected/da0cf5fb-f52a-4cff-98b8-75ce9fcf48b9-kube-api-access-jtxkd\") pod \"da0cf5fb-f52a-4cff-98b8-75ce9fcf48b9\" (UID: \"da0cf5fb-f52a-4cff-98b8-75ce9fcf48b9\") "
Oct 31 12:42:54.972481 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:54.969526    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da0cf5fb-f52a-4cff-98b8-75ce9fcf48b9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da0cf5fb-f52a-4cff-98b8-75ce9fcf48b9-maas-api-xks-ca\") pod \"da0cf5fb-f52a-4cff-98b8-75ce9fcf48b9\" (UID: \"da0cf5fb-f52a-4cff-98b8-75ce9fcf48b9\") "
Oct 31 12:42:54.972481 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:54.969585    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da0cf5fb-f52a-4cff-98b8-75ce9fcf48b9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da0cf5fb-f52a-4cff-98b8-75ce9fcf48b9-maas-api-openshift-ca\") pod \"da0cf5fb-f52a-4cff-98b8-75ce9fcf48b9\" (UID: \"da0cf5fb-f52a-4cff-98b8-75ce9fcf48b9\") "
Oct 31 12:42:55.023739 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:54.993651    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-b849a4c4-openshift-default-76d4f9446c-blrcp"]
Oct 31 12:42:55.023739 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:54.993739    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c67de2/simulator-w4-c67de2-kserve-868679cf9b-gqjbx"
Oct 31 12:42:55.023739 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:54.993777    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e6c79d/premium-w2-e6c79d-kserve-6468bf46d4-h66f8"
Oct 31 12:42:55.023739 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:54.993810    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x2qzv"
Oct 31 12:42:55.023739 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:54.995133    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b849a4c4-openshift-default-76d4f9446c-blrcp"
Oct 31 12:42:55.102899 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:55.035833    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c302cfbf-gt8wk"]
Oct 31 12:42:55.269166 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:55.073012    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/simulator-w4-c67de2-kserve-868679cf9b-gqjbx" podUID="e104d7da-1fb2-4c99-a480-b4ae84c4f273" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:42:55.269696 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:55.105098    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d243b8a-d515-4b0e-ae52-d74de6e56550-maas-api-openshift-ca" pod "7d243b8a-d515-4b0e-ae52-d74de6e56550" (UID: "7d243b8a-d515-4b0e-ae52-d74de6e56550"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:42:55.269857 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:55.184060    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da0cf5fb-f52a-4cff-98b8-75ce9fcf48b9-maas-api-openshift-ca" pod "da0cf5fb-f52a-4cff-98b8-75ce9fcf48b9" (UID: "da0cf5fb-f52a-4cff-98b8-75ce9fcf48b9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:42:55.269932 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:55.139270    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-b849a4c4-openshift-default-dockercfg-6kzj5\""
Oct 31 12:42:55.270035 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:55.117633    5060 scope.go:122] "RemoveContainer" containerID="3dd31712197133f0a2ba09b600d8435cf511c4068b7daedc3eb24bb1a2bbec70"
Oct 31 12:42:55.270239 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:55.145888    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd869a1f-8f33-4d47-86a9-b01a13d2c088-credential-socket\") pod \"e2e-ait-b849a4c4-openshift-default-76d4f9446c-blrcp\" (UID: \"fd869a1f-8f33-4d47-86a9-b01a13d2c088\") " pod="openshift-ingress/e2e-ait-b849a4c4-openshift-default-76d4f9446c-blrcp"
Oct 31 12:42:55.461579 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:55.461541    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd869a1f-8f33-4d47-86a9-b01a13d2c088-istiod-ca-cert\") pod \"e2e-ait-b849a4c4-openshift-default-76d4f9446c-blrcp\" (UID: \"fd869a1f-8f33-4d47-86a9-b01a13d2c088\") " pod="openshift-ingress/e2e-ait-b849a4c4-openshift-default-76d4f9446c-blrcp"
Oct 31 12:42:55.495545 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:55.487246    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd869a1f-8f33-4d47-86a9-b01a13d2c088-istio-data\") pod \"e2e-ait-b849a4c4-openshift-default-76d4f9446c-blrcp\" (UID: \"fd869a1f-8f33-4d47-86a9-b01a13d2c088\") " pod="openshift-ingress/e2e-ait-b849a4c4-openshift-default-76d4f9446c-blrcp"
Oct 31 12:42:55.495545 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:55.487336    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd869a1f-8f33-4d47-86a9-b01a13d2c088-workload-socket\") pod \"e2e-ait-b849a4c4-openshift-default-76d4f9446c-blrcp\" (UID: \"fd869a1f-8f33-4d47-86a9-b01a13d2c088\") " pod="openshift-ingress/e2e-ait-b849a4c4-openshift-default-76d4f9446c-blrcp"
Oct 31 12:42:55.495545 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:55.487460    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p66xk\" (UniqueName: \"kubernetes.io/projected/fd869a1f-8f33-4d47-86a9-b01a13d2c088-kube-api-access-p66xk\") pod \"e2e-ait-b849a4c4-openshift-default-76d4f9446c-blrcp\" (UID: \"fd869a1f-8f33-4d47-86a9-b01a13d2c088\") " pod="openshift-ingress/e2e-ait-b849a4c4-openshift-default-76d4f9446c-blrcp"
Oct 31 12:42:55.501668 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:55.501629    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd869a1f-8f33-4d47-86a9-b01a13d2c088-workload-certs\") pod \"e2e-ait-b849a4c4-openshift-default-76d4f9446c-blrcp\" (UID: \"fd869a1f-8f33-4d47-86a9-b01a13d2c088\") " pod="openshift-ingress/e2e-ait-b849a4c4-openshift-default-76d4f9446c-blrcp"
Oct 31 12:42:55.537541 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:55.525095    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd869a1f-8f33-4d47-86a9-b01a13d2c088-istio-token\") pod \"e2e-ait-b849a4c4-openshift-default-76d4f9446c-blrcp\" (UID: \"fd869a1f-8f33-4d47-86a9-b01a13d2c088\") " pod="openshift-ingress/e2e-ait-b849a4c4-openshift-default-76d4f9446c-blrcp"
Oct 31 12:42:55.537541 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:55.525277    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd869a1f-8f33-4d47-86a9-b01a13d2c088-istio-podinfo\") pod \"e2e-ait-b849a4c4-openshift-default-76d4f9446c-blrcp\" (UID: \"fd869a1f-8f33-4d47-86a9-b01a13d2c088\") " pod="openshift-ingress/e2e-ait-b849a4c4-openshift-default-76d4f9446c-blrcp"
Oct 31 12:42:55.537541 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:55.525312    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd869a1f-8f33-4d47-86a9-b01a13d2c088-istio-envoy\") pod \"e2e-ait-b849a4c4-openshift-default-76d4f9446c-blrcp\" (UID: \"fd869a1f-8f33-4d47-86a9-b01a13d2c088\") " pod="openshift-ingress/e2e-ait-b849a4c4-openshift-default-76d4f9446c-blrcp"
Oct 31 12:42:55.537541 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:55.525444    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da0cf5fb-f52a-4cff-98b8-75ce9fcf48b9-maas-api-openshift-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:42:55.537541 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:55.525463    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d243b8a-d515-4b0e-ae52-d74de6e56550-maas-api-openshift-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:42:55.537541 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:55.297584    5060 cpu_manager.go:414] "containerMap: removing container" podUID="da0cf5fb-f52a-4cff-98b8-75ce9fcf48b9" containerName="revoke-keys"
Oct 31 12:42:55.537541 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:55.525844    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da0cf5fb-f52a-4cff-98b8-75ce9fcf48b9" containerName="revoke-keys"
Oct 31 12:42:55.537541 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:55.525919    5060 cpu_manager.go:414] "containerMap: removing container" podUID="7d243b8a-d515-4b0e-ae52-d74de6e56550" containerName="revoke-keys"
Oct 31 12:42:55.537541 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:55.525928    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d243b8a-d515-4b0e-ae52-d74de6e56550" containerName="revoke-keys"
Oct 31 12:42:55.537541 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:55.526208    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="da0cf5fb-f52a-4cff-98b8-75ce9fcf48b9" containerName="revoke-keys"
Oct 31 12:42:55.537541 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:55.526234    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d243b8a-d515-4b0e-ae52-d74de6e56550" containerName="revoke-keys"
Oct 31 12:42:55.746614 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:55.569684    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c302cfbf-gt8wk"
Oct 31 12:42:55.746614 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:55.664124    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/premium-w2-e6c79d-kserve-6468bf46d4-h66f8" podUID="cd6a266f-8d12-4089-83ee-c75c716bf26c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Oct 31 12:42:55.746614 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:55.664347    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-2-w5-b25d3b-kserve-76f798d59f-vvzhg" podUID="394ed76c-84ac-4612-9acb-a7e36cb68d89" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": context deadline exceeded"
Oct 31 12:42:55.746614 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:55.522406    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da0cf5fb-f52a-4cff-98b8-75ce9fcf48b9-maas-api-xks-ca" pod "da0cf5fb-f52a-4cff-98b8-75ce9fcf48b9" (UID: "da0cf5fb-f52a-4cff-98b8-75ce9fcf48b9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:42:55.746614 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:55.692081    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd869a1f-8f33-4d47-86a9-b01a13d2c088-istio-data\") pod \"e2e-ait-b849a4c4-openshift-default-76d4f9446c-blrcp\" (UID: \"fd869a1f-8f33-4d47-86a9-b01a13d2c088\") " pod="openshift-ingress/e2e-ait-b849a4c4-openshift-default-76d4f9446c-blrcp"
Oct 31 12:42:55.746614 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:55.692147    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd869a1f-8f33-4d47-86a9-b01a13d2c088-workload-socket\") pod \"e2e-ait-b849a4c4-openshift-default-76d4f9446c-blrcp\" (UID: \"fd869a1f-8f33-4d47-86a9-b01a13d2c088\") " pod="openshift-ingress/e2e-ait-b849a4c4-openshift-default-76d4f9446c-blrcp"
Oct 31 12:42:55.746614 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:55.692219    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p66xk\" (UniqueName: \"kubernetes.io/projected/fd869a1f-8f33-4d47-86a9-b01a13d2c088-kube-api-access-p66xk\") pod \"e2e-ait-b849a4c4-openshift-default-76d4f9446c-blrcp\" (UID: \"fd869a1f-8f33-4d47-86a9-b01a13d2c088\") " pod="openshift-ingress/e2e-ait-b849a4c4-openshift-default-76d4f9446c-blrcp"
Oct 31 12:42:55.746614 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:55.692274    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd869a1f-8f33-4d47-86a9-b01a13d2c088-workload-certs\") pod \"e2e-ait-b849a4c4-openshift-default-76d4f9446c-blrcp\" (UID: \"fd869a1f-8f33-4d47-86a9-b01a13d2c088\") " pod="openshift-ingress/e2e-ait-b849a4c4-openshift-default-76d4f9446c-blrcp"
Oct 31 12:42:55.746614 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:55.692407    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd869a1f-8f33-4d47-86a9-b01a13d2c088-istio-token\") pod \"e2e-ait-b849a4c4-openshift-default-76d4f9446c-blrcp\" (UID: \"fd869a1f-8f33-4d47-86a9-b01a13d2c088\") " pod="openshift-ingress/e2e-ait-b849a4c4-openshift-default-76d4f9446c-blrcp"
Oct 31 12:42:55.746614 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:55.692550    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd869a1f-8f33-4d47-86a9-b01a13d2c088-istio-podinfo\") pod \"e2e-ait-b849a4c4-openshift-default-76d4f9446c-blrcp\" (UID: \"fd869a1f-8f33-4d47-86a9-b01a13d2c088\") " pod="openshift-ingress/e2e-ait-b849a4c4-openshift-default-76d4f9446c-blrcp"
Oct 31 12:42:55.746614 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:55.692580    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd869a1f-8f33-4d47-86a9-b01a13d2c088-istio-envoy\") pod \"e2e-ait-b849a4c4-openshift-default-76d4f9446c-blrcp\" (UID: \"fd869a1f-8f33-4d47-86a9-b01a13d2c088\") " pod="openshift-ingress/e2e-ait-b849a4c4-openshift-default-76d4f9446c-blrcp"
Oct 31 12:42:55.746614 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:55.692614    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd869a1f-8f33-4d47-86a9-b01a13d2c088-credential-socket\") pod \"e2e-ait-b849a4c4-openshift-default-76d4f9446c-blrcp\" (UID: \"fd869a1f-8f33-4d47-86a9-b01a13d2c088\") " pod="openshift-ingress/e2e-ait-b849a4c4-openshift-default-76d4f9446c-blrcp"
Oct 31 12:42:55.746614 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:55.692657    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd869a1f-8f33-4d47-86a9-b01a13d2c088-istiod-ca-cert\") pod \"e2e-ait-b849a4c4-openshift-default-76d4f9446c-blrcp\" (UID: \"fd869a1f-8f33-4d47-86a9-b01a13d2c088\") " pod="openshift-ingress/e2e-ait-b849a4c4-openshift-default-76d4f9446c-blrcp"
Oct 31 12:42:55.746614 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:55.692766    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da0cf5fb-f52a-4cff-98b8-75ce9fcf48b9-maas-api-xks-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:42:55.796856 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:55.771148    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd869a1f-8f33-4d47-86a9-b01a13d2c088-istio-data\") pod \"e2e-ait-b849a4c4-openshift-default-76d4f9446c-blrcp\" (UID: \"fd869a1f-8f33-4d47-86a9-b01a13d2c088\") " pod="openshift-ingress/e2e-ait-b849a4c4-openshift-default-76d4f9446c-blrcp"
Oct 31 12:42:55.828598 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:55.828414    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd869a1f-8f33-4d47-86a9-b01a13d2c088-workload-certs\") pod \"e2e-ait-b849a4c4-openshift-default-76d4f9446c-blrcp\" (UID: \"fd869a1f-8f33-4d47-86a9-b01a13d2c088\") " pod="openshift-ingress/e2e-ait-b849a4c4-openshift-default-76d4f9446c-blrcp"
Oct 31 12:42:55.899531 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:55.870878    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd869a1f-8f33-4d47-86a9-b01a13d2c088-istiod-ca-cert\") pod \"e2e-ait-b849a4c4-openshift-default-76d4f9446c-blrcp\" (UID: \"fd869a1f-8f33-4d47-86a9-b01a13d2c088\") " pod="openshift-ingress/e2e-ait-b849a4c4-openshift-default-76d4f9446c-blrcp"
Oct 31 12:42:56.012979 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:56.012891    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd869a1f-8f33-4d47-86a9-b01a13d2c088-workload-socket\") pod \"e2e-ait-b849a4c4-openshift-default-76d4f9446c-blrcp\" (UID: \"fd869a1f-8f33-4d47-86a9-b01a13d2c088\") " pod="openshift-ingress/e2e-ait-b849a4c4-openshift-default-76d4f9446c-blrcp"
Oct 31 12:42:56.020741 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:56.014127    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da0cf5fb-f52a-4cff-98b8-75ce9fcf48b9-kube-api-access-jtxkd" pod "da0cf5fb-f52a-4cff-98b8-75ce9fcf48b9" (UID: "da0cf5fb-f52a-4cff-98b8-75ce9fcf48b9"). InnerVolumeSpecName "kube-api-access-jtxkd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:42:56.177077 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:56.177043    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jtxkd\" (UniqueName: \"kubernetes.io/projected/da0cf5fb-f52a-4cff-98b8-75ce9fcf48b9-kube-api-access-jtxkd\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:42:56.197566 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:56.197523    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x2qzv" podUID="c247ed2a-aa90-424b-9b7e-4f2936d67e3f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded"
Oct 31 12:42:56.235517 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:56.235467    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-341d3105-c772h"
Oct 31 12:42:56.437151 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:56.437105    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-rm7cl" podUID="8292eaa8-0a02-46be-8654-3c4136ff604f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:42:56.511952 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:56.508749    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/simulator-w4-c67de2-kserve-868679cf9b-gqjbx" podUID="e104d7da-1fb2-4c99-a480-b4ae84c4f273" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded"
Oct 31 12:42:56.520123 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:56.520083    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-rbn8q" podUID="dea929f7-61c3-4921-9bdb-106f763e067e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:42:56.525913 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:56.522689    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e5cad97-aa60-45de-a22a-49ea851c9a38-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c302cfbf-gt8wk\" (UID: \"5e5cad97-aa60-45de-a22a-49ea851c9a38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c302cfbf-gt8wk"
Oct 31 12:42:56.525913 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:56.522932    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e5cad97-aa60-45de-a22a-49ea851c9a38-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c302cfbf-gt8wk\" (UID: \"5e5cad97-aa60-45de-a22a-49ea851c9a38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c302cfbf-gt8wk"
Oct 31 12:42:56.525913 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:56.523042    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dv46\" (UniqueName: \"kubernetes.io/projected/5e5cad97-aa60-45de-a22a-49ea851c9a38-kube-api-access-2dv46\") pod \"maas-api-revoke-sub-c302cfbf-gt8wk\" (UID: \"5e5cad97-aa60-45de-a22a-49ea851c9a38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c302cfbf-gt8wk"
Oct 31 12:42:56.575236 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:56.575185    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd869a1f-8f33-4d47-86a9-b01a13d2c088-credential-socket\") pod \"e2e-ait-b849a4c4-openshift-default-76d4f9446c-blrcp\" (UID: \"fd869a1f-8f33-4d47-86a9-b01a13d2c088\") " pod="openshift-ingress/e2e-ait-b849a4c4-openshift-default-76d4f9446c-blrcp"
Oct 31 12:42:56.653858 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:56.653817    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dv46\" (UniqueName: \"kubernetes.io/projected/5e5cad97-aa60-45de-a22a-49ea851c9a38-kube-api-access-2dv46\") pod \"maas-api-revoke-sub-c302cfbf-gt8wk\" (UID: \"5e5cad97-aa60-45de-a22a-49ea851c9a38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c302cfbf-gt8wk"
Oct 31 12:42:56.654315 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:56.654286    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e5cad97-aa60-45de-a22a-49ea851c9a38-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c302cfbf-gt8wk\" (UID: \"5e5cad97-aa60-45de-a22a-49ea851c9a38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c302cfbf-gt8wk"
Oct 31 12:42:56.654709 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:56.654684    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e5cad97-aa60-45de-a22a-49ea851c9a38-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c302cfbf-gt8wk\" (UID: \"5e5cad97-aa60-45de-a22a-49ea851c9a38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c302cfbf-gt8wk"
Oct 31 12:42:56.655986 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:56.655949    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e5cad97-aa60-45de-a22a-49ea851c9a38-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c302cfbf-gt8wk\" (UID: \"5e5cad97-aa60-45de-a22a-49ea851c9a38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c302cfbf-gt8wk"
Oct 31 12:42:56.668388 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:42:56.668361    5060 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.115s"
Oct 31 12:42:56.698928 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:56.678395    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-2-w5-b25d3b-kserve-76f798d59f-vvzhg"
Oct 31 12:42:56.699166 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:56.699148    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e6c79d/premium-w2-e6c79d-kserve-6468bf46d4-h66f8"
Oct 31 12:42:56.699463 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:56.699447    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c302cfbf-gt8wk"]
Oct 31 12:42:56.699745 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:56.699727    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-341d3105-c772h" event={"ID":"7d243b8a-d515-4b0e-ae52-d74de6e56550","Type":"ContainerDied","Data":"e0dad3cd46299ccd75d696b547ba3ea081eb9b64bcdd36c73b1266e51ae77310"}
Oct 31 12:42:56.708356 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:56.692950    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7017-30a720"
Oct 31 12:42:56.709897 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:56.682976    5060 generic.go:356] "Generic (PLEG): container finished" podID="731c6831-ebf8-4055-a5bc-875a4af865ce" containerID="40a9c5b5b642065c5f382c5d1152c0e1c055d47d3d9159f91dbc09a547f8d1e9" exitCode=137
Oct 31 12:42:56.716035 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:56.686811    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd869a1f-8f33-4d47-86a9-b01a13d2c088-istio-envoy\") pod \"e2e-ait-b849a4c4-openshift-default-76d4f9446c-blrcp\" (UID: \"fd869a1f-8f33-4d47-86a9-b01a13d2c088\") " pod="openshift-ingress/e2e-ait-b849a4c4-openshift-default-76d4f9446c-blrcp"
Oct 31 12:42:56.722607 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:56.689679    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd869a1f-8f33-4d47-86a9-b01a13d2c088-istio-token\") pod \"e2e-ait-b849a4c4-openshift-default-76d4f9446c-blrcp\" (UID: \"fd869a1f-8f33-4d47-86a9-b01a13d2c088\") " pod="openshift-ingress/e2e-ait-b849a4c4-openshift-default-76d4f9446c-blrcp"
Oct 31 12:42:56.746389 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:56.701047    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p66xk\" (UniqueName: \"kubernetes.io/projected/fd869a1f-8f33-4d47-86a9-b01a13d2c088-kube-api-access-p66xk\") pod \"e2e-ait-b849a4c4-openshift-default-76d4f9446c-blrcp\" (UID: \"fd869a1f-8f33-4d47-86a9-b01a13d2c088\") " pod="openshift-ingress/e2e-ait-b849a4c4-openshift-default-76d4f9446c-blrcp"
Oct 31 12:42:56.746893 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:56.708542    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e0dad3cd46299ccd75d696b547ba3ea081eb9b64bcdd36c73b1266e51ae77310"
Oct 31 12:42:56.755340 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:56.709680    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d243b8a-d515-4b0e-ae52-d74de6e56550-kube-api-access-bqmpk" pod "7d243b8a-d515-4b0e-ae52-d74de6e56550" (UID: "7d243b8a-d515-4b0e-ae52-d74de6e56550"). InnerVolumeSpecName "kube-api-access-bqmpk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:42:56.755340 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:56.709759    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8b8b447-7ff1-4387-bf6c-e5d61a6da00f-kube-api-access-jktd6" pod "f8b8b447-7ff1-4387-bf6c-e5d61a6da00f" (UID: "f8b8b447-7ff1-4387-bf6c-e5d61a6da00f"). InnerVolumeSpecName "kube-api-access-jktd6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:42:56.755340 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:56.709816    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d243b8a-d515-4b0e-ae52-d74de6e56550-maas-api-xks-ca" pod "7d243b8a-d515-4b0e-ae52-d74de6e56550" (UID: "7d243b8a-d515-4b0e-ae52-d74de6e56550"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:42:56.755340 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:56.721172    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7017-21ac45"
Oct 31 12:42:56.755340 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:56.712992    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dv46\" (UniqueName: \"kubernetes.io/projected/5e5cad97-aa60-45de-a22a-49ea851c9a38-kube-api-access-2dv46\") pod \"maas-api-revoke-sub-c302cfbf-gt8wk\" (UID: \"5e5cad97-aa60-45de-a22a-49ea851c9a38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c302cfbf-gt8wk"
Oct 31 12:42:56.755907 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:56.718994    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06ffe023-2fswn"
Oct 31 12:42:56.756605 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:56.756120    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x2qzv"
Oct 31 12:42:56.756908 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:56.756870    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06ffe023-2fswn" event={"ID":"da0cf5fb-f52a-4cff-98b8-75ce9fcf48b9","Type":"ContainerDied","Data":"7a88f64fb5efd1e776e251c8cbda3e10698a37d493d6af9debe73ca9516c276e"}
Oct 31 12:42:56.757179 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:56.757166    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7a88f64fb5efd1e776e251c8cbda3e10698a37d493d6af9debe73ca9516c276e"
Oct 31 12:42:56.782269 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:56.757036    5060 scope.go:122] "RemoveContainer" containerID="40a9c5b5b642065c5f382c5d1152c0e1c055d47d3d9159f91dbc09a547f8d1e9"
Oct 31 12:42:56.844963 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:56.844911    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd869a1f-8f33-4d47-86a9-b01a13d2c088-istio-podinfo\") pod \"e2e-ait-b849a4c4-openshift-default-76d4f9446c-blrcp\" (UID: \"fd869a1f-8f33-4d47-86a9-b01a13d2c088\") " pod="openshift-ingress/e2e-ait-b849a4c4-openshift-default-76d4f9446c-blrcp"
Oct 31 12:42:56.940853 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:56.940790    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e5cad97-aa60-45de-a22a-49ea851c9a38-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c302cfbf-gt8wk\" (UID: \"5e5cad97-aa60-45de-a22a-49ea851c9a38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c302cfbf-gt8wk"
Oct 31 12:42:56.966529 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:56.966420    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jktd6\" (UniqueName: \"kubernetes.io/projected/f8b8b447-7ff1-4387-bf6c-e5d61a6da00f-kube-api-access-jktd6\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:42:56.974350 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:56.974308    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d243b8a-d515-4b0e-ae52-d74de6e56550-maas-api-xks-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:42:56.974727 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:56.974700    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bqmpk\" (UniqueName: \"kubernetes.io/projected/7d243b8a-d515-4b0e-ae52-d74de6e56550-kube-api-access-bqmpk\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:42:56.988106 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:56.988026    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b849a4c4-openshift-default-76d4f9446c-blrcp"
Oct 31 12:42:57.004103 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:42:57.003676    5060 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda0cf5fb_f52a_4cff_98b8_75ce9fcf48b9.slice/crio-7a88f64fb5efd1e776e251c8cbda3e10698a37d493d6af9debe73ca9516c276e\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod731c6831_ebf8_4055_a5bc_875a4af865ce.slice/crio-40a9c5b5b642065c5f382c5d1152c0e1c055d47d3d9159f91dbc09a547f8d1e9.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod731c6831_ebf8_4055_a5bc_875a4af865ce.slice/crio-conmon-40a9c5b5b642065c5f382c5d1152c0e1c055d47d3d9159f91dbc09a547f8d1e9.scope\": RecentStats: unable to find data in memory cache]"
Oct 31 12:42:57.008557 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:57.007771    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c67de2/simulator-w4-c67de2-kserve-868679cf9b-gqjbx"
Oct 31 12:42:57.056795 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:57.056379    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c302cfbf-gt8wk"
Oct 31 12:42:57.079931 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:57.078755    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/731c6831-ebf8-4055-a5bc-875a4af865ce-kube-api-access-vplk9\" (UniqueName: \"kubernetes.io/projected/731c6831-ebf8-4055-a5bc-875a4af865ce-kube-api-access-vplk9\") pod \"731c6831-ebf8-4055-a5bc-875a4af865ce\" (UID: \"731c6831-ebf8-4055-a5bc-875a4af865ce\") "
Oct 31 12:42:57.128008 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:57.127936    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:42:57.128336 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:57.128304    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:42:57.128693 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:57.128666    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:42:57.137260 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:57.137212    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/731c6831-ebf8-4055-a5bc-875a4af865ce-kube-api-access-vplk9" pod "731c6831-ebf8-4055-a5bc-875a4af865ce" (UID: "731c6831-ebf8-4055-a5bc-875a4af865ce"). InnerVolumeSpecName "kube-api-access-vplk9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:42:57.157100 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:57.156970    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="731c6831-ebf8-4055-a5bc-875a4af865ce" path="/var/lib/kubelet/pods/731c6831-ebf8-4055-a5bc-875a4af865ce/volumes"
Oct 31 12:42:57.159597 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:57.159570    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8b8b447-7ff1-4387-bf6c-e5d61a6da00f" path="/var/lib/kubelet/pods/f8b8b447-7ff1-4387-bf6c-e5d61a6da00f/volumes"
Oct 31 12:42:57.279852 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:57.262343    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vplk9\" (UniqueName: \"kubernetes.io/projected/731c6831-ebf8-4055-a5bc-875a4af865ce-kube-api-access-vplk9\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:42:57.412899 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:57.412128    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-223f1a1d-l6hj5"]
Oct 31 12:42:57.418752 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:57.418724    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-223f1a1d-l6hj5"
Oct 31 12:42:57.423745 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:57.423623    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-223f1a1d-l6hj5"]
Oct 31 12:42:57.479554 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:42:57.479485    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e5cad97_aa60_45de_a22a_49ea851c9a38.slice/crio-69d13ccdd313fc38dbc53eb979eb1e497bbb1e0a341c832cafbbdab09ac7925e WatchSource:0}: Error finding container 69d13ccdd313fc38dbc53eb979eb1e497bbb1e0a341c832cafbbdab09ac7925e: Status 404 returned error can't find the container with id 69d13ccdd313fc38dbc53eb979eb1e497bbb1e0a341c832cafbbdab09ac7925e
Oct 31 12:42:57.513673 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:57.513539    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-d5e8c3f2-openshift-default-5dc86f657c-j4wrx"]
Oct 31 12:42:57.545840 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:57.545793    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c302cfbf-gt8wk"]
Oct 31 12:42:57.580718 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:57.580385    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7aadf9e8-6412-439d-9b07-9f468cd6d91a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-223f1a1d-l6hj5\" (UID: \"7aadf9e8-6412-439d-9b07-9f468cd6d91a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-223f1a1d-l6hj5"
Oct 31 12:42:57.580718 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:57.580434    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7aadf9e8-6412-439d-9b07-9f468cd6d91a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-223f1a1d-l6hj5\" (UID: \"7aadf9e8-6412-439d-9b07-9f468cd6d91a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-223f1a1d-l6hj5"
Oct 31 12:42:57.580718 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:57.580475    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kt58z\" (UniqueName: \"kubernetes.io/projected/7aadf9e8-6412-439d-9b07-9f468cd6d91a-kube-api-access-kt58z\") pod \"maas-api-revoke-sub-223f1a1d-l6hj5\" (UID: \"7aadf9e8-6412-439d-9b07-9f468cd6d91a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-223f1a1d-l6hj5"
Oct 31 12:42:57.657100 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:42:57.657067    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd869a1f_8f33_4d47_86a9_b01a13d2c088.slice/crio-3c97f87061c81292d08d86726d3daab6252884446997656305e9f1d56ee4ee16 WatchSource:0}: Error finding container 3c97f87061c81292d08d86726d3daab6252884446997656305e9f1d56ee4ee16: Status 404 returned error can't find the container with id 3c97f87061c81292d08d86726d3daab6252884446997656305e9f1d56ee4ee16
Oct 31 12:42:57.658530 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:57.658478    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-b849a4c4-openshift-default-76d4f9446c-blrcp"]
Oct 31 12:42:57.659966 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:57.659932    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:42:57.660065 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:57.660014    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:42:57.660065 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:57.660058    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:42:57.681947 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:57.681865    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7aadf9e8-6412-439d-9b07-9f468cd6d91a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-223f1a1d-l6hj5\" (UID: \"7aadf9e8-6412-439d-9b07-9f468cd6d91a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-223f1a1d-l6hj5"
Oct 31 12:42:57.682868 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:57.682143    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7aadf9e8-6412-439d-9b07-9f468cd6d91a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-223f1a1d-l6hj5\" (UID: \"7aadf9e8-6412-439d-9b07-9f468cd6d91a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-223f1a1d-l6hj5"
Oct 31 12:42:57.682868 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:57.682229    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kt58z\" (UniqueName: \"kubernetes.io/projected/7aadf9e8-6412-439d-9b07-9f468cd6d91a-kube-api-access-kt58z\") pod \"maas-api-revoke-sub-223f1a1d-l6hj5\" (UID: \"7aadf9e8-6412-439d-9b07-9f468cd6d91a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-223f1a1d-l6hj5"
Oct 31 12:42:57.685346 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:57.685323    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7aadf9e8-6412-439d-9b07-9f468cd6d91a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-223f1a1d-l6hj5\" (UID: \"7aadf9e8-6412-439d-9b07-9f468cd6d91a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-223f1a1d-l6hj5"
Oct 31 12:42:57.689297 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:57.689251    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7aadf9e8-6412-439d-9b07-9f468cd6d91a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-223f1a1d-l6hj5\" (UID: \"7aadf9e8-6412-439d-9b07-9f468cd6d91a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-223f1a1d-l6hj5"
Oct 31 12:42:57.697387 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:57.697339    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kt58z\" (UniqueName: \"kubernetes.io/projected/7aadf9e8-6412-439d-9b07-9f468cd6d91a-kube-api-access-kt58z\") pod \"maas-api-revoke-sub-223f1a1d-l6hj5\" (UID: \"7aadf9e8-6412-439d-9b07-9f468cd6d91a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-223f1a1d-l6hj5"
Oct 31 12:42:57.736957 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:57.736915    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7017-30a720"
Oct 31 12:42:57.740939 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:42:57.740882    5060 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7017-30a720\" is forbidden: User \"system:node:ip-10-0-130-190.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-190.us-east-2.compute.internal' and this object" podUID="731c6831-ebf8-4055-a5bc-875a4af865ce" pod="opendatahub/test-curl-7017-30a720"
Oct 31 12:42:57.741108 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:57.740988    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c302cfbf-gt8wk" event={"ID":"5e5cad97-aa60-45de-a22a-49ea851c9a38","Type":"ContainerStarted","Data":"69d13ccdd313fc38dbc53eb979eb1e497bbb1e0a341c832cafbbdab09ac7925e"}
Oct 31 12:42:57.744510 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:42:57.744454    5060 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7017-30a720\" is forbidden: User \"system:node:ip-10-0-130-190.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-190.us-east-2.compute.internal' and this object" podUID="731c6831-ebf8-4055-a5bc-875a4af865ce" pod="opendatahub/test-curl-7017-30a720"
Oct 31 12:42:57.745737 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:57.745707    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b849a4c4-openshift-default-76d4f9446c-blrcp" event={"ID":"fd869a1f-8f33-4d47-86a9-b01a13d2c088","Type":"ContainerStarted","Data":"3c97f87061c81292d08d86726d3daab6252884446997656305e9f1d56ee4ee16"}
Oct 31 12:42:57.746802 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:57.746783    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-223f1a1d-l6hj5"
Oct 31 12:42:57.748306 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:57.748274    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-d5e8c3f2-openshift-default-5dc86f657c-j4wrx" event={"ID":"f135ca99-10b3-4413-8d5f-1541b2aae5cf","Type":"ContainerStarted","Data":"a3ec3fae13ce05f9dfd02a3b10d4fc37bc7f4f5c366ef5d2f09480420a60f3eb"}
Oct 31 12:42:57.748714 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:57.748337    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-d5e8c3f2-openshift-default-5dc86f657c-j4wrx" event={"ID":"f135ca99-10b3-4413-8d5f-1541b2aae5cf","Type":"ContainerStarted","Data":"6c175cf4dacc8bbfd2208d8e8e1ba9e8225838e6ddaf430f07e1b07d386a550b"}
Oct 31 12:42:57.752001 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:42:57.751975    5060 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7017-30a720\" is forbidden: User \"system:node:ip-10-0-130-190.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-190.us-east-2.compute.internal' and this object" podUID="731c6831-ebf8-4055-a5bc-875a4af865ce" pod="opendatahub/test-curl-7017-30a720"
Oct 31 12:42:57.782434 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:57.782380    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-d5e8c3f2-openshift-default-5dc86f657c-j4wrx" podStartSLOduration=8.782364184 podStartE2EDuration="8.782364184s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:42:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:42:57.774563684 +0000 UTC m=+940.877631395" watchObservedRunningTime="2026-10-31 12:42:57.782364184 +0000 UTC m=+940.885431895"
Oct 31 12:42:57.976971 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:57.974057    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-d5e8c3f2-openshift-default-5dc86f657c-j4wrx"
Oct 31 12:42:58.097736 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:42:58.097410    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7aadf9e8_6412_439d_9b07_9f468cd6d91a.slice/crio-84d699bb3eeae4342a0c50809cc90742f384d339ba9895220318274536ccc9cb WatchSource:0}: Error finding container 84d699bb3eeae4342a0c50809cc90742f384d339ba9895220318274536ccc9cb: Status 404 returned error can't find the container with id 84d699bb3eeae4342a0c50809cc90742f384d339ba9895220318274536ccc9cb
Oct 31 12:42:58.098525 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:58.098478    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-223f1a1d-l6hj5"]
Oct 31 12:42:58.760273 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:58.760170    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-223f1a1d-l6hj5" event={"ID":"7aadf9e8-6412-439d-9b07-9f468cd6d91a","Type":"ContainerStarted","Data":"06ced8366fdcd41e638098e6810a00b993bbbc8d565213ecd4cbe8caf635334a"}
Oct 31 12:42:58.760924 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:58.760869    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-223f1a1d-l6hj5" event={"ID":"7aadf9e8-6412-439d-9b07-9f468cd6d91a","Type":"ContainerStarted","Data":"84d699bb3eeae4342a0c50809cc90742f384d339ba9895220318274536ccc9cb"}
Oct 31 12:42:58.762993 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:58.762805    5060 generic.go:356] "Generic (PLEG): container finished" podID="5e5cad97-aa60-45de-a22a-49ea851c9a38" containerID="c36b807976c8918d4b6f4efede835f597e3d6bcbf6da477aedbbd666c4aa48ed" exitCode=0
Oct 31 12:42:58.762993 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:58.762841    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c302cfbf-gt8wk" event={"ID":"5e5cad97-aa60-45de-a22a-49ea851c9a38","Type":"ContainerDied","Data":"c36b807976c8918d4b6f4efede835f597e3d6bcbf6da477aedbbd666c4aa48ed"}
Oct 31 12:42:58.763311 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:42:58.763283    5060 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7017-30a720\" is forbidden: User \"system:node:ip-10-0-130-190.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-190.us-east-2.compute.internal' and this object" podUID="731c6831-ebf8-4055-a5bc-875a4af865ce" pod="opendatahub/test-curl-7017-30a720"
Oct 31 12:42:58.765240 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:58.765212    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b849a4c4-openshift-default-76d4f9446c-blrcp" event={"ID":"fd869a1f-8f33-4d47-86a9-b01a13d2c088","Type":"ContainerStarted","Data":"1dde22342826df50733b8414dca10bcbb0a90199ec0164bb228c2c3528e2e25b"}
Oct 31 12:42:58.782828 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:58.782780    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-223f1a1d-l6hj5" podStartSLOduration=2.7827667209999998 podStartE2EDuration="2.782766721s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:42:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:42:58.780440252 +0000 UTC m=+941.883507965" watchObservedRunningTime="2026-10-31 12:42:58.782766721 +0000 UTC m=+941.885834433"
Oct 31 12:42:58.804577 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:58.804485    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-b849a4c4-openshift-default-76d4f9446c-blrcp" podStartSLOduration=7.804471347 podStartE2EDuration="7.804471347s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:42:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:42:58.803458104 +0000 UTC m=+941.906525816" watchObservedRunningTime="2026-10-31 12:42:58.804471347 +0000 UTC m=+941.907539058"
Oct 31 12:42:58.823256 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:42:58.823218    5060 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7017-30a720\" is forbidden: User \"system:node:ip-10-0-130-190.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-190.us-east-2.compute.internal' and this object" podUID="731c6831-ebf8-4055-a5bc-875a4af865ce" pod="opendatahub/test-curl-7017-30a720"
Oct 31 12:42:58.994142 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:58.994048    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-b849a4c4-openshift-default-76d4f9446c-blrcp"
Oct 31 12:42:59.001542 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:59.001122    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-d5e8c3f2-openshift-default-5dc86f657c-j4wrx"
Oct 31 12:42:59.002009 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:59.001980    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-b849a4c4-openshift-default-76d4f9446c-blrcp"
Oct 31 12:42:59.039564 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:42:59.038566    5060 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7017-30a720\" is forbidden: User \"system:node:ip-10-0-130-190.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-190.us-east-2.compute.internal' and this object" podUID="731c6831-ebf8-4055-a5bc-875a4af865ce" pod="opendatahub/test-curl-7017-30a720"
Oct 31 12:42:59.068730 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:42:59.068687    5060 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7017-30a720\" is forbidden: User \"system:node:ip-10-0-130-190.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-190.us-east-2.compute.internal' and this object" podUID="731c6831-ebf8-4055-a5bc-875a4af865ce" pod="opendatahub/test-curl-7017-30a720"
Oct 31 12:42:59.778970 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:59.778934    5060 generic.go:356] "Generic (PLEG): container finished" podID="7aadf9e8-6412-439d-9b07-9f468cd6d91a" containerID="06ced8366fdcd41e638098e6810a00b993bbbc8d565213ecd4cbe8caf635334a" exitCode=0
Oct 31 12:42:59.779416 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:59.779023    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-223f1a1d-l6hj5" event={"ID":"7aadf9e8-6412-439d-9b07-9f468cd6d91a","Type":"ContainerDied","Data":"06ced8366fdcd41e638098e6810a00b993bbbc8d565213ecd4cbe8caf635334a"}
Oct 31 12:42:59.780026 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:59.779924    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-d5e8c3f2-openshift-default-5dc86f657c-j4wrx"
Oct 31 12:42:59.780026 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:59.779975    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-b849a4c4-openshift-default-76d4f9446c-blrcp"
Oct 31 12:42:59.781265 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:59.781242    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-d5e8c3f2-openshift-default-5dc86f657c-j4wrx"
Oct 31 12:42:59.781433 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:42:59.781338    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-b849a4c4-openshift-default-76d4f9446c-blrcp"
Oct 31 12:42:59.935553 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:42:59.935359    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7phb" containerName="extractor"
Oct 31 12:43:00.120092 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:00.120066    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c302cfbf-gt8wk"
Oct 31 12:43:00.289413 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:00.289376    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e5cad97-aa60-45de-a22a-49ea851c9a38-kube-api-access-2dv46\" (UniqueName: \"kubernetes.io/projected/5e5cad97-aa60-45de-a22a-49ea851c9a38-kube-api-access-2dv46\") pod \"5e5cad97-aa60-45de-a22a-49ea851c9a38\" (UID: \"5e5cad97-aa60-45de-a22a-49ea851c9a38\") "
Oct 31 12:43:00.289620 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:00.289442    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e5cad97-aa60-45de-a22a-49ea851c9a38-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e5cad97-aa60-45de-a22a-49ea851c9a38-maas-api-xks-ca\") pod \"5e5cad97-aa60-45de-a22a-49ea851c9a38\" (UID: \"5e5cad97-aa60-45de-a22a-49ea851c9a38\") "
Oct 31 12:43:00.289620 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:00.289565    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e5cad97-aa60-45de-a22a-49ea851c9a38-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e5cad97-aa60-45de-a22a-49ea851c9a38-maas-api-openshift-ca\") pod \"5e5cad97-aa60-45de-a22a-49ea851c9a38\" (UID: \"5e5cad97-aa60-45de-a22a-49ea851c9a38\") "
Oct 31 12:43:00.289944 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:00.289913    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e5cad97-aa60-45de-a22a-49ea851c9a38-maas-api-openshift-ca" pod "5e5cad97-aa60-45de-a22a-49ea851c9a38" (UID: "5e5cad97-aa60-45de-a22a-49ea851c9a38"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:43:00.290378 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:00.290346    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e5cad97-aa60-45de-a22a-49ea851c9a38-maas-api-openshift-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:43:00.292023 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:00.291995    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e5cad97-aa60-45de-a22a-49ea851c9a38-kube-api-access-2dv46" pod "5e5cad97-aa60-45de-a22a-49ea851c9a38" (UID: "5e5cad97-aa60-45de-a22a-49ea851c9a38"). InnerVolumeSpecName "kube-api-access-2dv46". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:43:00.292552 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:00.292528    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e5cad97-aa60-45de-a22a-49ea851c9a38-maas-api-xks-ca" pod "5e5cad97-aa60-45de-a22a-49ea851c9a38" (UID: "5e5cad97-aa60-45de-a22a-49ea851c9a38"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:43:00.392095 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:00.391994    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2dv46\" (UniqueName: \"kubernetes.io/projected/5e5cad97-aa60-45de-a22a-49ea851c9a38-kube-api-access-2dv46\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:43:00.392095 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:00.392036    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e5cad97-aa60-45de-a22a-49ea851c9a38-maas-api-xks-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:43:00.798828 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:00.798791    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c302cfbf-gt8wk"
Oct 31 12:43:00.803015 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:00.801929    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c302cfbf-gt8wk" event={"ID":"5e5cad97-aa60-45de-a22a-49ea851c9a38","Type":"ContainerDied","Data":"69d13ccdd313fc38dbc53eb979eb1e497bbb1e0a341c832cafbbdab09ac7925e"}
Oct 31 12:43:00.803015 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:00.801990    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="69d13ccdd313fc38dbc53eb979eb1e497bbb1e0a341c832cafbbdab09ac7925e"
Oct 31 12:43:01.473717 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:01.473685    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-223f1a1d-l6hj5"
Oct 31 12:43:01.625665 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:01.625629    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7aadf9e8-6412-439d-9b07-9f468cd6d91a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7aadf9e8-6412-439d-9b07-9f468cd6d91a-maas-api-openshift-ca\") pod \"7aadf9e8-6412-439d-9b07-9f468cd6d91a\" (UID: \"7aadf9e8-6412-439d-9b07-9f468cd6d91a\") "
Oct 31 12:43:01.625852 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:01.625696    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7aadf9e8-6412-439d-9b07-9f468cd6d91a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7aadf9e8-6412-439d-9b07-9f468cd6d91a-maas-api-xks-ca\") pod \"7aadf9e8-6412-439d-9b07-9f468cd6d91a\" (UID: \"7aadf9e8-6412-439d-9b07-9f468cd6d91a\") "
Oct 31 12:43:01.625852 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:01.625784    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7aadf9e8-6412-439d-9b07-9f468cd6d91a-kube-api-access-kt58z\" (UniqueName: \"kubernetes.io/projected/7aadf9e8-6412-439d-9b07-9f468cd6d91a-kube-api-access-kt58z\") pod \"7aadf9e8-6412-439d-9b07-9f468cd6d91a\" (UID: \"7aadf9e8-6412-439d-9b07-9f468cd6d91a\") "
Oct 31 12:43:01.626113 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:01.626083    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7aadf9e8-6412-439d-9b07-9f468cd6d91a-maas-api-openshift-ca" pod "7aadf9e8-6412-439d-9b07-9f468cd6d91a" (UID: "7aadf9e8-6412-439d-9b07-9f468cd6d91a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:43:01.626532 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:01.626503    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7aadf9e8-6412-439d-9b07-9f468cd6d91a-maas-api-openshift-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:43:01.627857 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:01.627825    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7aadf9e8-6412-439d-9b07-9f468cd6d91a-maas-api-xks-ca" pod "7aadf9e8-6412-439d-9b07-9f468cd6d91a" (UID: "7aadf9e8-6412-439d-9b07-9f468cd6d91a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:43:01.627988 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:01.627909    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7aadf9e8-6412-439d-9b07-9f468cd6d91a-kube-api-access-kt58z" pod "7aadf9e8-6412-439d-9b07-9f468cd6d91a" (UID: "7aadf9e8-6412-439d-9b07-9f468cd6d91a"). InnerVolumeSpecName "kube-api-access-kt58z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:43:01.728444 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:01.728353    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7aadf9e8-6412-439d-9b07-9f468cd6d91a-maas-api-xks-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:43:01.728444 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:01.728390    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kt58z\" (UniqueName: \"kubernetes.io/projected/7aadf9e8-6412-439d-9b07-9f468cd6d91a-kube-api-access-kt58z\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:43:01.844873 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:01.844823    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-223f1a1d-l6hj5"
Oct 31 12:43:01.845281 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:01.844913    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-223f1a1d-l6hj5" event={"ID":"7aadf9e8-6412-439d-9b07-9f468cd6d91a","Type":"ContainerDied","Data":"84d699bb3eeae4342a0c50809cc90742f384d339ba9895220318274536ccc9cb"}
Oct 31 12:43:01.845281 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:01.844984    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="84d699bb3eeae4342a0c50809cc90742f384d339ba9895220318274536ccc9cb"
Oct 31 12:43:05.340056 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:05.340022    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f6c73a95-bcsgz"]
Oct 31 12:43:05.340807 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:05.340792    5060 cpu_manager.go:414] "containerMap: removing container" podUID="7aadf9e8-6412-439d-9b07-9f468cd6d91a" containerName="revoke-keys"
Oct 31 12:43:05.340876 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:05.340808    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7aadf9e8-6412-439d-9b07-9f468cd6d91a" containerName="revoke-keys"
Oct 31 12:43:05.340876 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:05.340834    5060 cpu_manager.go:414] "containerMap: removing container" podUID="5e5cad97-aa60-45de-a22a-49ea851c9a38" containerName="revoke-keys"
Oct 31 12:43:05.340876 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:05.340839    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e5cad97-aa60-45de-a22a-49ea851c9a38" containerName="revoke-keys"
Oct 31 12:43:05.340965 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:05.340951    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e5cad97-aa60-45de-a22a-49ea851c9a38" containerName="revoke-keys"
Oct 31 12:43:05.341004 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:05.340966    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="7aadf9e8-6412-439d-9b07-9f468cd6d91a" containerName="revoke-keys"
Oct 31 12:43:05.344920 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:05.344903    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6c73a95-bcsgz"
Oct 31 12:43:05.352545 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:05.348987    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 12:43:05.352545 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:05.349918    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 12:43:05.352545 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:05.350031    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 12:43:05.352545 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:05.350627    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7wsmb\""
Oct 31 12:43:05.356830 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:05.356514    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f6c73a95-bcsgz"]
Oct 31 12:43:05.460623 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:05.460554    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e86e368c-5055-4433-abb6-f1171b74c21d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f6c73a95-bcsgz\" (UID: \"e86e368c-5055-4433-abb6-f1171b74c21d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6c73a95-bcsgz"
Oct 31 12:43:05.460917 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:05.460890    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e86e368c-5055-4433-abb6-f1171b74c21d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f6c73a95-bcsgz\" (UID: \"e86e368c-5055-4433-abb6-f1171b74c21d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6c73a95-bcsgz"
Oct 31 12:43:05.461010 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:05.460991    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7zzk\" (UniqueName: \"kubernetes.io/projected/e86e368c-5055-4433-abb6-f1171b74c21d-kube-api-access-w7zzk\") pod \"maas-api-revoke-sub-f6c73a95-bcsgz\" (UID: \"e86e368c-5055-4433-abb6-f1171b74c21d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6c73a95-bcsgz"
Oct 31 12:43:05.562041 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:05.562002    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e86e368c-5055-4433-abb6-f1171b74c21d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f6c73a95-bcsgz\" (UID: \"e86e368c-5055-4433-abb6-f1171b74c21d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6c73a95-bcsgz"
Oct 31 12:43:05.562228 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:05.562162    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e86e368c-5055-4433-abb6-f1171b74c21d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f6c73a95-bcsgz\" (UID: \"e86e368c-5055-4433-abb6-f1171b74c21d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6c73a95-bcsgz"
Oct 31 12:43:05.562228 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:05.562220    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7zzk\" (UniqueName: \"kubernetes.io/projected/e86e368c-5055-4433-abb6-f1171b74c21d-kube-api-access-w7zzk\") pod \"maas-api-revoke-sub-f6c73a95-bcsgz\" (UID: \"e86e368c-5055-4433-abb6-f1171b74c21d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6c73a95-bcsgz"
Oct 31 12:43:05.562738 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:05.562712    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e86e368c-5055-4433-abb6-f1171b74c21d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f6c73a95-bcsgz\" (UID: \"e86e368c-5055-4433-abb6-f1171b74c21d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6c73a95-bcsgz"
Oct 31 12:43:05.564755 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:05.564726    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e86e368c-5055-4433-abb6-f1171b74c21d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f6c73a95-bcsgz\" (UID: \"e86e368c-5055-4433-abb6-f1171b74c21d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6c73a95-bcsgz"
Oct 31 12:43:05.579347 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:05.579323    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7zzk\" (UniqueName: \"kubernetes.io/projected/e86e368c-5055-4433-abb6-f1171b74c21d-kube-api-access-w7zzk\") pod \"maas-api-revoke-sub-f6c73a95-bcsgz\" (UID: \"e86e368c-5055-4433-abb6-f1171b74c21d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6c73a95-bcsgz"
Oct 31 12:43:05.673701 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:05.673629    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6c73a95-bcsgz"
Oct 31 12:43:06.097861 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:06.097835    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f6c73a95-bcsgz"]
Oct 31 12:43:06.100071 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:43:06.100044    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode86e368c_5055_4433_abb6_f1171b74c21d.slice/crio-14e5b56d62378dbc3f4afdba11ef8d9cba6c909e4fa9cfee620426fe86475ccd WatchSource:0}: Error finding container 14e5b56d62378dbc3f4afdba11ef8d9cba6c909e4fa9cfee620426fe86475ccd: Status 404 returned error can't find the container with id 14e5b56d62378dbc3f4afdba11ef8d9cba6c909e4fa9cfee620426fe86475ccd
Oct 31 12:43:06.929662 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:06.929622    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6c73a95-bcsgz" event={"ID":"e86e368c-5055-4433-abb6-f1171b74c21d","Type":"ContainerStarted","Data":"4030fb42cfe547701ce89efbfd321769c56427be03b55a00a1861a6fe5e0368c"}
Oct 31 12:43:06.930070 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:06.929680    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6c73a95-bcsgz" event={"ID":"e86e368c-5055-4433-abb6-f1171b74c21d","Type":"ContainerStarted","Data":"14e5b56d62378dbc3f4afdba11ef8d9cba6c909e4fa9cfee620426fe86475ccd"}
Oct 31 12:43:06.951180 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:06.951126    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6c73a95-bcsgz" podStartSLOduration=1.951111977 podStartE2EDuration="1.951111977s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:43:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:43:06.94955265 +0000 UTC m=+950.052620364" watchObservedRunningTime="2026-10-31 12:43:06.951111977 +0000 UTC m=+950.054179689"
Oct 31 12:43:07.945597 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:07.945556    5060 generic.go:356] "Generic (PLEG): container finished" podID="e86e368c-5055-4433-abb6-f1171b74c21d" containerID="4030fb42cfe547701ce89efbfd321769c56427be03b55a00a1861a6fe5e0368c" exitCode=0
Oct 31 12:43:07.946007 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:07.945655    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6c73a95-bcsgz" event={"ID":"e86e368c-5055-4433-abb6-f1171b74c21d","Type":"ContainerDied","Data":"4030fb42cfe547701ce89efbfd321769c56427be03b55a00a1861a6fe5e0368c"}
Oct 31 12:43:08.205955 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:08.205862    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-rr2hg"]
Oct 31 12:43:08.211418 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:08.211394    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-rr2hg"
Oct 31 12:43:08.227050 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:08.227022    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-fcn5x"]
Oct 31 12:43:08.232300 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:08.232277    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-fcn5x"
Oct 31 12:43:08.237774 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:08.237745    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-rr2hg"]
Oct 31 12:43:08.261003 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:08.260959    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-fcn5x"]
Oct 31 12:43:08.292517 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:08.289721    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prvgm\" (UniqueName: \"kubernetes.io/projected/aae46d99-cc6d-4be9-868b-8c4dda15a7a7-kube-api-access-prvgm\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-fcn5x\" (UID: \"aae46d99-cc6d-4be9-868b-8c4dda15a7a7\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-fcn5x"
Oct 31 12:43:08.292517 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:08.289837    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aae46d99-cc6d-4be9-868b-8c4dda15a7a7-workload-certs\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-fcn5x\" (UID: \"aae46d99-cc6d-4be9-868b-8c4dda15a7a7\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-fcn5x"
Oct 31 12:43:08.292517 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:08.289937    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15a47e5b-2ade-4ca3-b0d0-6e4c09917588-credential-socket\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-rr2hg\" (UID: \"15a47e5b-2ade-4ca3-b0d0-6e4c09917588\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-rr2hg"
Oct 31 12:43:08.292517 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:08.290035    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15a47e5b-2ade-4ca3-b0d0-6e4c09917588-istio-envoy\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-rr2hg\" (UID: \"15a47e5b-2ade-4ca3-b0d0-6e4c09917588\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-rr2hg"
Oct 31 12:43:08.292517 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:08.290114    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15a47e5b-2ade-4ca3-b0d0-6e4c09917588-istio-podinfo\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-rr2hg\" (UID: \"15a47e5b-2ade-4ca3-b0d0-6e4c09917588\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-rr2hg"
Oct 31 12:43:08.292517 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:08.290236    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15a47e5b-2ade-4ca3-b0d0-6e4c09917588-workload-certs\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-rr2hg\" (UID: \"15a47e5b-2ade-4ca3-b0d0-6e4c09917588\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-rr2hg"
Oct 31 12:43:08.292517 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:08.290301    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aae46d99-cc6d-4be9-868b-8c4dda15a7a7-istio-data\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-fcn5x\" (UID: \"aae46d99-cc6d-4be9-868b-8c4dda15a7a7\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-fcn5x"
Oct 31 12:43:08.292517 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:08.290325    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aae46d99-cc6d-4be9-868b-8c4dda15a7a7-istio-podinfo\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-fcn5x\" (UID: \"aae46d99-cc6d-4be9-868b-8c4dda15a7a7\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-fcn5x"
Oct 31 12:43:08.292517 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:08.290353    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aae46d99-cc6d-4be9-868b-8c4dda15a7a7-istio-envoy\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-fcn5x\" (UID: \"aae46d99-cc6d-4be9-868b-8c4dda15a7a7\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-fcn5x"
Oct 31 12:43:08.292517 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:08.290378    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aae46d99-cc6d-4be9-868b-8c4dda15a7a7-istiod-ca-cert\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-fcn5x\" (UID: \"aae46d99-cc6d-4be9-868b-8c4dda15a7a7\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-fcn5x"
Oct 31 12:43:08.292517 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:08.290415    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4wlk\" (UniqueName: \"kubernetes.io/projected/15a47e5b-2ade-4ca3-b0d0-6e4c09917588-kube-api-access-t4wlk\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-rr2hg\" (UID: \"15a47e5b-2ade-4ca3-b0d0-6e4c09917588\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-rr2hg"
Oct 31 12:43:08.292517 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:08.290480    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15a47e5b-2ade-4ca3-b0d0-6e4c09917588-istio-data\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-rr2hg\" (UID: \"15a47e5b-2ade-4ca3-b0d0-6e4c09917588\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-rr2hg"
Oct 31 12:43:08.292517 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:08.290605    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aae46d99-cc6d-4be9-868b-8c4dda15a7a7-workload-socket\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-fcn5x\" (UID: \"aae46d99-cc6d-4be9-868b-8c4dda15a7a7\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-fcn5x"
Oct 31 12:43:08.292517 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:08.290645    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15a47e5b-2ade-4ca3-b0d0-6e4c09917588-istiod-ca-cert\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-rr2hg\" (UID: \"15a47e5b-2ade-4ca3-b0d0-6e4c09917588\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-rr2hg"
Oct 31 12:43:08.292517 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:08.290689    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aae46d99-cc6d-4be9-868b-8c4dda15a7a7-credential-socket\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-fcn5x\" (UID: \"aae46d99-cc6d-4be9-868b-8c4dda15a7a7\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-fcn5x"
Oct 31 12:43:08.292517 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:08.290712    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aae46d99-cc6d-4be9-868b-8c4dda15a7a7-istio-token\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-fcn5x\" (UID: \"aae46d99-cc6d-4be9-868b-8c4dda15a7a7\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-fcn5x"
Oct 31 12:43:08.292517 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:08.290757    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15a47e5b-2ade-4ca3-b0d0-6e4c09917588-istio-token\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-rr2hg\" (UID: \"15a47e5b-2ade-4ca3-b0d0-6e4c09917588\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-rr2hg"
Oct 31 12:43:08.292517 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:08.290849    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15a47e5b-2ade-4ca3-b0d0-6e4c09917588-workload-socket\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-rr2hg\" (UID: \"15a47e5b-2ade-4ca3-b0d0-6e4c09917588\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-rr2hg"
Oct 31 12:43:08.392803 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:08.392762    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15a47e5b-2ade-4ca3-b0d0-6e4c09917588-istio-envoy\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-rr2hg\" (UID: \"15a47e5b-2ade-4ca3-b0d0-6e4c09917588\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-rr2hg"
Oct 31 12:43:08.393000 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:08.392824    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15a47e5b-2ade-4ca3-b0d0-6e4c09917588-istio-podinfo\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-rr2hg\" (UID: \"15a47e5b-2ade-4ca3-b0d0-6e4c09917588\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-rr2hg"
Oct 31 12:43:08.393082 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:08.393058    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15a47e5b-2ade-4ca3-b0d0-6e4c09917588-workload-certs\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-rr2hg\" (UID: \"15a47e5b-2ade-4ca3-b0d0-6e4c09917588\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-rr2hg"
Oct 31 12:43:08.393147 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:08.393121    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aae46d99-cc6d-4be9-868b-8c4dda15a7a7-istio-data\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-fcn5x\" (UID: \"aae46d99-cc6d-4be9-868b-8c4dda15a7a7\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-fcn5x"
Oct 31 12:43:08.393204 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:08.393147    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aae46d99-cc6d-4be9-868b-8c4dda15a7a7-istio-podinfo\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-fcn5x\" (UID: \"aae46d99-cc6d-4be9-868b-8c4dda15a7a7\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-fcn5x"
Oct 31 12:43:08.393204 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:08.393175    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aae46d99-cc6d-4be9-868b-8c4dda15a7a7-istio-envoy\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-fcn5x\" (UID: \"aae46d99-cc6d-4be9-868b-8c4dda15a7a7\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-fcn5x"
Oct 31 12:43:08.393312 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:08.393201    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aae46d99-cc6d-4be9-868b-8c4dda15a7a7-istiod-ca-cert\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-fcn5x\" (UID: \"aae46d99-cc6d-4be9-868b-8c4dda15a7a7\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-fcn5x"
Oct 31 12:43:08.393312 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:08.393239    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4wlk\" (UniqueName: \"kubernetes.io/projected/15a47e5b-2ade-4ca3-b0d0-6e4c09917588-kube-api-access-t4wlk\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-rr2hg\" (UID: \"15a47e5b-2ade-4ca3-b0d0-6e4c09917588\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-rr2hg"
Oct 31 12:43:08.393422 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:08.393313    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15a47e5b-2ade-4ca3-b0d0-6e4c09917588-istio-data\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-rr2hg\" (UID: \"15a47e5b-2ade-4ca3-b0d0-6e4c09917588\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-rr2hg"
Oct 31 12:43:08.393422 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:08.393379    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aae46d99-cc6d-4be9-868b-8c4dda15a7a7-workload-socket\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-fcn5x\" (UID: \"aae46d99-cc6d-4be9-868b-8c4dda15a7a7\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-fcn5x"
Oct 31 12:43:08.393565 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:08.393421    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15a47e5b-2ade-4ca3-b0d0-6e4c09917588-istiod-ca-cert\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-rr2hg\" (UID: \"15a47e5b-2ade-4ca3-b0d0-6e4c09917588\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-rr2hg"
Oct 31 12:43:08.393565 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:08.393467    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aae46d99-cc6d-4be9-868b-8c4dda15a7a7-credential-socket\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-fcn5x\" (UID: \"aae46d99-cc6d-4be9-868b-8c4dda15a7a7\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-fcn5x"
Oct 31 12:43:08.393565 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:08.393505    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aae46d99-cc6d-4be9-868b-8c4dda15a7a7-istio-token\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-fcn5x\" (UID: \"aae46d99-cc6d-4be9-868b-8c4dda15a7a7\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-fcn5x"
Oct 31 12:43:08.393565 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:08.393551    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15a47e5b-2ade-4ca3-b0d0-6e4c09917588-istio-token\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-rr2hg\" (UID: \"15a47e5b-2ade-4ca3-b0d0-6e4c09917588\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-rr2hg"
Oct 31 12:43:08.393787 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:08.393623    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15a47e5b-2ade-4ca3-b0d0-6e4c09917588-workload-socket\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-rr2hg\" (UID: \"15a47e5b-2ade-4ca3-b0d0-6e4c09917588\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-rr2hg"
Oct 31 12:43:08.393787 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:08.393671    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prvgm\" (UniqueName: \"kubernetes.io/projected/aae46d99-cc6d-4be9-868b-8c4dda15a7a7-kube-api-access-prvgm\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-fcn5x\" (UID: \"aae46d99-cc6d-4be9-868b-8c4dda15a7a7\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-fcn5x"
Oct 31 12:43:08.393787 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:08.393735    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aae46d99-cc6d-4be9-868b-8c4dda15a7a7-workload-certs\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-fcn5x\" (UID: \"aae46d99-cc6d-4be9-868b-8c4dda15a7a7\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-fcn5x"
Oct 31 12:43:08.393787 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:08.393763    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15a47e5b-2ade-4ca3-b0d0-6e4c09917588-credential-socket\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-rr2hg\" (UID: \"15a47e5b-2ade-4ca3-b0d0-6e4c09917588\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-rr2hg"
Oct 31 12:43:08.394179 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:08.394155    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15a47e5b-2ade-4ca3-b0d0-6e4c09917588-credential-socket\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-rr2hg\" (UID: \"15a47e5b-2ade-4ca3-b0d0-6e4c09917588\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-rr2hg"
Oct 31 12:43:08.394579 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:08.394546    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15a47e5b-2ade-4ca3-b0d0-6e4c09917588-workload-certs\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-rr2hg\" (UID: \"15a47e5b-2ade-4ca3-b0d0-6e4c09917588\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-rr2hg"
Oct 31 12:43:08.395108 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:08.394929    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15a47e5b-2ade-4ca3-b0d0-6e4c09917588-workload-socket\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-rr2hg\" (UID: \"15a47e5b-2ade-4ca3-b0d0-6e4c09917588\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-rr2hg"
Oct 31 12:43:08.397088 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:08.395692    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15a47e5b-2ade-4ca3-b0d0-6e4c09917588-istio-data\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-rr2hg\" (UID: \"15a47e5b-2ade-4ca3-b0d0-6e4c09917588\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-rr2hg"
Oct 31 12:43:08.397088 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:08.395872    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15a47e5b-2ade-4ca3-b0d0-6e4c09917588-istio-envoy\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-rr2hg\" (UID: \"15a47e5b-2ade-4ca3-b0d0-6e4c09917588\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-rr2hg"
Oct 31 12:43:08.397088 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:08.396069    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15a47e5b-2ade-4ca3-b0d0-6e4c09917588-istio-podinfo\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-rr2hg\" (UID: \"15a47e5b-2ade-4ca3-b0d0-6e4c09917588\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-rr2hg"
Oct 31 12:43:08.397088 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:08.396146    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aae46d99-cc6d-4be9-868b-8c4dda15a7a7-workload-socket\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-fcn5x\" (UID: \"aae46d99-cc6d-4be9-868b-8c4dda15a7a7\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-fcn5x"
Oct 31 12:43:08.397088 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:08.396408    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aae46d99-cc6d-4be9-868b-8c4dda15a7a7-istiod-ca-cert\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-fcn5x\" (UID: \"aae46d99-cc6d-4be9-868b-8c4dda15a7a7\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-fcn5x"
Oct 31 12:43:08.397088 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:08.396650    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aae46d99-cc6d-4be9-868b-8c4dda15a7a7-workload-certs\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-fcn5x\" (UID: \"aae46d99-cc6d-4be9-868b-8c4dda15a7a7\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-fcn5x"
Oct 31 12:43:08.397514 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:08.397100    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15a47e5b-2ade-4ca3-b0d0-6e4c09917588-istiod-ca-cert\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-rr2hg\" (UID: \"15a47e5b-2ade-4ca3-b0d0-6e4c09917588\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-rr2hg"
Oct 31 12:43:08.397514 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:08.397407    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aae46d99-cc6d-4be9-868b-8c4dda15a7a7-istio-data\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-fcn5x\" (UID: \"aae46d99-cc6d-4be9-868b-8c4dda15a7a7\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-fcn5x"
Oct 31 12:43:08.397671 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:08.397650    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aae46d99-cc6d-4be9-868b-8c4dda15a7a7-istio-envoy\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-fcn5x\" (UID: \"aae46d99-cc6d-4be9-868b-8c4dda15a7a7\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-fcn5x"
Oct 31 12:43:08.397725 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:08.397708    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aae46d99-cc6d-4be9-868b-8c4dda15a7a7-credential-socket\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-fcn5x\" (UID: \"aae46d99-cc6d-4be9-868b-8c4dda15a7a7\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-fcn5x"
Oct 31 12:43:08.397854 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:08.397837    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aae46d99-cc6d-4be9-868b-8c4dda15a7a7-istio-podinfo\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-fcn5x\" (UID: \"aae46d99-cc6d-4be9-868b-8c4dda15a7a7\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-fcn5x"
Oct 31 12:43:08.409030 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:08.409007    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4wlk\" (UniqueName: \"kubernetes.io/projected/15a47e5b-2ade-4ca3-b0d0-6e4c09917588-kube-api-access-t4wlk\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-rr2hg\" (UID: \"15a47e5b-2ade-4ca3-b0d0-6e4c09917588\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-rr2hg"
Oct 31 12:43:08.409171 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:08.409147    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15a47e5b-2ade-4ca3-b0d0-6e4c09917588-istio-token\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-rr2hg\" (UID: \"15a47e5b-2ade-4ca3-b0d0-6e4c09917588\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-rr2hg"
Oct 31 12:43:08.409343 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:08.409327    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aae46d99-cc6d-4be9-868b-8c4dda15a7a7-istio-token\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-fcn5x\" (UID: \"aae46d99-cc6d-4be9-868b-8c4dda15a7a7\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-fcn5x"
Oct 31 12:43:08.409409 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:08.409387    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-prvgm\" (UniqueName: \"kubernetes.io/projected/aae46d99-cc6d-4be9-868b-8c4dda15a7a7-kube-api-access-prvgm\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-fcn5x\" (UID: \"aae46d99-cc6d-4be9-868b-8c4dda15a7a7\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-fcn5x"
Oct 31 12:43:08.533863 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:08.533828    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-rr2hg"
Oct 31 12:43:08.570936 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:08.570895    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-fcn5x"
Oct 31 12:43:09.083920 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:09.083828    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:43:09.084312 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:09.084008    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:43:09.084312 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:09.084098    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:43:09.097390 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:09.097345    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:43:09.097528 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:09.097483    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:43:09.097610 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:09.097587    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:43:09.100990 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:09.100964    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-rr2hg"]
Oct 31 12:43:09.101151 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:09.101133    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-fcn5x"]
Oct 31 12:43:09.338624 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:09.338523    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-43c7f8da-gw-openshift-default-69d4944d9c-mxmg5"]
Oct 31 12:43:09.348514 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:09.345563    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-43c7f8da-gw-openshift-default-69d4944d9c-mxmg5"
Oct 31 12:43:09.391574 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:09.376815    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-43c7f8da-gw-openshift-default-dockercfg-wlndr\""
Oct 31 12:43:09.450751 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:09.442715    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77382c0a-8546-4b30-bbc6-e21a77328be8-workload-socket\") pod \"e2e-migrate-43c7f8da-gw-openshift-default-69d4944d9c-mxmg5\" (UID: \"77382c0a-8546-4b30-bbc6-e21a77328be8\") " pod="openshift-ingress/e2e-migrate-43c7f8da-gw-openshift-default-69d4944d9c-mxmg5"
Oct 31 12:43:09.450751 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:09.442764    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77382c0a-8546-4b30-bbc6-e21a77328be8-istio-podinfo\") pod \"e2e-migrate-43c7f8da-gw-openshift-default-69d4944d9c-mxmg5\" (UID: \"77382c0a-8546-4b30-bbc6-e21a77328be8\") " pod="openshift-ingress/e2e-migrate-43c7f8da-gw-openshift-default-69d4944d9c-mxmg5"
Oct 31 12:43:09.450751 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:09.442866    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77382c0a-8546-4b30-bbc6-e21a77328be8-istio-token\") pod \"e2e-migrate-43c7f8da-gw-openshift-default-69d4944d9c-mxmg5\" (UID: \"77382c0a-8546-4b30-bbc6-e21a77328be8\") " pod="openshift-ingress/e2e-migrate-43c7f8da-gw-openshift-default-69d4944d9c-mxmg5"
Oct 31 12:43:09.450751 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:09.442901    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77382c0a-8546-4b30-bbc6-e21a77328be8-istio-envoy\") pod \"e2e-migrate-43c7f8da-gw-openshift-default-69d4944d9c-mxmg5\" (UID: \"77382c0a-8546-4b30-bbc6-e21a77328be8\") " pod="openshift-ingress/e2e-migrate-43c7f8da-gw-openshift-default-69d4944d9c-mxmg5"
Oct 31 12:43:09.450751 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:09.443040    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bsnkz\" (UniqueName: \"kubernetes.io/projected/77382c0a-8546-4b30-bbc6-e21a77328be8-kube-api-access-bsnkz\") pod \"e2e-migrate-43c7f8da-gw-openshift-default-69d4944d9c-mxmg5\" (UID: \"77382c0a-8546-4b30-bbc6-e21a77328be8\") " pod="openshift-ingress/e2e-migrate-43c7f8da-gw-openshift-default-69d4944d9c-mxmg5"
Oct 31 12:43:09.450751 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:09.443086    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77382c0a-8546-4b30-bbc6-e21a77328be8-istiod-ca-cert\") pod \"e2e-migrate-43c7f8da-gw-openshift-default-69d4944d9c-mxmg5\" (UID: \"77382c0a-8546-4b30-bbc6-e21a77328be8\") " pod="openshift-ingress/e2e-migrate-43c7f8da-gw-openshift-default-69d4944d9c-mxmg5"
Oct 31 12:43:09.450751 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:09.443304    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77382c0a-8546-4b30-bbc6-e21a77328be8-istio-data\") pod \"e2e-migrate-43c7f8da-gw-openshift-default-69d4944d9c-mxmg5\" (UID: \"77382c0a-8546-4b30-bbc6-e21a77328be8\") " pod="openshift-ingress/e2e-migrate-43c7f8da-gw-openshift-default-69d4944d9c-mxmg5"
Oct 31 12:43:09.450751 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:09.443362    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77382c0a-8546-4b30-bbc6-e21a77328be8-credential-socket\") pod \"e2e-migrate-43c7f8da-gw-openshift-default-69d4944d9c-mxmg5\" (UID: \"77382c0a-8546-4b30-bbc6-e21a77328be8\") " pod="openshift-ingress/e2e-migrate-43c7f8da-gw-openshift-default-69d4944d9c-mxmg5"
Oct 31 12:43:09.450751 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:09.443443    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77382c0a-8546-4b30-bbc6-e21a77328be8-workload-certs\") pod \"e2e-migrate-43c7f8da-gw-openshift-default-69d4944d9c-mxmg5\" (UID: \"77382c0a-8546-4b30-bbc6-e21a77328be8\") " pod="openshift-ingress/e2e-migrate-43c7f8da-gw-openshift-default-69d4944d9c-mxmg5"
Oct 31 12:43:09.607562 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:09.605582    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77382c0a-8546-4b30-bbc6-e21a77328be8-istio-token\") pod \"e2e-migrate-43c7f8da-gw-openshift-default-69d4944d9c-mxmg5\" (UID: \"77382c0a-8546-4b30-bbc6-e21a77328be8\") " pod="openshift-ingress/e2e-migrate-43c7f8da-gw-openshift-default-69d4944d9c-mxmg5"
Oct 31 12:43:09.607562 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:09.605664    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77382c0a-8546-4b30-bbc6-e21a77328be8-istio-envoy\") pod \"e2e-migrate-43c7f8da-gw-openshift-default-69d4944d9c-mxmg5\" (UID: \"77382c0a-8546-4b30-bbc6-e21a77328be8\") " pod="openshift-ingress/e2e-migrate-43c7f8da-gw-openshift-default-69d4944d9c-mxmg5"
Oct 31 12:43:09.607562 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:09.605792    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsnkz\" (UniqueName: \"kubernetes.io/projected/77382c0a-8546-4b30-bbc6-e21a77328be8-kube-api-access-bsnkz\") pod \"e2e-migrate-43c7f8da-gw-openshift-default-69d4944d9c-mxmg5\" (UID: \"77382c0a-8546-4b30-bbc6-e21a77328be8\") " pod="openshift-ingress/e2e-migrate-43c7f8da-gw-openshift-default-69d4944d9c-mxmg5"
Oct 31 12:43:09.607562 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:09.605834    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77382c0a-8546-4b30-bbc6-e21a77328be8-istiod-ca-cert\") pod \"e2e-migrate-43c7f8da-gw-openshift-default-69d4944d9c-mxmg5\" (UID: \"77382c0a-8546-4b30-bbc6-e21a77328be8\") " pod="openshift-ingress/e2e-migrate-43c7f8da-gw-openshift-default-69d4944d9c-mxmg5"
Oct 31 12:43:09.607562 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:09.606019    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77382c0a-8546-4b30-bbc6-e21a77328be8-istio-data\") pod \"e2e-migrate-43c7f8da-gw-openshift-default-69d4944d9c-mxmg5\" (UID: \"77382c0a-8546-4b30-bbc6-e21a77328be8\") " pod="openshift-ingress/e2e-migrate-43c7f8da-gw-openshift-default-69d4944d9c-mxmg5"
Oct 31 12:43:09.607562 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:09.606067    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77382c0a-8546-4b30-bbc6-e21a77328be8-credential-socket\") pod \"e2e-migrate-43c7f8da-gw-openshift-default-69d4944d9c-mxmg5\" (UID: \"77382c0a-8546-4b30-bbc6-e21a77328be8\") " pod="openshift-ingress/e2e-migrate-43c7f8da-gw-openshift-default-69d4944d9c-mxmg5"
Oct 31 12:43:09.607562 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:09.606140    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77382c0a-8546-4b30-bbc6-e21a77328be8-workload-certs\") pod \"e2e-migrate-43c7f8da-gw-openshift-default-69d4944d9c-mxmg5\" (UID: \"77382c0a-8546-4b30-bbc6-e21a77328be8\") " pod="openshift-ingress/e2e-migrate-43c7f8da-gw-openshift-default-69d4944d9c-mxmg5"
Oct 31 12:43:09.607562 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:09.606215    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77382c0a-8546-4b30-bbc6-e21a77328be8-workload-socket\") pod \"e2e-migrate-43c7f8da-gw-openshift-default-69d4944d9c-mxmg5\" (UID: \"77382c0a-8546-4b30-bbc6-e21a77328be8\") " pod="openshift-ingress/e2e-migrate-43c7f8da-gw-openshift-default-69d4944d9c-mxmg5"
Oct 31 12:43:09.607562 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:09.606246    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77382c0a-8546-4b30-bbc6-e21a77328be8-istio-podinfo\") pod \"e2e-migrate-43c7f8da-gw-openshift-default-69d4944d9c-mxmg5\" (UID: \"77382c0a-8546-4b30-bbc6-e21a77328be8\") " pod="openshift-ingress/e2e-migrate-43c7f8da-gw-openshift-default-69d4944d9c-mxmg5"
Oct 31 12:43:09.668520 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:09.623483    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77382c0a-8546-4b30-bbc6-e21a77328be8-istiod-ca-cert\") pod \"e2e-migrate-43c7f8da-gw-openshift-default-69d4944d9c-mxmg5\" (UID: \"77382c0a-8546-4b30-bbc6-e21a77328be8\") " pod="openshift-ingress/e2e-migrate-43c7f8da-gw-openshift-default-69d4944d9c-mxmg5"
Oct 31 12:43:09.668520 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:09.647824    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77382c0a-8546-4b30-bbc6-e21a77328be8-istio-envoy\") pod \"e2e-migrate-43c7f8da-gw-openshift-default-69d4944d9c-mxmg5\" (UID: \"77382c0a-8546-4b30-bbc6-e21a77328be8\") " pod="openshift-ingress/e2e-migrate-43c7f8da-gw-openshift-default-69d4944d9c-mxmg5"
Oct 31 12:43:09.668520 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:09.648432    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77382c0a-8546-4b30-bbc6-e21a77328be8-credential-socket\") pod \"e2e-migrate-43c7f8da-gw-openshift-default-69d4944d9c-mxmg5\" (UID: \"77382c0a-8546-4b30-bbc6-e21a77328be8\") " pod="openshift-ingress/e2e-migrate-43c7f8da-gw-openshift-default-69d4944d9c-mxmg5"
Oct 31 12:43:09.668520 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:09.648710    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77382c0a-8546-4b30-bbc6-e21a77328be8-istio-data\") pod \"e2e-migrate-43c7f8da-gw-openshift-default-69d4944d9c-mxmg5\" (UID: \"77382c0a-8546-4b30-bbc6-e21a77328be8\") " pod="openshift-ingress/e2e-migrate-43c7f8da-gw-openshift-default-69d4944d9c-mxmg5"
Oct 31 12:43:09.668520 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:09.648962    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77382c0a-8546-4b30-bbc6-e21a77328be8-workload-certs\") pod \"e2e-migrate-43c7f8da-gw-openshift-default-69d4944d9c-mxmg5\" (UID: \"77382c0a-8546-4b30-bbc6-e21a77328be8\") " pod="openshift-ingress/e2e-migrate-43c7f8da-gw-openshift-default-69d4944d9c-mxmg5"
Oct 31 12:43:09.668520 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:09.649187    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77382c0a-8546-4b30-bbc6-e21a77328be8-workload-socket\") pod \"e2e-migrate-43c7f8da-gw-openshift-default-69d4944d9c-mxmg5\" (UID: \"77382c0a-8546-4b30-bbc6-e21a77328be8\") " pod="openshift-ingress/e2e-migrate-43c7f8da-gw-openshift-default-69d4944d9c-mxmg5"
Oct 31 12:43:09.668520 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:09.667597    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-43c7f8da-gw-openshift-default-69d4944d9c-mxmg5"]
Oct 31 12:43:09.753136 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:09.753061    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bsnkz\" (UniqueName: \"kubernetes.io/projected/77382c0a-8546-4b30-bbc6-e21a77328be8-kube-api-access-bsnkz\") pod \"e2e-migrate-43c7f8da-gw-openshift-default-69d4944d9c-mxmg5\" (UID: \"77382c0a-8546-4b30-bbc6-e21a77328be8\") " pod="openshift-ingress/e2e-migrate-43c7f8da-gw-openshift-default-69d4944d9c-mxmg5"
Oct 31 12:43:09.753820 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:09.753796    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77382c0a-8546-4b30-bbc6-e21a77328be8-istio-podinfo\") pod \"e2e-migrate-43c7f8da-gw-openshift-default-69d4944d9c-mxmg5\" (UID: \"77382c0a-8546-4b30-bbc6-e21a77328be8\") " pod="openshift-ingress/e2e-migrate-43c7f8da-gw-openshift-default-69d4944d9c-mxmg5"
Oct 31 12:43:09.755249 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:09.754889    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6c73a95-bcsgz"
Oct 31 12:43:09.762745 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:09.762697    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77382c0a-8546-4b30-bbc6-e21a77328be8-istio-token\") pod \"e2e-migrate-43c7f8da-gw-openshift-default-69d4944d9c-mxmg5\" (UID: \"77382c0a-8546-4b30-bbc6-e21a77328be8\") " pod="openshift-ingress/e2e-migrate-43c7f8da-gw-openshift-default-69d4944d9c-mxmg5"
Oct 31 12:43:09.830122 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:09.830014    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e86e368c-5055-4433-abb6-f1171b74c21d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e86e368c-5055-4433-abb6-f1171b74c21d-maas-api-xks-ca\") pod \"e86e368c-5055-4433-abb6-f1171b74c21d\" (UID: \"e86e368c-5055-4433-abb6-f1171b74c21d\") "
Oct 31 12:43:09.830309 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:09.830188    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e86e368c-5055-4433-abb6-f1171b74c21d-kube-api-access-w7zzk\" (UniqueName: \"kubernetes.io/projected/e86e368c-5055-4433-abb6-f1171b74c21d-kube-api-access-w7zzk\") pod \"e86e368c-5055-4433-abb6-f1171b74c21d\" (UID: \"e86e368c-5055-4433-abb6-f1171b74c21d\") "
Oct 31 12:43:09.830833 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:09.830417    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e86e368c-5055-4433-abb6-f1171b74c21d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e86e368c-5055-4433-abb6-f1171b74c21d-maas-api-openshift-ca\") pod \"e86e368c-5055-4433-abb6-f1171b74c21d\" (UID: \"e86e368c-5055-4433-abb6-f1171b74c21d\") "
Oct 31 12:43:09.831392 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:09.831345    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e86e368c-5055-4433-abb6-f1171b74c21d-maas-api-openshift-ca" pod "e86e368c-5055-4433-abb6-f1171b74c21d" (UID: "e86e368c-5055-4433-abb6-f1171b74c21d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:43:09.843389 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:09.840708    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e86e368c-5055-4433-abb6-f1171b74c21d-maas-api-xks-ca" pod "e86e368c-5055-4433-abb6-f1171b74c21d" (UID: "e86e368c-5055-4433-abb6-f1171b74c21d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:43:09.843389 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:09.842745    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e86e368c-5055-4433-abb6-f1171b74c21d-kube-api-access-w7zzk" pod "e86e368c-5055-4433-abb6-f1171b74c21d" (UID: "e86e368c-5055-4433-abb6-f1171b74c21d"). InnerVolumeSpecName "kube-api-access-w7zzk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:43:09.932755 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:09.932679    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e86e368c-5055-4433-abb6-f1171b74c21d-maas-api-xks-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:43:09.932755 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:09.932711    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7zzk\" (UniqueName: \"kubernetes.io/projected/e86e368c-5055-4433-abb6-f1171b74c21d-kube-api-access-w7zzk\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:43:09.932755 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:09.932721    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e86e368c-5055-4433-abb6-f1171b74c21d-maas-api-openshift-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:43:10.047461 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:10.047425    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-43c7f8da-gw-openshift-default-69d4944d9c-mxmg5"
Oct 31 12:43:10.142986 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:10.142943    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-fcn5x" event={"ID":"aae46d99-cc6d-4be9-868b-8c4dda15a7a7","Type":"ContainerStarted","Data":"147a513a685d7a0c89a75e10e6b013e2bf878ffdfedb1103a44b4588a8e07e27"}
Oct 31 12:43:10.143652 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:10.143019    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-fcn5x" event={"ID":"aae46d99-cc6d-4be9-868b-8c4dda15a7a7","Type":"ContainerStarted","Data":"35c5fcef0e641246cde02165fdfdff980cc110c1fff1018e2e07767b548d35a9"}
Oct 31 12:43:10.150346 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:10.148916    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6c73a95-bcsgz"
Oct 31 12:43:10.150346 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:10.148974    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6c73a95-bcsgz" event={"ID":"e86e368c-5055-4433-abb6-f1171b74c21d","Type":"ContainerDied","Data":"14e5b56d62378dbc3f4afdba11ef8d9cba6c909e4fa9cfee620426fe86475ccd"}
Oct 31 12:43:10.150346 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:10.149027    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="14e5b56d62378dbc3f4afdba11ef8d9cba6c909e4fa9cfee620426fe86475ccd"
Oct 31 12:43:10.151315 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:10.151280    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-rr2hg" event={"ID":"15a47e5b-2ade-4ca3-b0d0-6e4c09917588","Type":"ContainerStarted","Data":"2bb4f5e3c1b84da3d002959b676fa2bd230119ff2186a6007a5553bcc7cc4af6"}
Oct 31 12:43:10.151434 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:10.151335    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-rr2hg" event={"ID":"15a47e5b-2ade-4ca3-b0d0-6e4c09917588","Type":"ContainerStarted","Data":"ef5a4210fb1b20db23e6f2a9f7c16813f879a1544bf1caa39387291627fc7172"}
Oct 31 12:43:10.180865 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:10.180815    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-fcn5x" podStartSLOduration=2.18079871 podStartE2EDuration="2.18079871s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:43:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:43:10.171337302 +0000 UTC m=+953.274405014" watchObservedRunningTime="2026-10-31 12:43:10.18079871 +0000 UTC m=+953.283866421"
Oct 31 12:43:10.226722 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:10.224990    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-rr2hg" podStartSLOduration=2.224968305 podStartE2EDuration="2.224968305s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:43:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:43:10.20539784 +0000 UTC m=+953.308465552" watchObservedRunningTime="2026-10-31 12:43:10.224968305 +0000 UTC m=+953.328036020"
Oct 31 12:43:10.534372 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:10.534136    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-rr2hg"
Oct 31 12:43:10.539005 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:10.535466    5060 patch_prober.go:29] interesting pod/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-rr2hg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body=
Oct 31 12:43:10.539005 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:10.535543    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-rr2hg" podUID="15a47e5b-2ade-4ca3-b0d0-6e4c09917588" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused"
Oct 31 12:43:10.539005 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:10.538059    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-43c7f8da-gw-openshift-default-69d4944d9c-mxmg5"]
Oct 31 12:43:10.545607 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:10.545571    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:43:10.545823 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:10.545800    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:43:10.545968 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:10.545948    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:43:10.592519 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:10.584834    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-fcn5x"
Oct 31 12:43:10.620465 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:10.619730    5060 patch_prober.go:29] interesting pod/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-fcn5x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body=
Oct 31 12:43:10.620465 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:10.619790    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-fcn5x" podUID="aae46d99-cc6d-4be9-868b-8c4dda15a7a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused"
Oct 31 12:43:11.228479 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:11.228387    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-43c7f8da-gw-openshift-default-69d4944d9c-mxmg5" event={"ID":"77382c0a-8546-4b30-bbc6-e21a77328be8","Type":"ContainerStarted","Data":"613fc368611ab8081f078d3777d92a99d93ea6301fb0bcd19a3baf52f1b22cc9"}
Oct 31 12:43:11.228479 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:11.228445    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-43c7f8da-gw-openshift-default-69d4944d9c-mxmg5" event={"ID":"77382c0a-8546-4b30-bbc6-e21a77328be8","Type":"ContainerStarted","Data":"92d0a54a15c14ec846ade8db47e50b92b59e99ae7a2e870dc88ab7500913e0b3"}
Oct 31 12:43:11.294522 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:11.289362    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-43c7f8da-gw-openshift-default-69d4944d9c-mxmg5" podStartSLOduration=2.28933973 podStartE2EDuration="2.28933973s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:43:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:43:11.254428938 +0000 UTC m=+954.357496649" watchObservedRunningTime="2026-10-31 12:43:11.28933973 +0000 UTC m=+954.392407443"
Oct 31 12:43:11.568562 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:11.568525    5060 patch_prober.go:29] interesting pod/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-rr2hg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body=
Oct 31 12:43:11.568833 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:11.568800    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-rr2hg" podUID="15a47e5b-2ade-4ca3-b0d0-6e4c09917588" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused"
Oct 31 12:43:11.596813 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:11.596775    5060 patch_prober.go:29] interesting pod/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-fcn5x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body=
Oct 31 12:43:11.597047 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:11.597023    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-fcn5x" podUID="aae46d99-cc6d-4be9-868b-8c4dda15a7a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused"
Oct 31 12:43:12.077567 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:12.076792    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-43c7f8da-gw-openshift-default-69d4944d9c-mxmg5"
Oct 31 12:43:12.130813 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:12.130414    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-43c7f8da-gw-openshift-default-69d4944d9c-mxmg5"
Oct 31 12:43:12.242153 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:12.241326    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-43c7f8da-gw-openshift-default-69d4944d9c-mxmg5"
Oct 31 12:43:12.242584 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:12.242443    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-43c7f8da-gw-openshift-default-69d4944d9c-mxmg5"
Oct 31 12:43:12.572520 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:12.571645    5060 patch_prober.go:29] interesting pod/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-fcn5x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body=
Oct 31 12:43:12.572520 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:12.571728    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-fcn5x" podUID="aae46d99-cc6d-4be9-868b-8c4dda15a7a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused"
Oct 31 12:43:12.613563 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:12.598553    5060 patch_prober.go:29] interesting pod/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-rr2hg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body=
Oct 31 12:43:12.613563 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:12.598637    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-rr2hg" podUID="15a47e5b-2ade-4ca3-b0d0-6e4c09917588" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused"
Oct 31 12:43:13.021827 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:13.021741    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d250eff7-d4jg7"]
Oct 31 12:43:13.022637 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:13.022620    5060 cpu_manager.go:414] "containerMap: removing container" podUID="e86e368c-5055-4433-abb6-f1171b74c21d" containerName="revoke-keys"
Oct 31 12:43:13.022693 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:13.022639    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e86e368c-5055-4433-abb6-f1171b74c21d" containerName="revoke-keys"
Oct 31 12:43:13.022803 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:13.022794    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="e86e368c-5055-4433-abb6-f1171b74c21d" containerName="revoke-keys"
Oct 31 12:43:13.027207 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:13.027152    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d250eff7-d4jg7"
Oct 31 12:43:13.031916 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:13.031888    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 12:43:13.032910 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:13.032689    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7wsmb\""
Oct 31 12:43:13.032910 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:13.032719    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 12:43:13.032910 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:13.032848    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 12:43:13.040228 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:13.040204    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d250eff7-d4jg7"]
Oct 31 12:43:13.153557 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:13.153518    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9baea4fa-5a62-48d0-b065-8167e31b8236-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d250eff7-d4jg7\" (UID: \"9baea4fa-5a62-48d0-b065-8167e31b8236\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d250eff7-d4jg7"
Oct 31 12:43:13.153738 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:13.153688    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9baea4fa-5a62-48d0-b065-8167e31b8236-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d250eff7-d4jg7\" (UID: \"9baea4fa-5a62-48d0-b065-8167e31b8236\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d250eff7-d4jg7"
Oct 31 12:43:13.153790 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:13.153750    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xj58r\" (UniqueName: \"kubernetes.io/projected/9baea4fa-5a62-48d0-b065-8167e31b8236-kube-api-access-xj58r\") pod \"maas-api-revoke-sub-d250eff7-d4jg7\" (UID: \"9baea4fa-5a62-48d0-b065-8167e31b8236\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d250eff7-d4jg7"
Oct 31 12:43:13.282563 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:13.255745    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xj58r\" (UniqueName: \"kubernetes.io/projected/9baea4fa-5a62-48d0-b065-8167e31b8236-kube-api-access-xj58r\") pod \"maas-api-revoke-sub-d250eff7-d4jg7\" (UID: \"9baea4fa-5a62-48d0-b065-8167e31b8236\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d250eff7-d4jg7"
Oct 31 12:43:13.282563 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:13.255962    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9baea4fa-5a62-48d0-b065-8167e31b8236-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d250eff7-d4jg7\" (UID: \"9baea4fa-5a62-48d0-b065-8167e31b8236\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d250eff7-d4jg7"
Oct 31 12:43:13.282563 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:13.256191    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9baea4fa-5a62-48d0-b065-8167e31b8236-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d250eff7-d4jg7\" (UID: \"9baea4fa-5a62-48d0-b065-8167e31b8236\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d250eff7-d4jg7"
Oct 31 12:43:13.284279 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:13.284245    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9baea4fa-5a62-48d0-b065-8167e31b8236-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d250eff7-d4jg7\" (UID: \"9baea4fa-5a62-48d0-b065-8167e31b8236\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d250eff7-d4jg7"
Oct 31 12:43:13.355903 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:13.335283    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xj58r\" (UniqueName: \"kubernetes.io/projected/9baea4fa-5a62-48d0-b065-8167e31b8236-kube-api-access-xj58r\") pod \"maas-api-revoke-sub-d250eff7-d4jg7\" (UID: \"9baea4fa-5a62-48d0-b065-8167e31b8236\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d250eff7-d4jg7"
Oct 31 12:43:13.436264 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:13.417331    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9baea4fa-5a62-48d0-b065-8167e31b8236-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d250eff7-d4jg7\" (UID: \"9baea4fa-5a62-48d0-b065-8167e31b8236\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d250eff7-d4jg7"
Oct 31 12:43:13.535605 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:13.535191    5060 patch_prober.go:29] interesting pod/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-rr2hg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body=
Oct 31 12:43:13.535605 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:13.535241    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-rr2hg" podUID="15a47e5b-2ade-4ca3-b0d0-6e4c09917588" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused"
Oct 31 12:43:13.576132 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:13.575819    5060 patch_prober.go:29] interesting pod/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-fcn5x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body=
Oct 31 12:43:13.576132 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:13.575897    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-fcn5x" podUID="aae46d99-cc6d-4be9-868b-8c4dda15a7a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused"
Oct 31 12:43:13.646610 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:13.646575    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d250eff7-d4jg7"
Oct 31 12:43:14.588521 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:14.575540    5060 patch_prober.go:29] interesting pod/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-fcn5x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body=
Oct 31 12:43:14.588521 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:14.575601    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-fcn5x" podUID="aae46d99-cc6d-4be9-868b-8c4dda15a7a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused"
Oct 31 12:43:14.600806 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:14.600766    5060 patch_prober.go:29] interesting pod/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-rr2hg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body=
Oct 31 12:43:14.601100 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:14.601067    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-rr2hg" podUID="15a47e5b-2ade-4ca3-b0d0-6e4c09917588" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused"
Oct 31 12:43:14.675124 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:14.657678    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d250eff7-d4jg7"]
Oct 31 12:43:15.478074 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:15.478014    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d250eff7-d4jg7" event={"ID":"9baea4fa-5a62-48d0-b065-8167e31b8236","Type":"ContainerStarted","Data":"89167f849aa1f4e33d0d71d5f4f97ddf16f9f60e520c0edb4c9b4e67cc5f8a2d"}
Oct 31 12:43:15.534405 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:15.534369    5060 patch_prober.go:29] interesting pod/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-rr2hg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body=
Oct 31 12:43:15.534572 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:15.534425    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-rr2hg" podUID="15a47e5b-2ade-4ca3-b0d0-6e4c09917588" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused"
Oct 31 12:43:15.610251 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:15.609794    5060 patch_prober.go:29] interesting pod/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-fcn5x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body=
Oct 31 12:43:15.610251 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:15.609858    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-fcn5x" podUID="aae46d99-cc6d-4be9-868b-8c4dda15a7a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused"
Oct 31 12:43:16.351759 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:16.351727    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-59fd8a28-jtp8n"]
Oct 31 12:43:16.384709 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:16.384674    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-59fd8a28-jtp8n"
Oct 31 12:43:16.430385 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:16.430354    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-59fd8a28-jtp8n"]
Oct 31 12:43:16.439277 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:16.439242    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/335aa917-cbfc-4e05-8075-38e3a2f85155-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-59fd8a28-jtp8n\" (UID: \"335aa917-cbfc-4e05-8075-38e3a2f85155\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-59fd8a28-jtp8n"
Oct 31 12:43:16.439720 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:16.439699    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fzd8\" (UniqueName: \"kubernetes.io/projected/335aa917-cbfc-4e05-8075-38e3a2f85155-kube-api-access-9fzd8\") pod \"maas-api-revoke-keys-e2e-ait-59fd8a28-jtp8n\" (UID: \"335aa917-cbfc-4e05-8075-38e3a2f85155\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-59fd8a28-jtp8n"
Oct 31 12:43:16.439931 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:16.439914    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/335aa917-cbfc-4e05-8075-38e3a2f85155-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-59fd8a28-jtp8n\" (UID: \"335aa917-cbfc-4e05-8075-38e3a2f85155\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-59fd8a28-jtp8n"
Oct 31 12:43:16.534572 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:16.534484    5060 patch_prober.go:29] interesting pod/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-rr2hg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body=
Oct 31 12:43:16.534775 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:16.534613    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-rr2hg" podUID="15a47e5b-2ade-4ca3-b0d0-6e4c09917588" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused"
Oct 31 12:43:16.537615 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:16.537586    5060 generic.go:356] "Generic (PLEG): container finished" podID="9baea4fa-5a62-48d0-b065-8167e31b8236" containerID="fb9347a728d418c82cd8782dca3ac902277750b8975f91f859a27dff89dac0b7" exitCode=0
Oct 31 12:43:16.538269 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:16.538243    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d250eff7-d4jg7" event={"ID":"9baea4fa-5a62-48d0-b065-8167e31b8236","Type":"ContainerDied","Data":"fb9347a728d418c82cd8782dca3ac902277750b8975f91f859a27dff89dac0b7"}
Oct 31 12:43:16.541143 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:16.541119    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fzd8\" (UniqueName: \"kubernetes.io/projected/335aa917-cbfc-4e05-8075-38e3a2f85155-kube-api-access-9fzd8\") pod \"maas-api-revoke-keys-e2e-ait-59fd8a28-jtp8n\" (UID: \"335aa917-cbfc-4e05-8075-38e3a2f85155\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-59fd8a28-jtp8n"
Oct 31 12:43:16.541248 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:16.541230    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/335aa917-cbfc-4e05-8075-38e3a2f85155-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-59fd8a28-jtp8n\" (UID: \"335aa917-cbfc-4e05-8075-38e3a2f85155\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-59fd8a28-jtp8n"
Oct 31 12:43:16.541350 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:16.541334    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/335aa917-cbfc-4e05-8075-38e3a2f85155-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-59fd8a28-jtp8n\" (UID: \"335aa917-cbfc-4e05-8075-38e3a2f85155\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-59fd8a28-jtp8n"
Oct 31 12:43:16.542230 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:16.542208    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/335aa917-cbfc-4e05-8075-38e3a2f85155-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-59fd8a28-jtp8n\" (UID: \"335aa917-cbfc-4e05-8075-38e3a2f85155\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-59fd8a28-jtp8n"
Oct 31 12:43:16.544275 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:16.544247    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/335aa917-cbfc-4e05-8075-38e3a2f85155-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-59fd8a28-jtp8n\" (UID: \"335aa917-cbfc-4e05-8075-38e3a2f85155\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-59fd8a28-jtp8n"
Oct 31 12:43:16.553852 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:16.553834    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fzd8\" (UniqueName: \"kubernetes.io/projected/335aa917-cbfc-4e05-8075-38e3a2f85155-kube-api-access-9fzd8\") pod \"maas-api-revoke-keys-e2e-ait-59fd8a28-jtp8n\" (UID: \"335aa917-cbfc-4e05-8075-38e3a2f85155\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-59fd8a28-jtp8n"
Oct 31 12:43:16.616186 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:16.616141    5060 patch_prober.go:29] interesting pod/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-fcn5x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body=
Oct 31 12:43:16.616775 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:16.616202    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-fcn5x" podUID="aae46d99-cc6d-4be9-868b-8c4dda15a7a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused"
Oct 31 12:43:16.770138 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:16.764270    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-59fd8a28-jtp8n"
Oct 31 12:43:17.286225 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:43:17.285779    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod335aa917_cbfc_4e05_8075_38e3a2f85155.slice/crio-dc0f7975db39c9cf34b9b4421c0ed185c883c5f2265944106fa7fdbce1a249e2 WatchSource:0}: Error finding container dc0f7975db39c9cf34b9b4421c0ed185c883c5f2265944106fa7fdbce1a249e2: Status 404 returned error can't find the container with id dc0f7975db39c9cf34b9b4421c0ed185c883c5f2265944106fa7fdbce1a249e2
Oct 31 12:43:17.306343 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:17.306287    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-59fd8a28-jtp8n"]
Oct 31 12:43:17.534675 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:17.534584    5060 patch_prober.go:29] interesting pod/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-rr2hg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body=
Oct 31 12:43:17.534675 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:17.534651    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-rr2hg" podUID="15a47e5b-2ade-4ca3-b0d0-6e4c09917588" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused"
Oct 31 12:43:17.553916 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:17.553882    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-59fd8a28-jtp8n" event={"ID":"335aa917-cbfc-4e05-8075-38e3a2f85155","Type":"ContainerStarted","Data":"dc0f7975db39c9cf34b9b4421c0ed185c883c5f2265944106fa7fdbce1a249e2"}
Oct 31 12:43:17.571970 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:17.571935    5060 patch_prober.go:29] interesting pod/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-fcn5x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body=
Oct 31 12:43:17.572099 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:17.572002    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-fcn5x" podUID="aae46d99-cc6d-4be9-868b-8c4dda15a7a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused"
Oct 31 12:43:17.942251 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:17.942210    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d250eff7-d4jg7"
Oct 31 12:43:18.060363 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:18.060274    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9baea4fa-5a62-48d0-b065-8167e31b8236-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9baea4fa-5a62-48d0-b065-8167e31b8236-maas-api-xks-ca\") pod \"9baea4fa-5a62-48d0-b065-8167e31b8236\" (UID: \"9baea4fa-5a62-48d0-b065-8167e31b8236\") "
Oct 31 12:43:18.060655 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:18.060637    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9baea4fa-5a62-48d0-b065-8167e31b8236-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9baea4fa-5a62-48d0-b065-8167e31b8236-maas-api-openshift-ca\") pod \"9baea4fa-5a62-48d0-b065-8167e31b8236\" (UID: \"9baea4fa-5a62-48d0-b065-8167e31b8236\") "
Oct 31 12:43:18.060867 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:18.060852    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9baea4fa-5a62-48d0-b065-8167e31b8236-kube-api-access-xj58r\" (UniqueName: \"kubernetes.io/projected/9baea4fa-5a62-48d0-b065-8167e31b8236-kube-api-access-xj58r\") pod \"9baea4fa-5a62-48d0-b065-8167e31b8236\" (UID: \"9baea4fa-5a62-48d0-b065-8167e31b8236\") "
Oct 31 12:43:18.065047 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:18.065011    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9baea4fa-5a62-48d0-b065-8167e31b8236-maas-api-xks-ca" pod "9baea4fa-5a62-48d0-b065-8167e31b8236" (UID: "9baea4fa-5a62-48d0-b065-8167e31b8236"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:43:18.065355 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:18.065330    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9baea4fa-5a62-48d0-b065-8167e31b8236-maas-api-openshift-ca" pod "9baea4fa-5a62-48d0-b065-8167e31b8236" (UID: "9baea4fa-5a62-48d0-b065-8167e31b8236"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:43:18.068244 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:18.068195    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9baea4fa-5a62-48d0-b065-8167e31b8236-kube-api-access-xj58r" pod "9baea4fa-5a62-48d0-b065-8167e31b8236" (UID: "9baea4fa-5a62-48d0-b065-8167e31b8236"). InnerVolumeSpecName "kube-api-access-xj58r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:43:18.163746 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:18.163713    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9baea4fa-5a62-48d0-b065-8167e31b8236-maas-api-xks-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:43:18.163746 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:18.163744    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9baea4fa-5a62-48d0-b065-8167e31b8236-maas-api-openshift-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:43:18.163921 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:18.163755    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xj58r\" (UniqueName: \"kubernetes.io/projected/9baea4fa-5a62-48d0-b065-8167e31b8236-kube-api-access-xj58r\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:43:18.294926 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:18.294888    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-cr49v"]
Oct 31 12:43:18.310901 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:18.310787    5060 cpu_manager.go:414] "containerMap: removing container" podUID="9baea4fa-5a62-48d0-b065-8167e31b8236" containerName="revoke-keys"
Oct 31 12:43:18.311162 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:18.311126    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9baea4fa-5a62-48d0-b065-8167e31b8236" containerName="revoke-keys"
Oct 31 12:43:18.311821 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:18.311772    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="9baea4fa-5a62-48d0-b065-8167e31b8236" containerName="revoke-keys"
Oct 31 12:43:18.514515 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:18.513752    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-cr49v"]
Oct 31 12:43:18.514515 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:18.513891    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-cr49v"
Oct 31 12:43:18.541971 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:18.536820    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-ed25a690-openshift-default-dockercfg-wwhr7\""
Oct 31 12:43:18.558895 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:18.544711    5060 patch_prober.go:29] interesting pod/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-rr2hg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body=
Oct 31 12:43:18.867532 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:18.867478    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-rr2hg" podUID="15a47e5b-2ade-4ca3-b0d0-6e4c09917588" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused"
Oct 31 12:43:18.890608 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:18.770322    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f957e25b-5bf0-4259-afbd-fcd50a9364d5-istio-envoy\") pod \"e2e-mt-ed25a690-openshift-default-d54cbd5df-cr49v\" (UID: \"f957e25b-5bf0-4259-afbd-fcd50a9364d5\") " pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-cr49v"
Oct 31 12:43:18.890608 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:18.882326    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f957e25b-5bf0-4259-afbd-fcd50a9364d5-istio-data\") pod \"e2e-mt-ed25a690-openshift-default-d54cbd5df-cr49v\" (UID: \"f957e25b-5bf0-4259-afbd-fcd50a9364d5\") " pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-cr49v"
Oct 31 12:43:18.890608 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:18.882484    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f957e25b-5bf0-4259-afbd-fcd50a9364d5-workload-certs\") pod \"e2e-mt-ed25a690-openshift-default-d54cbd5df-cr49v\" (UID: \"f957e25b-5bf0-4259-afbd-fcd50a9364d5\") " pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-cr49v"
Oct 31 12:43:18.890608 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:18.882581    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f957e25b-5bf0-4259-afbd-fcd50a9364d5-credential-socket\") pod \"e2e-mt-ed25a690-openshift-default-d54cbd5df-cr49v\" (UID: \"f957e25b-5bf0-4259-afbd-fcd50a9364d5\") " pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-cr49v"
Oct 31 12:43:18.890608 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:18.882605    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbrbd\" (UniqueName: \"kubernetes.io/projected/f957e25b-5bf0-4259-afbd-fcd50a9364d5-kube-api-access-pbrbd\") pod \"e2e-mt-ed25a690-openshift-default-d54cbd5df-cr49v\" (UID: \"f957e25b-5bf0-4259-afbd-fcd50a9364d5\") " pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-cr49v"
Oct 31 12:43:18.890608 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:18.882679    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f957e25b-5bf0-4259-afbd-fcd50a9364d5-workload-socket\") pod \"e2e-mt-ed25a690-openshift-default-d54cbd5df-cr49v\" (UID: \"f957e25b-5bf0-4259-afbd-fcd50a9364d5\") " pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-cr49v"
Oct 31 12:43:18.890608 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:18.882735    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f957e25b-5bf0-4259-afbd-fcd50a9364d5-istiod-ca-cert\") pod \"e2e-mt-ed25a690-openshift-default-d54cbd5df-cr49v\" (UID: \"f957e25b-5bf0-4259-afbd-fcd50a9364d5\") " pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-cr49v"
Oct 31 12:43:18.890608 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:18.882820    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f957e25b-5bf0-4259-afbd-fcd50a9364d5-istio-podinfo\") pod \"e2e-mt-ed25a690-openshift-default-d54cbd5df-cr49v\" (UID: \"f957e25b-5bf0-4259-afbd-fcd50a9364d5\") " pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-cr49v"
Oct 31 12:43:18.890608 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:18.882861    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f957e25b-5bf0-4259-afbd-fcd50a9364d5-istio-token\") pod \"e2e-mt-ed25a690-openshift-default-d54cbd5df-cr49v\" (UID: \"f957e25b-5bf0-4259-afbd-fcd50a9364d5\") " pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-cr49v"
Oct 31 12:43:18.988373 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:18.988333    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f957e25b-5bf0-4259-afbd-fcd50a9364d5-istio-envoy\") pod \"e2e-mt-ed25a690-openshift-default-d54cbd5df-cr49v\" (UID: \"f957e25b-5bf0-4259-afbd-fcd50a9364d5\") " pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-cr49v"
Oct 31 12:43:18.988904 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:18.988410    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f957e25b-5bf0-4259-afbd-fcd50a9364d5-istio-data\") pod \"e2e-mt-ed25a690-openshift-default-d54cbd5df-cr49v\" (UID: \"f957e25b-5bf0-4259-afbd-fcd50a9364d5\") " pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-cr49v"
Oct 31 12:43:18.988904 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:18.988543    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f957e25b-5bf0-4259-afbd-fcd50a9364d5-workload-certs\") pod \"e2e-mt-ed25a690-openshift-default-d54cbd5df-cr49v\" (UID: \"f957e25b-5bf0-4259-afbd-fcd50a9364d5\") " pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-cr49v"
Oct 31 12:43:18.988904 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:18.988602    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f957e25b-5bf0-4259-afbd-fcd50a9364d5-credential-socket\") pod \"e2e-mt-ed25a690-openshift-default-d54cbd5df-cr49v\" (UID: \"f957e25b-5bf0-4259-afbd-fcd50a9364d5\") " pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-cr49v"
Oct 31 12:43:18.988904 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:18.988626    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbrbd\" (UniqueName: \"kubernetes.io/projected/f957e25b-5bf0-4259-afbd-fcd50a9364d5-kube-api-access-pbrbd\") pod \"e2e-mt-ed25a690-openshift-default-d54cbd5df-cr49v\" (UID: \"f957e25b-5bf0-4259-afbd-fcd50a9364d5\") " pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-cr49v"
Oct 31 12:43:18.988904 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:18.988684    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f957e25b-5bf0-4259-afbd-fcd50a9364d5-workload-socket\") pod \"e2e-mt-ed25a690-openshift-default-d54cbd5df-cr49v\" (UID: \"f957e25b-5bf0-4259-afbd-fcd50a9364d5\") " pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-cr49v"
Oct 31 12:43:18.988904 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:18.988733    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f957e25b-5bf0-4259-afbd-fcd50a9364d5-istiod-ca-cert\") pod \"e2e-mt-ed25a690-openshift-default-d54cbd5df-cr49v\" (UID: \"f957e25b-5bf0-4259-afbd-fcd50a9364d5\") " pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-cr49v"
Oct 31 12:43:18.988904 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:18.988800    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f957e25b-5bf0-4259-afbd-fcd50a9364d5-istio-podinfo\") pod \"e2e-mt-ed25a690-openshift-default-d54cbd5df-cr49v\" (UID: \"f957e25b-5bf0-4259-afbd-fcd50a9364d5\") " pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-cr49v"
Oct 31 12:43:18.988904 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:18.988832    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f957e25b-5bf0-4259-afbd-fcd50a9364d5-istio-token\") pod \"e2e-mt-ed25a690-openshift-default-d54cbd5df-cr49v\" (UID: \"f957e25b-5bf0-4259-afbd-fcd50a9364d5\") " pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-cr49v"
Oct 31 12:43:19.010536 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:18.989731    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f957e25b-5bf0-4259-afbd-fcd50a9364d5-istiod-ca-cert\") pod \"e2e-mt-ed25a690-openshift-default-d54cbd5df-cr49v\" (UID: \"f957e25b-5bf0-4259-afbd-fcd50a9364d5\") " pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-cr49v"
Oct 31 12:43:19.010536 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:18.989845    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f957e25b-5bf0-4259-afbd-fcd50a9364d5-workload-certs\") pod \"e2e-mt-ed25a690-openshift-default-d54cbd5df-cr49v\" (UID: \"f957e25b-5bf0-4259-afbd-fcd50a9364d5\") " pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-cr49v"
Oct 31 12:43:19.010536 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:18.990111    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f957e25b-5bf0-4259-afbd-fcd50a9364d5-istio-data\") pod \"e2e-mt-ed25a690-openshift-default-d54cbd5df-cr49v\" (UID: \"f957e25b-5bf0-4259-afbd-fcd50a9364d5\") " pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-cr49v"
Oct 31 12:43:19.010536 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:18.990361    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f957e25b-5bf0-4259-afbd-fcd50a9364d5-credential-socket\") pod \"e2e-mt-ed25a690-openshift-default-d54cbd5df-cr49v\" (UID: \"f957e25b-5bf0-4259-afbd-fcd50a9364d5\") " pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-cr49v"
Oct 31 12:43:19.010536 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:19.009554    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f957e25b-5bf0-4259-afbd-fcd50a9364d5-workload-socket\") pod \"e2e-mt-ed25a690-openshift-default-d54cbd5df-cr49v\" (UID: \"f957e25b-5bf0-4259-afbd-fcd50a9364d5\") " pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-cr49v"
Oct 31 12:43:19.025090 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:19.021877    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f957e25b-5bf0-4259-afbd-fcd50a9364d5-istio-podinfo\") pod \"e2e-mt-ed25a690-openshift-default-d54cbd5df-cr49v\" (UID: \"f957e25b-5bf0-4259-afbd-fcd50a9364d5\") " pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-cr49v"
Oct 31 12:43:19.053941 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:19.053903    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f957e25b-5bf0-4259-afbd-fcd50a9364d5-istio-envoy\") pod \"e2e-mt-ed25a690-openshift-default-d54cbd5df-cr49v\" (UID: \"f957e25b-5bf0-4259-afbd-fcd50a9364d5\") " pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-cr49v"
Oct 31 12:43:19.175294 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:19.175212    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f957e25b-5bf0-4259-afbd-fcd50a9364d5-istio-token\") pod \"e2e-mt-ed25a690-openshift-default-d54cbd5df-cr49v\" (UID: \"f957e25b-5bf0-4259-afbd-fcd50a9364d5\") " pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-cr49v"
Oct 31 12:43:19.190785 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:19.190749    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbrbd\" (UniqueName: \"kubernetes.io/projected/f957e25b-5bf0-4259-afbd-fcd50a9364d5-kube-api-access-pbrbd\") pod \"e2e-mt-ed25a690-openshift-default-d54cbd5df-cr49v\" (UID: \"f957e25b-5bf0-4259-afbd-fcd50a9364d5\") " pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-cr49v"
Oct 31 12:43:19.263343 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:19.263294    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d250eff7-d4jg7" event={"ID":"9baea4fa-5a62-48d0-b065-8167e31b8236","Type":"ContainerDied","Data":"89167f849aa1f4e33d0d71d5f4f97ddf16f9f60e520c0edb4c9b4e67cc5f8a2d"}
Oct 31 12:43:19.263644 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:19.263600    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="89167f849aa1f4e33d0d71d5f4f97ddf16f9f60e520c0edb4c9b4e67cc5f8a2d"
Oct 31 12:43:19.263896 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:19.263877    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d250eff7-d4jg7"
Oct 31 12:43:19.310533 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:19.303227    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-59fd8a28-jtp8n" event={"ID":"335aa917-cbfc-4e05-8075-38e3a2f85155","Type":"ContainerStarted","Data":"259072df86166fd266c2bbe36f55175413c641c7825cd14d4aad99fb22ca7b5e"}
Oct 31 12:43:19.413635 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:19.413605    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-cr49v"
Oct 31 12:43:19.636565 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:19.625585    5060 patch_prober.go:29] interesting pod/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-fcn5x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 12:43:19.636565 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:19.625657    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-fcn5x" podUID="aae46d99-cc6d-4be9-868b-8c4dda15a7a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:43:20.223766 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:20.223721    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:43:20.224207 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:20.223824    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:43:20.224207 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:20.223876    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:43:20.255636 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:20.255602    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-cr49v"]
Oct 31 12:43:20.313156 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:20.293761    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-fcn5x"
Oct 31 12:43:20.313156 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:20.310911    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-fcn5x"
Oct 31 12:43:20.317237 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:20.317153    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-fcn5x"
Oct 31 12:43:20.338567 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:20.337856    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-cr49v" event={"ID":"f957e25b-5bf0-4259-afbd-fcd50a9364d5","Type":"ContainerStarted","Data":"fa5bf1df3de5e9e3c6dbb7fd8d753b72bf72c33ada2ac1bec2284ed36f4709f0"}
Oct 31 12:43:20.350529 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:20.350480    5060 generic.go:356] "Generic (PLEG): container finished" podID="335aa917-cbfc-4e05-8075-38e3a2f85155" containerID="259072df86166fd266c2bbe36f55175413c641c7825cd14d4aad99fb22ca7b5e" exitCode=0
Oct 31 12:43:20.350741 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:20.350615    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-59fd8a28-jtp8n" event={"ID":"335aa917-cbfc-4e05-8075-38e3a2f85155","Type":"ContainerDied","Data":"259072df86166fd266c2bbe36f55175413c641c7825cd14d4aad99fb22ca7b5e"}
Oct 31 12:43:20.433916 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:20.432129    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-rr2hg"
Oct 31 12:43:20.434614 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:20.434570    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-rr2hg"
Oct 31 12:43:20.442399 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:20.436280    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-rr2hg"
Oct 31 12:43:20.805901 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:20.805036    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-59fd8a28-jtp8n"
Oct 31 12:43:20.898920 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:20.898608    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/335aa917-cbfc-4e05-8075-38e3a2f85155-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/335aa917-cbfc-4e05-8075-38e3a2f85155-maas-api-openshift-ca\") pod \"335aa917-cbfc-4e05-8075-38e3a2f85155\" (UID: \"335aa917-cbfc-4e05-8075-38e3a2f85155\") "
Oct 31 12:43:20.898920 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:20.898719    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/335aa917-cbfc-4e05-8075-38e3a2f85155-kube-api-access-9fzd8\" (UniqueName: \"kubernetes.io/projected/335aa917-cbfc-4e05-8075-38e3a2f85155-kube-api-access-9fzd8\") pod \"335aa917-cbfc-4e05-8075-38e3a2f85155\" (UID: \"335aa917-cbfc-4e05-8075-38e3a2f85155\") "
Oct 31 12:43:20.899207 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:20.898957    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/335aa917-cbfc-4e05-8075-38e3a2f85155-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/335aa917-cbfc-4e05-8075-38e3a2f85155-maas-api-xks-ca\") pod \"335aa917-cbfc-4e05-8075-38e3a2f85155\" (UID: \"335aa917-cbfc-4e05-8075-38e3a2f85155\") "
Oct 31 12:43:20.901300 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:20.900717    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/335aa917-cbfc-4e05-8075-38e3a2f85155-maas-api-openshift-ca" pod "335aa917-cbfc-4e05-8075-38e3a2f85155" (UID: "335aa917-cbfc-4e05-8075-38e3a2f85155"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:43:20.917719 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:20.917682    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/335aa917-cbfc-4e05-8075-38e3a2f85155-maas-api-xks-ca" pod "335aa917-cbfc-4e05-8075-38e3a2f85155" (UID: "335aa917-cbfc-4e05-8075-38e3a2f85155"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:43:20.918944 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:20.918901    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/335aa917-cbfc-4e05-8075-38e3a2f85155-kube-api-access-9fzd8" pod "335aa917-cbfc-4e05-8075-38e3a2f85155" (UID: "335aa917-cbfc-4e05-8075-38e3a2f85155"). InnerVolumeSpecName "kube-api-access-9fzd8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:43:21.001243 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:21.001149    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/335aa917-cbfc-4e05-8075-38e3a2f85155-maas-api-openshift-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:43:21.001243 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:21.001184    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9fzd8\" (UniqueName: \"kubernetes.io/projected/335aa917-cbfc-4e05-8075-38e3a2f85155-kube-api-access-9fzd8\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:43:21.001243 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:21.001203    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/335aa917-cbfc-4e05-8075-38e3a2f85155-maas-api-xks-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:43:21.395284 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:21.395247    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-cr49v" event={"ID":"f957e25b-5bf0-4259-afbd-fcd50a9364d5","Type":"ContainerStarted","Data":"3fdffae8b53e3c69a2867c4800bf0be6233f9ec417c3386499390379583398f6"}
Oct 31 12:43:21.397234 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:21.397209    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-59fd8a28-jtp8n"
Oct 31 12:43:21.397346 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:21.397228    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-59fd8a28-jtp8n" event={"ID":"335aa917-cbfc-4e05-8075-38e3a2f85155","Type":"ContainerDied","Data":"dc0f7975db39c9cf34b9b4421c0ed185c883c5f2265944106fa7fdbce1a249e2"}
Oct 31 12:43:21.397346 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:21.397285    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dc0f7975db39c9cf34b9b4421c0ed185c883c5f2265944106fa7fdbce1a249e2"
Oct 31 12:43:21.414130 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:21.414105    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-cr49v"
Oct 31 12:43:21.420924 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:21.420889    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-cr49v"
Oct 31 12:43:21.424930 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:21.424890    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-cr49v" podStartSLOduration=3.424876749 podStartE2EDuration="3.424876749s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:43:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:43:21.419257562 +0000 UTC m=+964.522325274" watchObservedRunningTime="2026-10-31 12:43:21.424876749 +0000 UTC m=+964.527944460"
Oct 31 12:43:22.412423 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:22.412385    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-cr49v"
Oct 31 12:43:22.413417 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:22.413395    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-cr49v"
Oct 31 12:43:23.298250 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:23.298213    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-sc6ff"]
Oct 31 12:43:23.299083 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:23.299057    5060 cpu_manager.go:414] "containerMap: removing container" podUID="335aa917-cbfc-4e05-8075-38e3a2f85155" containerName="revoke-keys"
Oct 31 12:43:23.299083 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:23.299081    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="335aa917-cbfc-4e05-8075-38e3a2f85155" containerName="revoke-keys"
Oct 31 12:43:23.299328 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:23.299311    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="335aa917-cbfc-4e05-8075-38e3a2f85155" containerName="revoke-keys"
Oct 31 12:43:23.304043 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:23.304022    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-sc6ff"
Oct 31 12:43:23.323539 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:23.322623    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-sc6ff"]
Oct 31 12:43:23.331223 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:23.331193    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44c7d33e-8f9zt"]
Oct 31 12:43:23.487513 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:23.405926    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44c7d33e-8f9zt"
Oct 31 12:43:23.487513 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:23.417963    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/360843c7-795f-4f22-84cd-76317addf9c2-workload-certs\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-sc6ff\" (UID: \"360843c7-795f-4f22-84cd-76317addf9c2\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-sc6ff"
Oct 31 12:43:23.487513 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:23.418161    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/360843c7-795f-4f22-84cd-76317addf9c2-istio-token\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-sc6ff\" (UID: \"360843c7-795f-4f22-84cd-76317addf9c2\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-sc6ff"
Oct 31 12:43:23.487513 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:23.418220    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/360843c7-795f-4f22-84cd-76317addf9c2-istio-envoy\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-sc6ff\" (UID: \"360843c7-795f-4f22-84cd-76317addf9c2\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-sc6ff"
Oct 31 12:43:23.487513 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:23.418293    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnvvb\" (UniqueName: \"kubernetes.io/projected/360843c7-795f-4f22-84cd-76317addf9c2-kube-api-access-mnvvb\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-sc6ff\" (UID: \"360843c7-795f-4f22-84cd-76317addf9c2\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-sc6ff"
Oct 31 12:43:23.487513 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:23.418386    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/360843c7-795f-4f22-84cd-76317addf9c2-workload-socket\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-sc6ff\" (UID: \"360843c7-795f-4f22-84cd-76317addf9c2\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-sc6ff"
Oct 31 12:43:23.487513 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:23.418410    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/360843c7-795f-4f22-84cd-76317addf9c2-credential-socket\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-sc6ff\" (UID: \"360843c7-795f-4f22-84cd-76317addf9c2\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-sc6ff"
Oct 31 12:43:23.487513 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:23.418531    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/360843c7-795f-4f22-84cd-76317addf9c2-istio-podinfo\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-sc6ff\" (UID: \"360843c7-795f-4f22-84cd-76317addf9c2\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-sc6ff"
Oct 31 12:43:23.487513 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:23.424727    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/360843c7-795f-4f22-84cd-76317addf9c2-istio-data\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-sc6ff\" (UID: \"360843c7-795f-4f22-84cd-76317addf9c2\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-sc6ff"
Oct 31 12:43:23.487513 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:23.425068    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/360843c7-795f-4f22-84cd-76317addf9c2-istiod-ca-cert\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-sc6ff\" (UID: \"360843c7-795f-4f22-84cd-76317addf9c2\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-sc6ff"
Oct 31 12:43:23.487513 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:23.434507    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7wsmb\""
Oct 31 12:43:23.487513 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:23.453589    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 12:43:23.487513 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:23.453723    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 12:43:23.533193 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:23.533148    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44c7d33e-8f9zt"]
Oct 31 12:43:23.538781 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:23.538750    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 12:43:23.606912 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:23.606831    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a85046f-2e8f-4dbf-8ce1-26f18bfda012-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-44c7d33e-8f9zt\" (UID: \"5a85046f-2e8f-4dbf-8ce1-26f18bfda012\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44c7d33e-8f9zt"
Oct 31 12:43:23.606912 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:23.606893    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/360843c7-795f-4f22-84cd-76317addf9c2-istiod-ca-cert\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-sc6ff\" (UID: \"360843c7-795f-4f22-84cd-76317addf9c2\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-sc6ff"
Oct 31 12:43:23.607106 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:23.606932    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58wg2\" (UniqueName: \"kubernetes.io/projected/5a85046f-2e8f-4dbf-8ce1-26f18bfda012-kube-api-access-58wg2\") pod \"maas-api-revoke-sub-44c7d33e-8f9zt\" (UID: \"5a85046f-2e8f-4dbf-8ce1-26f18bfda012\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44c7d33e-8f9zt"
Oct 31 12:43:23.607106 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:23.606985    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/360843c7-795f-4f22-84cd-76317addf9c2-workload-certs\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-sc6ff\" (UID: \"360843c7-795f-4f22-84cd-76317addf9c2\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-sc6ff"
Oct 31 12:43:23.607278 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:23.607185    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/360843c7-795f-4f22-84cd-76317addf9c2-istio-token\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-sc6ff\" (UID: \"360843c7-795f-4f22-84cd-76317addf9c2\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-sc6ff"
Oct 31 12:43:23.607342 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:23.607264    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/360843c7-795f-4f22-84cd-76317addf9c2-istio-envoy\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-sc6ff\" (UID: \"360843c7-795f-4f22-84cd-76317addf9c2\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-sc6ff"
Oct 31 12:43:23.607394 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:23.607353    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnvvb\" (UniqueName: \"kubernetes.io/projected/360843c7-795f-4f22-84cd-76317addf9c2-kube-api-access-mnvvb\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-sc6ff\" (UID: \"360843c7-795f-4f22-84cd-76317addf9c2\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-sc6ff"
Oct 31 12:43:23.607452 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:23.607394    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/360843c7-795f-4f22-84cd-76317addf9c2-workload-socket\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-sc6ff\" (UID: \"360843c7-795f-4f22-84cd-76317addf9c2\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-sc6ff"
Oct 31 12:43:23.607452 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:23.607419    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/360843c7-795f-4f22-84cd-76317addf9c2-credential-socket\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-sc6ff\" (UID: \"360843c7-795f-4f22-84cd-76317addf9c2\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-sc6ff"
Oct 31 12:43:23.607574 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:23.607483    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/360843c7-795f-4f22-84cd-76317addf9c2-workload-certs\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-sc6ff\" (UID: \"360843c7-795f-4f22-84cd-76317addf9c2\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-sc6ff"
Oct 31 12:43:23.607777 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:23.607758    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/360843c7-795f-4f22-84cd-76317addf9c2-credential-socket\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-sc6ff\" (UID: \"360843c7-795f-4f22-84cd-76317addf9c2\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-sc6ff"
Oct 31 12:43:23.607860 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:23.607816    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/360843c7-795f-4f22-84cd-76317addf9c2-workload-socket\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-sc6ff\" (UID: \"360843c7-795f-4f22-84cd-76317addf9c2\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-sc6ff"
Oct 31 12:43:23.607860 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:23.607821    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/360843c7-795f-4f22-84cd-76317addf9c2-istio-podinfo\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-sc6ff\" (UID: \"360843c7-795f-4f22-84cd-76317addf9c2\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-sc6ff"
Oct 31 12:43:23.607978 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:23.607938    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a85046f-2e8f-4dbf-8ce1-26f18bfda012-maas-api-xks-ca\") pod \"maas-api-revoke-sub-44c7d33e-8f9zt\" (UID: \"5a85046f-2e8f-4dbf-8ce1-26f18bfda012\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44c7d33e-8f9zt"
Oct 31 12:43:23.608436 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:23.608412    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/360843c7-795f-4f22-84cd-76317addf9c2-istio-data\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-sc6ff\" (UID: \"360843c7-795f-4f22-84cd-76317addf9c2\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-sc6ff"
Oct 31 12:43:23.608944 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:23.608901    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/360843c7-795f-4f22-84cd-76317addf9c2-istio-data\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-sc6ff\" (UID: \"360843c7-795f-4f22-84cd-76317addf9c2\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-sc6ff"
Oct 31 12:43:23.609060 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:23.608900    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/360843c7-795f-4f22-84cd-76317addf9c2-istiod-ca-cert\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-sc6ff\" (UID: \"360843c7-795f-4f22-84cd-76317addf9c2\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-sc6ff"
Oct 31 12:43:23.628341 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:23.628275    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/360843c7-795f-4f22-84cd-76317addf9c2-istio-podinfo\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-sc6ff\" (UID: \"360843c7-795f-4f22-84cd-76317addf9c2\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-sc6ff"
Oct 31 12:43:23.629482 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:23.629396    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/360843c7-795f-4f22-84cd-76317addf9c2-istio-envoy\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-sc6ff\" (UID: \"360843c7-795f-4f22-84cd-76317addf9c2\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-sc6ff"
Oct 31 12:43:23.630699 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:23.630659    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/360843c7-795f-4f22-84cd-76317addf9c2-istio-token\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-sc6ff\" (UID: \"360843c7-795f-4f22-84cd-76317addf9c2\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-sc6ff"
Oct 31 12:43:23.632459 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:23.632433    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnvvb\" (UniqueName: \"kubernetes.io/projected/360843c7-795f-4f22-84cd-76317addf9c2-kube-api-access-mnvvb\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-sc6ff\" (UID: \"360843c7-795f-4f22-84cd-76317addf9c2\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-sc6ff"
Oct 31 12:43:23.709832 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:23.709792    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a85046f-2e8f-4dbf-8ce1-26f18bfda012-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-44c7d33e-8f9zt\" (UID: \"5a85046f-2e8f-4dbf-8ce1-26f18bfda012\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44c7d33e-8f9zt"
Oct 31 12:43:23.709998 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:23.709882    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58wg2\" (UniqueName: \"kubernetes.io/projected/5a85046f-2e8f-4dbf-8ce1-26f18bfda012-kube-api-access-58wg2\") pod \"maas-api-revoke-sub-44c7d33e-8f9zt\" (UID: \"5a85046f-2e8f-4dbf-8ce1-26f18bfda012\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44c7d33e-8f9zt"
Oct 31 12:43:23.710087 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:23.710073    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a85046f-2e8f-4dbf-8ce1-26f18bfda012-maas-api-xks-ca\") pod \"maas-api-revoke-sub-44c7d33e-8f9zt\" (UID: \"5a85046f-2e8f-4dbf-8ce1-26f18bfda012\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44c7d33e-8f9zt"
Oct 31 12:43:23.710653 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:23.710634    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a85046f-2e8f-4dbf-8ce1-26f18bfda012-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-44c7d33e-8f9zt\" (UID: \"5a85046f-2e8f-4dbf-8ce1-26f18bfda012\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44c7d33e-8f9zt"
Oct 31 12:43:23.712570 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:23.712542    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a85046f-2e8f-4dbf-8ce1-26f18bfda012-maas-api-xks-ca\") pod \"maas-api-revoke-sub-44c7d33e-8f9zt\" (UID: \"5a85046f-2e8f-4dbf-8ce1-26f18bfda012\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44c7d33e-8f9zt"
Oct 31 12:43:23.732891 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:23.732859    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58wg2\" (UniqueName: \"kubernetes.io/projected/5a85046f-2e8f-4dbf-8ce1-26f18bfda012-kube-api-access-58wg2\") pod \"maas-api-revoke-sub-44c7d33e-8f9zt\" (UID: \"5a85046f-2e8f-4dbf-8ce1-26f18bfda012\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44c7d33e-8f9zt"
Oct 31 12:43:23.847514 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:23.846767    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44c7d33e-8f9zt"
Oct 31 12:43:23.966752 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:23.965832    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-sc6ff"
Oct 31 12:43:24.483334 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:24.482125    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d5e8c3f2-pz9b6"]
Oct 31 12:43:24.492265 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:24.490923    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d5e8c3f2-pz9b6"
Oct 31 12:43:24.497198 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:24.497098    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d5e8c3f2-pz9b6"]
Oct 31 12:43:24.506408 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:24.506381    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44c7d33e-8f9zt"]
Oct 31 12:43:24.517839 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:24.517786    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-sc6ff"]
Oct 31 12:43:24.530341 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:24.529700    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:43:24.530341 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:24.529967    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:43:24.530341 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:24.530060    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:43:24.546359 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:24.546302    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-sc6ff" event={"ID":"360843c7-795f-4f22-84cd-76317addf9c2","Type":"ContainerStarted","Data":"aff6b7cd9c407005e7fba993db40d14ce04e6cf0f1a2e8566c594d3f71a04ad6"}
Oct 31 12:43:24.549354 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:24.549326    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44c7d33e-8f9zt" event={"ID":"5a85046f-2e8f-4dbf-8ce1-26f18bfda012","Type":"ContainerStarted","Data":"19ffa9cd7727e33a897ccdeae693806bd09f7431152fefafeda86048a887092f"}
Oct 31 12:43:24.583435 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:24.581620    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfm26\" (UniqueName: \"kubernetes.io/projected/3126c9e8-c907-422c-94b7-0c9baea649c9-kube-api-access-sfm26\") pod \"maas-api-revoke-keys-e2e-derive-d5e8c3f2-pz9b6\" (UID: \"3126c9e8-c907-422c-94b7-0c9baea649c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d5e8c3f2-pz9b6"
Oct 31 12:43:24.583435 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:24.581763    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3126c9e8-c907-422c-94b7-0c9baea649c9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-d5e8c3f2-pz9b6\" (UID: \"3126c9e8-c907-422c-94b7-0c9baea649c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d5e8c3f2-pz9b6"
Oct 31 12:43:24.583435 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:24.582023    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3126c9e8-c907-422c-94b7-0c9baea649c9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-d5e8c3f2-pz9b6\" (UID: \"3126c9e8-c907-422c-94b7-0c9baea649c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d5e8c3f2-pz9b6"
Oct 31 12:43:24.684602 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:24.684557    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfm26\" (UniqueName: \"kubernetes.io/projected/3126c9e8-c907-422c-94b7-0c9baea649c9-kube-api-access-sfm26\") pod \"maas-api-revoke-keys-e2e-derive-d5e8c3f2-pz9b6\" (UID: \"3126c9e8-c907-422c-94b7-0c9baea649c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d5e8c3f2-pz9b6"
Oct 31 12:43:24.684784 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:24.684745    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3126c9e8-c907-422c-94b7-0c9baea649c9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-d5e8c3f2-pz9b6\" (UID: \"3126c9e8-c907-422c-94b7-0c9baea649c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d5e8c3f2-pz9b6"
Oct 31 12:43:24.685037 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:24.685018    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3126c9e8-c907-422c-94b7-0c9baea649c9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-d5e8c3f2-pz9b6\" (UID: \"3126c9e8-c907-422c-94b7-0c9baea649c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d5e8c3f2-pz9b6"
Oct 31 12:43:24.686083 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:24.686052    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3126c9e8-c907-422c-94b7-0c9baea649c9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-d5e8c3f2-pz9b6\" (UID: \"3126c9e8-c907-422c-94b7-0c9baea649c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d5e8c3f2-pz9b6"
Oct 31 12:43:24.697040 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:24.697005    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3126c9e8-c907-422c-94b7-0c9baea649c9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-d5e8c3f2-pz9b6\" (UID: \"3126c9e8-c907-422c-94b7-0c9baea649c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d5e8c3f2-pz9b6"
Oct 31 12:43:24.701219 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:24.701146    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfm26\" (UniqueName: \"kubernetes.io/projected/3126c9e8-c907-422c-94b7-0c9baea649c9-kube-api-access-sfm26\") pod \"maas-api-revoke-keys-e2e-derive-d5e8c3f2-pz9b6\" (UID: \"3126c9e8-c907-422c-94b7-0c9baea649c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d5e8c3f2-pz9b6"
Oct 31 12:43:24.849174 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:24.849134    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d5e8c3f2-pz9b6"
Oct 31 12:43:25.746550 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:25.741526    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d5e8c3f2-pz9b6"]
Oct 31 12:43:25.748064 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:25.748025    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44c7d33e-8f9zt" event={"ID":"5a85046f-2e8f-4dbf-8ce1-26f18bfda012","Type":"ContainerStarted","Data":"4a8e2162d8ca6fce13c30b1b0cfb178283457dbd528c4e20d4f901a2a7edaa66"}
Oct 31 12:43:25.755323 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:25.754868    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-sc6ff" event={"ID":"360843c7-795f-4f22-84cd-76317addf9c2","Type":"ContainerStarted","Data":"b163f7ffd10c6999491edd6aadba751bcc983950890ceebea3672e7540b9ec48"}
Oct 31 12:43:25.765052 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:25.757157    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d5e8c3f2-pz9b6" event={"ID":"3126c9e8-c907-422c-94b7-0c9baea649c9","Type":"ContainerStarted","Data":"f59f684313c442a22baf0a5a9ef907e70d07ff1906268c8c3a76a396981d9bab"}
Oct 31 12:43:25.770462 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:25.770409    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44c7d33e-8f9zt" podStartSLOduration=2.770394768 podStartE2EDuration="2.770394768s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:43:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:43:25.768378781 +0000 UTC m=+968.871446494" watchObservedRunningTime="2026-10-31 12:43:25.770394768 +0000 UTC m=+968.873462477"
Oct 31 12:43:25.822458 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:25.822331    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-sc6ff" podStartSLOduration=2.822310312 podStartE2EDuration="2.822310312s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:43:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:43:25.804070054 +0000 UTC m=+968.907137774" watchObservedRunningTime="2026-10-31 12:43:25.822310312 +0000 UTC m=+968.925378028"
Oct 31 12:43:25.989663 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:25.971700    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-sc6ff"
Oct 31 12:43:25.989930 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:25.975638    5060 patch_prober.go:29] interesting pod/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-sc6ff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body=
Oct 31 12:43:25.990099 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:25.990071    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-sc6ff" podUID="360843c7-795f-4f22-84cd-76317addf9c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused"
Oct 31 12:43:26.329754 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:26.329696    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b849a4c4-s8d6j"]
Oct 31 12:43:26.340154 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:26.340125    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b849a4c4-s8d6j"
Oct 31 12:43:26.341770 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:26.341744    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b849a4c4-s8d6j"]
Oct 31 12:43:26.406535 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:26.406428    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ed56313-9c9d-4e0f-9764-c8b9dacdfda2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-b849a4c4-s8d6j\" (UID: \"6ed56313-9c9d-4e0f-9764-c8b9dacdfda2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b849a4c4-s8d6j"
Oct 31 12:43:26.406707 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:26.406571    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ed56313-9c9d-4e0f-9764-c8b9dacdfda2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-b849a4c4-s8d6j\" (UID: \"6ed56313-9c9d-4e0f-9764-c8b9dacdfda2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b849a4c4-s8d6j"
Oct 31 12:43:26.406773 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:26.406726    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqsz4\" (UniqueName: \"kubernetes.io/projected/6ed56313-9c9d-4e0f-9764-c8b9dacdfda2-kube-api-access-zqsz4\") pod \"maas-api-revoke-keys-e2e-ait-b849a4c4-s8d6j\" (UID: \"6ed56313-9c9d-4e0f-9764-c8b9dacdfda2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b849a4c4-s8d6j"
Oct 31 12:43:26.515517 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:26.508858    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ed56313-9c9d-4e0f-9764-c8b9dacdfda2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-b849a4c4-s8d6j\" (UID: \"6ed56313-9c9d-4e0f-9764-c8b9dacdfda2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b849a4c4-s8d6j"
Oct 31 12:43:26.515517 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:26.508961    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ed56313-9c9d-4e0f-9764-c8b9dacdfda2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-b849a4c4-s8d6j\" (UID: \"6ed56313-9c9d-4e0f-9764-c8b9dacdfda2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b849a4c4-s8d6j"
Oct 31 12:43:26.515517 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:26.509134    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqsz4\" (UniqueName: \"kubernetes.io/projected/6ed56313-9c9d-4e0f-9764-c8b9dacdfda2-kube-api-access-zqsz4\") pod \"maas-api-revoke-keys-e2e-ait-b849a4c4-s8d6j\" (UID: \"6ed56313-9c9d-4e0f-9764-c8b9dacdfda2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b849a4c4-s8d6j"
Oct 31 12:43:26.515517 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:26.513029    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ed56313-9c9d-4e0f-9764-c8b9dacdfda2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-b849a4c4-s8d6j\" (UID: \"6ed56313-9c9d-4e0f-9764-c8b9dacdfda2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b849a4c4-s8d6j"
Oct 31 12:43:26.548519 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:26.546760    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ed56313-9c9d-4e0f-9764-c8b9dacdfda2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-b849a4c4-s8d6j\" (UID: \"6ed56313-9c9d-4e0f-9764-c8b9dacdfda2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b849a4c4-s8d6j"
Oct 31 12:43:26.555509 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:26.551071    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqsz4\" (UniqueName: \"kubernetes.io/projected/6ed56313-9c9d-4e0f-9764-c8b9dacdfda2-kube-api-access-zqsz4\") pod \"maas-api-revoke-keys-e2e-ait-b849a4c4-s8d6j\" (UID: \"6ed56313-9c9d-4e0f-9764-c8b9dacdfda2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b849a4c4-s8d6j"
Oct 31 12:43:26.709950 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:26.709867    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b849a4c4-s8d6j"
Oct 31 12:43:26.816573 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:26.816145    5060 generic.go:356] "Generic (PLEG): container finished" podID="5a85046f-2e8f-4dbf-8ce1-26f18bfda012" containerID="4a8e2162d8ca6fce13c30b1b0cfb178283457dbd528c4e20d4f901a2a7edaa66" exitCode=0
Oct 31 12:43:26.816573 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:26.816463    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44c7d33e-8f9zt" event={"ID":"5a85046f-2e8f-4dbf-8ce1-26f18bfda012","Type":"ContainerDied","Data":"4a8e2162d8ca6fce13c30b1b0cfb178283457dbd528c4e20d4f901a2a7edaa66"}
Oct 31 12:43:26.819279 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:26.819235    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d5e8c3f2-pz9b6" event={"ID":"3126c9e8-c907-422c-94b7-0c9baea649c9","Type":"ContainerStarted","Data":"be123b8add8c5d7494dee8a9a60179d44dd325fb208e2e70fdf6aa466f992175"}
Oct 31 12:43:26.914675 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:26.914604    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d5e8c3f2-pz9b6" podStartSLOduration=2.914586264 podStartE2EDuration="2.914586264s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:43:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:43:26.866964771 +0000 UTC m=+969.970032484" watchObservedRunningTime="2026-10-31 12:43:26.914586264 +0000 UTC m=+970.017653976"
Oct 31 12:43:26.975957 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:26.975033    5060 patch_prober.go:29] interesting pod/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-sc6ff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body=
Oct 31 12:43:26.975957 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:26.975091    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-sc6ff" podUID="360843c7-795f-4f22-84cd-76317addf9c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused"
Oct 31 12:43:27.453632 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:27.448601    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b849a4c4-s8d6j"]
Oct 31 12:43:27.830979 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:27.830946    5060 generic.go:356] "Generic (PLEG): container finished" podID="3126c9e8-c907-422c-94b7-0c9baea649c9" containerID="be123b8add8c5d7494dee8a9a60179d44dd325fb208e2e70fdf6aa466f992175" exitCode=0
Oct 31 12:43:27.831396 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:27.831032    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d5e8c3f2-pz9b6" event={"ID":"3126c9e8-c907-422c-94b7-0c9baea649c9","Type":"ContainerDied","Data":"be123b8add8c5d7494dee8a9a60179d44dd325fb208e2e70fdf6aa466f992175"}
Oct 31 12:43:27.832524 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:27.832483    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b849a4c4-s8d6j" event={"ID":"6ed56313-9c9d-4e0f-9764-c8b9dacdfda2","Type":"ContainerStarted","Data":"0ff90a078e6d1afa19cc990d146168f88215d412fbce2e96e1b5f695d25bc870"}
Oct 31 12:43:27.993517 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:27.992106    5060 patch_prober.go:29] interesting pod/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-sc6ff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body=
Oct 31 12:43:27.993517 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:27.992184    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-sc6ff" podUID="360843c7-795f-4f22-84cd-76317addf9c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused"
Oct 31 12:43:28.192045 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:28.191968    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44c7d33e-8f9zt"
Oct 31 12:43:28.282515 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:28.274596    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5a85046f-2e8f-4dbf-8ce1-26f18bfda012-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a85046f-2e8f-4dbf-8ce1-26f18bfda012-maas-api-xks-ca\") pod \"5a85046f-2e8f-4dbf-8ce1-26f18bfda012\" (UID: \"5a85046f-2e8f-4dbf-8ce1-26f18bfda012\") "
Oct 31 12:43:28.282515 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:28.274675    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a85046f-2e8f-4dbf-8ce1-26f18bfda012-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a85046f-2e8f-4dbf-8ce1-26f18bfda012-maas-api-openshift-ca\") pod \"5a85046f-2e8f-4dbf-8ce1-26f18bfda012\" (UID: \"5a85046f-2e8f-4dbf-8ce1-26f18bfda012\") "
Oct 31 12:43:28.282515 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:28.274747    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a85046f-2e8f-4dbf-8ce1-26f18bfda012-kube-api-access-58wg2\" (UniqueName: \"kubernetes.io/projected/5a85046f-2e8f-4dbf-8ce1-26f18bfda012-kube-api-access-58wg2\") pod \"5a85046f-2e8f-4dbf-8ce1-26f18bfda012\" (UID: \"5a85046f-2e8f-4dbf-8ce1-26f18bfda012\") "
Oct 31 12:43:28.296822 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:28.296749    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a85046f-2e8f-4dbf-8ce1-26f18bfda012-maas-api-openshift-ca" pod "5a85046f-2e8f-4dbf-8ce1-26f18bfda012" (UID: "5a85046f-2e8f-4dbf-8ce1-26f18bfda012"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:43:28.300726 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:28.300675    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a85046f-2e8f-4dbf-8ce1-26f18bfda012-kube-api-access-58wg2" pod "5a85046f-2e8f-4dbf-8ce1-26f18bfda012" (UID: "5a85046f-2e8f-4dbf-8ce1-26f18bfda012"). InnerVolumeSpecName "kube-api-access-58wg2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:43:28.305708 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:28.305589    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a85046f-2e8f-4dbf-8ce1-26f18bfda012-maas-api-xks-ca" pod "5a85046f-2e8f-4dbf-8ce1-26f18bfda012" (UID: "5a85046f-2e8f-4dbf-8ce1-26f18bfda012"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:43:28.388585 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:28.378208    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a85046f-2e8f-4dbf-8ce1-26f18bfda012-maas-api-xks-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:43:28.408027 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:28.407989    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a85046f-2e8f-4dbf-8ce1-26f18bfda012-maas-api-openshift-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:43:28.425073 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:28.425042    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58wg2\" (UniqueName: \"kubernetes.io/projected/5a85046f-2e8f-4dbf-8ce1-26f18bfda012-kube-api-access-58wg2\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:43:28.976951 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:28.976911    5060 patch_prober.go:29] interesting pod/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-sc6ff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body=
Oct 31 12:43:29.016675 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:29.016576    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-sc6ff" podUID="360843c7-795f-4f22-84cd-76317addf9c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused"
Oct 31 12:43:29.158515 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:29.154924    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b849a4c4-s8d6j" event={"ID":"6ed56313-9c9d-4e0f-9764-c8b9dacdfda2","Type":"ContainerStarted","Data":"41c3aca603a4f0b29ce259ea8da92927ca9c6c13f57317d6f6dead9e8721479b"}
Oct 31 12:43:29.203529 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:29.202613    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44c7d33e-8f9zt"
Oct 31 12:43:29.207514 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:29.204903    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44c7d33e-8f9zt" event={"ID":"5a85046f-2e8f-4dbf-8ce1-26f18bfda012","Type":"ContainerDied","Data":"19ffa9cd7727e33a897ccdeae693806bd09f7431152fefafeda86048a887092f"}
Oct 31 12:43:29.207514 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:29.205000    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="19ffa9cd7727e33a897ccdeae693806bd09f7431152fefafeda86048a887092f"
Oct 31 12:43:29.689570 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:29.683276    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d5e8c3f2-pz9b6"
Oct 31 12:43:29.796527 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:29.793280    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3126c9e8-c907-422c-94b7-0c9baea649c9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3126c9e8-c907-422c-94b7-0c9baea649c9-maas-api-openshift-ca\") pod \"3126c9e8-c907-422c-94b7-0c9baea649c9\" (UID: \"3126c9e8-c907-422c-94b7-0c9baea649c9\") "
Oct 31 12:43:29.796527 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:29.793430    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3126c9e8-c907-422c-94b7-0c9baea649c9-kube-api-access-sfm26\" (UniqueName: \"kubernetes.io/projected/3126c9e8-c907-422c-94b7-0c9baea649c9-kube-api-access-sfm26\") pod \"3126c9e8-c907-422c-94b7-0c9baea649c9\" (UID: \"3126c9e8-c907-422c-94b7-0c9baea649c9\") "
Oct 31 12:43:29.796527 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:29.793476    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3126c9e8-c907-422c-94b7-0c9baea649c9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3126c9e8-c907-422c-94b7-0c9baea649c9-maas-api-xks-ca\") pod \"3126c9e8-c907-422c-94b7-0c9baea649c9\" (UID: \"3126c9e8-c907-422c-94b7-0c9baea649c9\") "
Oct 31 12:43:29.796527 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:29.794923    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3126c9e8-c907-422c-94b7-0c9baea649c9-maas-api-openshift-ca" pod "3126c9e8-c907-422c-94b7-0c9baea649c9" (UID: "3126c9e8-c907-422c-94b7-0c9baea649c9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:43:29.801545 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:29.800093    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3126c9e8-c907-422c-94b7-0c9baea649c9-kube-api-access-sfm26" pod "3126c9e8-c907-422c-94b7-0c9baea649c9" (UID: "3126c9e8-c907-422c-94b7-0c9baea649c9"). InnerVolumeSpecName "kube-api-access-sfm26". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:43:29.802985 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:29.802953    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3126c9e8-c907-422c-94b7-0c9baea649c9-maas-api-xks-ca" pod "3126c9e8-c907-422c-94b7-0c9baea649c9" (UID: "3126c9e8-c907-422c-94b7-0c9baea649c9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:43:29.895983 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:29.895901    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3126c9e8-c907-422c-94b7-0c9baea649c9-maas-api-openshift-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:43:29.895983 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:29.895932    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sfm26\" (UniqueName: \"kubernetes.io/projected/3126c9e8-c907-422c-94b7-0c9baea649c9-kube-api-access-sfm26\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:43:29.895983 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:29.895948    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3126c9e8-c907-422c-94b7-0c9baea649c9-maas-api-xks-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:43:29.933749 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:43:29.933671    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7phb" containerName="extractor"
Oct 31 12:43:29.971531 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:29.969939    5060 patch_prober.go:29] interesting pod/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-sc6ff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body=
Oct 31 12:43:29.971531 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:29.970003    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-sc6ff" podUID="360843c7-795f-4f22-84cd-76317addf9c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused"
Oct 31 12:43:30.216019 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:30.215935    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d5e8c3f2-pz9b6"
Oct 31 12:43:30.216454 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:30.215941    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d5e8c3f2-pz9b6" event={"ID":"3126c9e8-c907-422c-94b7-0c9baea649c9","Type":"ContainerDied","Data":"f59f684313c442a22baf0a5a9ef907e70d07ff1906268c8c3a76a396981d9bab"}
Oct 31 12:43:30.216454 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:30.216060    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f59f684313c442a22baf0a5a9ef907e70d07ff1906268c8c3a76a396981d9bab"
Oct 31 12:43:30.218299 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:30.218267    5060 generic.go:356] "Generic (PLEG): container finished" podID="6ed56313-9c9d-4e0f-9764-c8b9dacdfda2" containerID="41c3aca603a4f0b29ce259ea8da92927ca9c6c13f57317d6f6dead9e8721479b" exitCode=0
Oct 31 12:43:30.218445 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:30.218341    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b849a4c4-s8d6j" event={"ID":"6ed56313-9c9d-4e0f-9764-c8b9dacdfda2","Type":"ContainerDied","Data":"41c3aca603a4f0b29ce259ea8da92927ca9c6c13f57317d6f6dead9e8721479b"}
Oct 31 12:43:30.513668 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:30.513632    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b849a4c4-s8d6j"
Oct 31 12:43:30.628523 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:30.615988    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ed56313-9c9d-4e0f-9764-c8b9dacdfda2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ed56313-9c9d-4e0f-9764-c8b9dacdfda2-maas-api-openshift-ca\") pod \"6ed56313-9c9d-4e0f-9764-c8b9dacdfda2\" (UID: \"6ed56313-9c9d-4e0f-9764-c8b9dacdfda2\") "
Oct 31 12:43:30.628523 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:30.616321    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ed56313-9c9d-4e0f-9764-c8b9dacdfda2-kube-api-access-zqsz4\" (UniqueName: \"kubernetes.io/projected/6ed56313-9c9d-4e0f-9764-c8b9dacdfda2-kube-api-access-zqsz4\") pod \"6ed56313-9c9d-4e0f-9764-c8b9dacdfda2\" (UID: \"6ed56313-9c9d-4e0f-9764-c8b9dacdfda2\") "
Oct 31 12:43:30.628523 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:30.616391    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6ed56313-9c9d-4e0f-9764-c8b9dacdfda2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ed56313-9c9d-4e0f-9764-c8b9dacdfda2-maas-api-xks-ca\") pod \"6ed56313-9c9d-4e0f-9764-c8b9dacdfda2\" (UID: \"6ed56313-9c9d-4e0f-9764-c8b9dacdfda2\") "
Oct 31 12:43:30.628523 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:30.617876    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ed56313-9c9d-4e0f-9764-c8b9dacdfda2-maas-api-openshift-ca" pod "6ed56313-9c9d-4e0f-9764-c8b9dacdfda2" (UID: "6ed56313-9c9d-4e0f-9764-c8b9dacdfda2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:43:30.639628 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:30.634002    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ed56313-9c9d-4e0f-9764-c8b9dacdfda2-maas-api-xks-ca" pod "6ed56313-9c9d-4e0f-9764-c8b9dacdfda2" (UID: "6ed56313-9c9d-4e0f-9764-c8b9dacdfda2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:43:30.640096 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:30.640065    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ed56313-9c9d-4e0f-9764-c8b9dacdfda2-kube-api-access-zqsz4" pod "6ed56313-9c9d-4e0f-9764-c8b9dacdfda2" (UID: "6ed56313-9c9d-4e0f-9764-c8b9dacdfda2"). InnerVolumeSpecName "kube-api-access-zqsz4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:43:30.747599 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:30.746966    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ed56313-9c9d-4e0f-9764-c8b9dacdfda2-maas-api-openshift-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:43:30.747599 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:30.747009    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqsz4\" (UniqueName: \"kubernetes.io/projected/6ed56313-9c9d-4e0f-9764-c8b9dacdfda2-kube-api-access-zqsz4\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:43:30.747599 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:30.747028    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ed56313-9c9d-4e0f-9764-c8b9dacdfda2-maas-api-xks-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:43:31.009567 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:30.997121    5060 patch_prober.go:29] interesting pod/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-sc6ff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body=
Oct 31 12:43:31.009567 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:30.997180    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-sc6ff" podUID="360843c7-795f-4f22-84cd-76317addf9c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused"
Oct 31 12:43:31.505716 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:31.505594    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b849a4c4-s8d6j" event={"ID":"6ed56313-9c9d-4e0f-9764-c8b9dacdfda2","Type":"ContainerDied","Data":"0ff90a078e6d1afa19cc990d146168f88215d412fbce2e96e1b5f695d25bc870"}
Oct 31 12:43:31.505716 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:31.505676    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0ff90a078e6d1afa19cc990d146168f88215d412fbce2e96e1b5f695d25bc870"
Oct 31 12:43:31.506383 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:31.506357    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b849a4c4-s8d6j"
Oct 31 12:43:31.587235 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:31.587189    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-2-w5-b25d3b-kserve-76f798d59f-vvzhg" podUID="394ed76c-84ac-4612-9acb-a7e36cb68d89" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": context deadline exceeded"
Oct 31 12:43:31.610374 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:31.610327    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/premium-w2-e6c79d-kserve-6468bf46d4-h66f8" podUID="cd6a266f-8d12-4089-83ee-c75c716bf26c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Oct 31 12:43:31.682554 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:31.681281    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x2qzv" podUID="c247ed2a-aa90-424b-9b7e-4f2936d67e3f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:43:31.971517 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:31.967967    5060 patch_prober.go:29] interesting pod/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-sc6ff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body=
Oct 31 12:43:31.971814 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:31.971777    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-sc6ff" podUID="360843c7-795f-4f22-84cd-76317addf9c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused"
Oct 31 12:43:32.274705 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:32.274615    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/simulator-w4-c67de2-kserve-868679cf9b-gqjbx" podUID="e104d7da-1fb2-4c99-a480-b4ae84c4f273" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded"
Oct 31 12:43:32.393045 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:32.392998    5060 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x2qzv" podUID="c247ed2a-aa90-424b-9b7e-4f2936d67e3f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": context deadline exceeded"
Oct 31 12:43:32.602641 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:32.588160    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-2-w5-b25d3b-kserve-76f798d59f-vvzhg" podUID="394ed76c-84ac-4612-9acb-a7e36cb68d89" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": context deadline exceeded"
Oct 31 12:43:32.602641 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:32.588941    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-2-w5-b25d3b-kserve-76f798d59f-vvzhg"
Oct 31 12:43:32.706563 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:32.651917    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/premium-w2-e6c79d-kserve-6468bf46d4-h66f8" podUID="cd6a266f-8d12-4089-83ee-c75c716bf26c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Oct 31 12:43:33.107219 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:32.825151    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x2qzv" podUID="c247ed2a-aa90-424b-9b7e-4f2936d67e3f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded"
Oct 31 12:43:33.107428 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:33.075682    5060 patch_prober.go:29] interesting pod/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-sc6ff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body=
Oct 31 12:43:33.107428 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:33.107379    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-sc6ff" podUID="360843c7-795f-4f22-84cd-76317addf9c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused"
Oct 31 12:43:33.312648 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:33.312598    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/simulator-w4-c67de2-kserve-868679cf9b-gqjbx" podUID="e104d7da-1fb2-4c99-a480-b4ae84c4f273" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded"
Oct 31 12:43:33.678095 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:33.678042    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-2-w5-b25d3b-kserve-76f798d59f-vvzhg" podUID="394ed76c-84ac-4612-9acb-a7e36cb68d89" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": context deadline exceeded"
Oct 31 12:43:33.791517 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:33.787609    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e6c79d/premium-w2-e6c79d-kserve-6468bf46d4-h66f8"
Oct 31 12:43:33.894600 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:33.894566    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x2qzv"
Oct 31 12:43:33.894881 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:33.894863    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c67de2/simulator-w4-c67de2-kserve-868679cf9b-gqjbx"
Oct 31 12:43:34.154521 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:34.095625    5060 patch_prober.go:29] interesting pod/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-sc6ff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body=
Oct 31 12:43:34.154521 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:34.095691    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-sc6ff" podUID="360843c7-795f-4f22-84cd-76317addf9c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused"
Oct 31 12:43:34.154521 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:34.111596    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x2qzv" podUID="c247ed2a-aa90-424b-9b7e-4f2936d67e3f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:43:34.154521 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:34.112019    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/premium-w2-e6c79d-kserve-6468bf46d4-h66f8" podUID="cd6a266f-8d12-4089-83ee-c75c716bf26c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Oct 31 12:43:34.353536 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:34.346059    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/simulator-w4-c67de2-kserve-868679cf9b-gqjbx" podUID="e104d7da-1fb2-4c99-a480-b4ae84c4f273" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:43:34.355823 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:34.355789    5060 patch_prober.go:29] interesting pod/e2e-worker-w1-85ce08-openshift-default-77d6458845-sv4w2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 12:43:34.359512 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:34.356041    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-sv4w2" podUID="38bdca08-eb2b-43c5-8c07-7059e5c9e199" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:43:34.691731 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:34.691669    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-2-w5-b25d3b-kserve-76f798d59f-vvzhg" podUID="394ed76c-84ac-4612-9acb-a7e36cb68d89" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:43:34.982720 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:34.982623    5060 patch_prober.go:29] interesting pod/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-njl9z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 12:43:34.986515 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:34.982992    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-njl9z" podUID="29b176fa-229b-433e-928f-41daf33d1f37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:43:35.035870 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:34.997919    5060 patch_prober.go:29] interesting pod/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-sc6ff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body=
Oct 31 12:43:35.042614 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:35.042568    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-sc6ff" podUID="360843c7-795f-4f22-84cd-76317addf9c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused"
Oct 31 12:43:35.112689 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:35.112638    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/premium-w2-e6c79d-kserve-6468bf46d4-h66f8" podUID="cd6a266f-8d12-4089-83ee-c75c716bf26c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:43:35.223349 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:35.112946    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x2qzv" podUID="c247ed2a-aa90-424b-9b7e-4f2936d67e3f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:43:35.422575 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:35.417546    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/simulator-w4-c67de2-kserve-868679cf9b-gqjbx" podUID="e104d7da-1fb2-4c99-a480-b4ae84c4f273" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:43:35.550558 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:35.387975    5060 patch_prober.go:29] interesting pod/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-56grg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 12:43:35.550852 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:35.550804    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-56grg" podUID="f43cdb3b-9bd6-46a7-a14c-feec07c2c755" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:43:35.740578 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:35.739588    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-2-w5-b25d3b-kserve-76f798d59f-vvzhg" podUID="394ed76c-84ac-4612-9acb-a7e36cb68d89" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": context deadline exceeded"
Oct 31 12:43:35.924523 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:35.920421    5060 patch_prober.go:29] interesting pod/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-l8n27 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 12:43:35.924523 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:35.920533    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-l8n27" podUID="c5c79f52-3049-4611-a935-ef2f4eb5e0bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:43:35.982619 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:35.982579    5060 patch_prober.go:29] interesting pod/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-sc6ff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body=
Oct 31 12:43:35.990524 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:35.982865    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-sc6ff" podUID="360843c7-795f-4f22-84cd-76317addf9c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused"
Oct 31 12:43:36.065648 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:36.065605    5060 patch_prober.go:29] interesting pod/e2e-worker-w1-85ce08-openshift-default-77d6458845-f8gp2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 12:43:36.085528 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:36.081567    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-f8gp2" podUID="df05433b-b6c1-4bd0-88a1-11e769461d69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:43:36.248555 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:36.247562    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x2qzv" podUID="c247ed2a-aa90-424b-9b7e-4f2936d67e3f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:43:36.283462 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:36.269665    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/premium-w2-e6c79d-kserve-6468bf46d4-h66f8" podUID="cd6a266f-8d12-4089-83ee-c75c716bf26c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Oct 31 12:43:36.349316 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:36.349225    5060 patch_prober.go:29] interesting pod/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-fcn5x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 12:43:36.425520 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:36.353160    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-fcn5x" podUID="aae46d99-cc6d-4be9-868b-8c4dda15a7a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:43:36.576539 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:36.514752    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/simulator-w4-c67de2-kserve-868679cf9b-gqjbx" podUID="e104d7da-1fb2-4c99-a480-b4ae84c4f273" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:43:36.604830 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:36.604734    5060 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-2-w5-b25d3b-kserve-76f798d59f-vvzhg" podUID="394ed76c-84ac-4612-9acb-a7e36cb68d89" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:43:36.679092 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:36.672413    5060 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-c67de2/simulator-w4-c67de2-kserve-868679cf9b-gqjbx" podUID="e104d7da-1fb2-4c99-a480-b4ae84c4f273" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:43:36.743714 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:36.743662    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-2-w5-b25d3b-kserve-76f798d59f-vvzhg" podUID="394ed76c-84ac-4612-9acb-a7e36cb68d89" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:43:36.796520 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:36.791603    5060 patch_prober.go:29] interesting pod/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-rr2hg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 12:43:36.796520 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:36.791670    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-rr2hg" podUID="15a47e5b-2ade-4ca3-b0d0-6e4c09917588" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:43:36.971142 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:36.971048    5060 patch_prober.go:29] interesting pod/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-sc6ff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body=
Oct 31 12:43:36.971408 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:36.971376    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-sc6ff" podUID="360843c7-795f-4f22-84cd-76317addf9c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused"
Oct 31 12:43:37.035072 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:37.035030    5060 patch_prober.go:29] interesting pod/e2e-worker-w1-85ce08-openshift-default-77d6458845-gxs26 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 12:43:37.035353 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:37.035315    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-gxs26" podUID="86c84864-127f-4507-8f60-2dbb684c687a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:43:37.270864 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:37.268427    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x2qzv" podUID="c247ed2a-aa90-424b-9b7e-4f2936d67e3f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded"
Oct 31 12:43:37.300289 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:37.285425    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/premium-w2-e6c79d-kserve-6468bf46d4-h66f8" podUID="cd6a266f-8d12-4089-83ee-c75c716bf26c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:43:37.559557 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:37.557309    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/simulator-w4-c67de2-kserve-868679cf9b-gqjbx" podUID="e104d7da-1fb2-4c99-a480-b4ae84c4f273" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:43:37.788283 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:37.788230    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-2-w5-b25d3b-kserve-76f798d59f-vvzhg" podUID="394ed76c-84ac-4612-9acb-a7e36cb68d89" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:43:38.155518 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:38.155332    5060 patch_prober.go:29] interesting pod/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-sc6ff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body=
Oct 31 12:43:38.155518 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:38.155408    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-sc6ff" podUID="360843c7-795f-4f22-84cd-76317addf9c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused"
Oct 31 12:43:38.359945 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:38.359894    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x2qzv" podUID="c247ed2a-aa90-424b-9b7e-4f2936d67e3f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:43:38.716735 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:38.591208    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/premium-w2-e6c79d-kserve-6468bf46d4-h66f8" podUID="cd6a266f-8d12-4089-83ee-c75c716bf26c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Oct 31 12:43:38.785225 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:38.785176    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/simulator-w4-c67de2-kserve-868679cf9b-gqjbx" podUID="e104d7da-1fb2-4c99-a480-b4ae84c4f273" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded"
Oct 31 12:43:38.805480 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:38.804798    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-2-w5-b25d3b-kserve-76f798d59f-vvzhg" podUID="394ed76c-84ac-4612-9acb-a7e36cb68d89" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:43:38.860674 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:38.860631    5060 patch_prober.go:29] interesting pod/e2e-worker-w3-b218d3-openshift-default-67596dd89f-2fn6t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 12:43:38.860983 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:38.860936    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-2fn6t" podUID="60d395e9-e4e9-4c68-ab10-9ef5371df1b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:43:38.911515 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:43:38.892003    5060 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.848s"
Oct 31 12:43:39.070635 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:39.060137    5060 patch_prober.go:29] interesting pod/e2e-worker-w3-b218d3-openshift-default-67596dd89f-28t2l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 12:43:39.070635 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:39.060228    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-28t2l" podUID="66295870-6cfc-4f48-b353-566fe73a6bf0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:43:39.337568 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:39.336236    5060 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-e6c79d/premium-w2-e6c79d-kserve-6468bf46d4-h66f8" podUID="cd6a266f-8d12-4089-83ee-c75c716bf26c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": context deadline exceeded"
Oct 31 12:43:39.361956 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:39.345707    5060 patch_prober.go:29] interesting pod/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-sc6ff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body=
Oct 31 12:43:39.361956 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:39.345770    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-sc6ff" podUID="360843c7-795f-4f22-84cd-76317addf9c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused"
Oct 31 12:43:39.403663 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:39.388819    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x2qzv" podUID="c247ed2a-aa90-424b-9b7e-4f2936d67e3f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:43:39.620145 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:39.620047    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/premium-w2-e6c79d-kserve-6468bf46d4-h66f8" podUID="cd6a266f-8d12-4089-83ee-c75c716bf26c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Oct 31 12:43:39.807134 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:39.807090    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-2-w5-b25d3b-kserve-76f798d59f-vvzhg" podUID="394ed76c-84ac-4612-9acb-a7e36cb68d89" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:43:39.842517 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:39.820907    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/simulator-w4-c67de2-kserve-868679cf9b-gqjbx" podUID="e104d7da-1fb2-4c99-a480-b4ae84c4f273" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded"
Oct 31 12:43:39.898572 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:39.887327    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-sc6ff"
Oct 31 12:43:40.153606 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:40.012945    5060 patch_prober.go:29] interesting pod/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-sc6ff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body=
Oct 31 12:43:40.201044 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:40.200800    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-sc6ff" podUID="360843c7-795f-4f22-84cd-76317addf9c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused"
Oct 31 12:43:40.397158 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:40.397111    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x2qzv" podUID="c247ed2a-aa90-424b-9b7e-4f2936d67e3f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:43:40.719444 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:40.716583    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/premium-w2-e6c79d-kserve-6468bf46d4-h66f8" podUID="cd6a266f-8d12-4089-83ee-c75c716bf26c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Oct 31 12:43:40.816520 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:40.811715    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-2-w5-b25d3b-kserve-76f798d59f-vvzhg" podUID="394ed76c-84ac-4612-9acb-a7e36cb68d89" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": context deadline exceeded"
Oct 31 12:43:40.823109 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:40.823068    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/simulator-w4-c67de2-kserve-868679cf9b-gqjbx" podUID="e104d7da-1fb2-4c99-a480-b4ae84c4f273" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded"
Oct 31 12:43:40.974703 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:40.974250    5060 patch_prober.go:29] interesting pod/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-sc6ff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body=
Oct 31 12:43:40.974703 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:40.974313    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-sc6ff" podUID="360843c7-795f-4f22-84cd-76317addf9c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused"
Oct 31 12:43:41.411516 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:41.397769    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x2qzv" podUID="c247ed2a-aa90-424b-9b7e-4f2936d67e3f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:43:41.758573 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:41.757594    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/premium-w2-e6c79d-kserve-6468bf46d4-h66f8" podUID="cd6a266f-8d12-4089-83ee-c75c716bf26c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:43:41.782571 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:41.776591    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c67de2/simulator-w4-c67de2-kserve-868679cf9b-gqjbx"
Oct 31 12:43:41.816181 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:41.815631    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-2-w5-b25d3b-kserve-76f798d59f-vvzhg" podUID="394ed76c-84ac-4612-9acb-a7e36cb68d89" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:43:42.058880 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:42.058839    5060 patch_prober.go:29] interesting pod/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-sc6ff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body=
Oct 31 12:43:42.059513 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:42.059461    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-sc6ff" podUID="360843c7-795f-4f22-84cd-76317addf9c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused"
Oct 31 12:43:42.154985 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:42.154942    5060 patch_prober.go:29] interesting pod/e2e-worker-w6-67d410-openshift-default-676b978549-pz8cf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 12:43:42.155242 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:42.155208    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-pz8cf" podUID="326d88b8-e041-4b4f-ab99-f1d2622d76b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:43:42.403594 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:42.383280    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e6c79d/premium-w2-e6c79d-kserve-6468bf46d4-h66f8"
Oct 31 12:43:42.403594 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:42.402569    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-59fd8a28-jtp8n"]
Oct 31 12:43:42.403594 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:42.402671    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x2qzv" podUID="c247ed2a-aa90-424b-9b7e-4f2936d67e3f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:43:42.430513 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:42.404206    5060 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x2qzv" podUID="c247ed2a-aa90-424b-9b7e-4f2936d67e3f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": context deadline exceeded"
Oct 31 12:43:42.644235 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:42.644151    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-59fd8a28-jtp8n"]
Oct 31 12:43:42.764704 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:42.764607    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/simulator-w4-c67de2-kserve-868679cf9b-gqjbx" podUID="e104d7da-1fb2-4c99-a480-b4ae84c4f273" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:43:42.819524 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:42.819190    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-2-w5-b25d3b-kserve-76f798d59f-vvzhg" podUID="394ed76c-84ac-4612-9acb-a7e36cb68d89" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:43:42.847391 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:42.847355    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-2-w5-b25d3b-kserve-76f798d59f-vvzhg"
Oct 31 12:43:42.973085 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:42.973050    5060 patch_prober.go:29] interesting pod/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-sc6ff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body=
Oct 31 12:43:42.973583 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:42.973320    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-sc6ff" podUID="360843c7-795f-4f22-84cd-76317addf9c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused"
Oct 31 12:43:43.091515 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:43.090635    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="335aa917-cbfc-4e05-8075-38e3a2f85155" path="/var/lib/kubelet/pods/335aa917-cbfc-4e05-8075-38e3a2f85155/volumes"
Oct 31 12:43:43.413648 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:43.406550    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x2qzv" podUID="c247ed2a-aa90-424b-9b7e-4f2936d67e3f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:43:43.500590 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:43.463132    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-43c7f8da-vdb55"]
Oct 31 12:43:43.514650 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:43.514611    5060 cpu_manager.go:414] "containerMap: removing container" podUID="5a85046f-2e8f-4dbf-8ce1-26f18bfda012" containerName="revoke-keys"
Oct 31 12:43:43.514901 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:43.514879    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a85046f-2e8f-4dbf-8ce1-26f18bfda012" containerName="revoke-keys"
Oct 31 12:43:43.515043 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:43.515030    5060 cpu_manager.go:414] "containerMap: removing container" podUID="3126c9e8-c907-422c-94b7-0c9baea649c9" containerName="revoke-keys"
Oct 31 12:43:43.515151 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:43.515137    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3126c9e8-c907-422c-94b7-0c9baea649c9" containerName="revoke-keys"
Oct 31 12:43:43.515329 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:43.515302    5060 cpu_manager.go:414] "containerMap: removing container" podUID="6ed56313-9c9d-4e0f-9764-c8b9dacdfda2" containerName="revoke-keys"
Oct 31 12:43:43.515439 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:43.515426    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ed56313-9c9d-4e0f-9764-c8b9dacdfda2" containerName="revoke-keys"
Oct 31 12:43:43.515911 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:43.515884    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a85046f-2e8f-4dbf-8ce1-26f18bfda012" containerName="revoke-keys"
Oct 31 12:43:43.516069 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:43.516054    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="3126c9e8-c907-422c-94b7-0c9baea649c9" containerName="revoke-keys"
Oct 31 12:43:43.516216 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:43.516203    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ed56313-9c9d-4e0f-9764-c8b9dacdfda2" containerName="revoke-keys"
Oct 31 12:43:43.527785 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:43.526861    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/premium-w2-e6c79d-kserve-6468bf46d4-h66f8" podUID="cd6a266f-8d12-4089-83ee-c75c716bf26c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:43:43.530739 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:43.530715    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-43c7f8da-vdb55"
Oct 31 12:43:43.538795 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:43.537844    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 12:43:43.538795 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:43.538538    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7wsmb\""
Oct 31 12:43:43.538795 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:43.538597    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 12:43:43.538795 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:43.538657    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 12:43:43.542190 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:43.540140    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x2qzv"
Oct 31 12:43:43.548831 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:43.548801    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-43c7f8da-vdb55"]
Oct 31 12:43:43.797515 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:43.786800    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1e03045-95b9-48cc-925f-d54675bf1e7b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-43c7f8da-vdb55\" (UID: \"f1e03045-95b9-48cc-925f-d54675bf1e7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-43c7f8da-vdb55"
Oct 31 12:43:43.797515 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:43.787007    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbznn\" (UniqueName: \"kubernetes.io/projected/f1e03045-95b9-48cc-925f-d54675bf1e7b-kube-api-access-hbznn\") pod \"maas-api-revoke-keys-e2e-migrate-43c7f8da-vdb55\" (UID: \"f1e03045-95b9-48cc-925f-d54675bf1e7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-43c7f8da-vdb55"
Oct 31 12:43:43.797515 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:43.787236    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1e03045-95b9-48cc-925f-d54675bf1e7b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-43c7f8da-vdb55\" (UID: \"f1e03045-95b9-48cc-925f-d54675bf1e7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-43c7f8da-vdb55"
Oct 31 12:43:43.990518 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:43.988707    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1e03045-95b9-48cc-925f-d54675bf1e7b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-43c7f8da-vdb55\" (UID: \"f1e03045-95b9-48cc-925f-d54675bf1e7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-43c7f8da-vdb55"
Oct 31 12:43:44.002515 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:43.995035    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbznn\" (UniqueName: \"kubernetes.io/projected/f1e03045-95b9-48cc-925f-d54675bf1e7b-kube-api-access-hbznn\") pod \"maas-api-revoke-keys-e2e-migrate-43c7f8da-vdb55\" (UID: \"f1e03045-95b9-48cc-925f-d54675bf1e7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-43c7f8da-vdb55"
Oct 31 12:43:44.002515 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:43.995271    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1e03045-95b9-48cc-925f-d54675bf1e7b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-43c7f8da-vdb55\" (UID: \"f1e03045-95b9-48cc-925f-d54675bf1e7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-43c7f8da-vdb55"
Oct 31 12:43:44.002515 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:43.996394    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1e03045-95b9-48cc-925f-d54675bf1e7b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-43c7f8da-vdb55\" (UID: \"f1e03045-95b9-48cc-925f-d54675bf1e7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-43c7f8da-vdb55"
Oct 31 12:43:44.002515 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:43.997247    5060 patch_prober.go:29] interesting pod/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-sc6ff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body=
Oct 31 12:43:44.002515 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:43.997294    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-sc6ff" podUID="360843c7-795f-4f22-84cd-76317addf9c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused"
Oct 31 12:43:44.103407 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:44.099417    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbznn\" (UniqueName: \"kubernetes.io/projected/f1e03045-95b9-48cc-925f-d54675bf1e7b-kube-api-access-hbznn\") pod \"maas-api-revoke-keys-e2e-migrate-43c7f8da-vdb55\" (UID: \"f1e03045-95b9-48cc-925f-d54675bf1e7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-43c7f8da-vdb55"
Oct 31 12:43:44.134469 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:44.124986    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1e03045-95b9-48cc-925f-d54675bf1e7b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-43c7f8da-vdb55\" (UID: \"f1e03045-95b9-48cc-925f-d54675bf1e7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-43c7f8da-vdb55"
Oct 31 12:43:44.246923 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:44.246885    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-43c7f8da-vdb55"
Oct 31 12:43:45.794568 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:45.771365    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-43c7f8da-vdb55"]
Oct 31 12:43:46.090594 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:46.088661    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w6-9b17b9-openshift-default-598d5bf87-rr64l"]
Oct 31 12:43:46.091442 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:46.091400    5060 patch_prober.go:29] interesting pod/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-sc6ff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 12:43:46.091676 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:46.091642    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-sc6ff" podUID="360843c7-795f-4f22-84cd-76317addf9c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:43:46.332934 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:46.332898    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-9b17b9-openshift-default-598d5bf87-rr64l"]
Oct 31 12:43:46.333102 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:46.332954    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-43c7f8da-vdb55" event={"ID":"f1e03045-95b9-48cc-925f-d54675bf1e7b","Type":"ContainerStarted","Data":"e1949d45f298544e521497a010ec142ac5f59c0ff4dbff226b94813390c564bc"}
Oct 31 12:43:46.333102 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:46.333056    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-9b17b9-openshift-default-598d5bf87-rr64l"
Oct 31 12:43:46.352613 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:46.352476    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcde778f-8f1a-4c1a-a497-6c66733d54ff-istiod-ca-cert\") pod \"e2e-shared-a-w6-9b17b9-openshift-default-598d5bf87-rr64l\" (UID: \"fcde778f-8f1a-4c1a-a497-6c66733d54ff\") " pod="openshift-ingress/e2e-shared-a-w6-9b17b9-openshift-default-598d5bf87-rr64l"
Oct 31 12:43:46.353034 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:46.352993    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gb8hv\" (UniqueName: \"kubernetes.io/projected/fcde778f-8f1a-4c1a-a497-6c66733d54ff-kube-api-access-gb8hv\") pod \"e2e-shared-a-w6-9b17b9-openshift-default-598d5bf87-rr64l\" (UID: \"fcde778f-8f1a-4c1a-a497-6c66733d54ff\") " pod="openshift-ingress/e2e-shared-a-w6-9b17b9-openshift-default-598d5bf87-rr64l"
Oct 31 12:43:46.353260 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:46.353231    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcde778f-8f1a-4c1a-a497-6c66733d54ff-workload-socket\") pod \"e2e-shared-a-w6-9b17b9-openshift-default-598d5bf87-rr64l\" (UID: \"fcde778f-8f1a-4c1a-a497-6c66733d54ff\") " pod="openshift-ingress/e2e-shared-a-w6-9b17b9-openshift-default-598d5bf87-rr64l"
Oct 31 12:43:46.353508 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:46.353465    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcde778f-8f1a-4c1a-a497-6c66733d54ff-istio-envoy\") pod \"e2e-shared-a-w6-9b17b9-openshift-default-598d5bf87-rr64l\" (UID: \"fcde778f-8f1a-4c1a-a497-6c66733d54ff\") " pod="openshift-ingress/e2e-shared-a-w6-9b17b9-openshift-default-598d5bf87-rr64l"
Oct 31 12:43:46.353646 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:46.353628    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fcde778f-8f1a-4c1a-a497-6c66733d54ff-istio-token\") pod \"e2e-shared-a-w6-9b17b9-openshift-default-598d5bf87-rr64l\" (UID: \"fcde778f-8f1a-4c1a-a497-6c66733d54ff\") " pod="openshift-ingress/e2e-shared-a-w6-9b17b9-openshift-default-598d5bf87-rr64l"
Oct 31 12:43:46.353783 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:46.353767    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcde778f-8f1a-4c1a-a497-6c66733d54ff-credential-socket\") pod \"e2e-shared-a-w6-9b17b9-openshift-default-598d5bf87-rr64l\" (UID: \"fcde778f-8f1a-4c1a-a497-6c66733d54ff\") " pod="openshift-ingress/e2e-shared-a-w6-9b17b9-openshift-default-598d5bf87-rr64l"
Oct 31 12:43:46.353965 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:46.353937    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcde778f-8f1a-4c1a-a497-6c66733d54ff-istio-podinfo\") pod \"e2e-shared-a-w6-9b17b9-openshift-default-598d5bf87-rr64l\" (UID: \"fcde778f-8f1a-4c1a-a497-6c66733d54ff\") " pod="openshift-ingress/e2e-shared-a-w6-9b17b9-openshift-default-598d5bf87-rr64l"
Oct 31 12:43:46.354202 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:46.354182    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcde778f-8f1a-4c1a-a497-6c66733d54ff-workload-certs\") pod \"e2e-shared-a-w6-9b17b9-openshift-default-598d5bf87-rr64l\" (UID: \"fcde778f-8f1a-4c1a-a497-6c66733d54ff\") " pod="openshift-ingress/e2e-shared-a-w6-9b17b9-openshift-default-598d5bf87-rr64l"
Oct 31 12:43:46.354290 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:46.354253    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcde778f-8f1a-4c1a-a497-6c66733d54ff-istio-data\") pod \"e2e-shared-a-w6-9b17b9-openshift-default-598d5bf87-rr64l\" (UID: \"fcde778f-8f1a-4c1a-a497-6c66733d54ff\") " pod="openshift-ingress/e2e-shared-a-w6-9b17b9-openshift-default-598d5bf87-rr64l"
Oct 31 12:43:46.385152 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:46.385120    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w6-9b17b9-openshift-default-dockercfg-ntnh6\""
Oct 31 12:43:46.503157 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:46.477213    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gb8hv\" (UniqueName: \"kubernetes.io/projected/fcde778f-8f1a-4c1a-a497-6c66733d54ff-kube-api-access-gb8hv\") pod \"e2e-shared-a-w6-9b17b9-openshift-default-598d5bf87-rr64l\" (UID: \"fcde778f-8f1a-4c1a-a497-6c66733d54ff\") " pod="openshift-ingress/e2e-shared-a-w6-9b17b9-openshift-default-598d5bf87-rr64l"
Oct 31 12:43:46.503157 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:46.477305    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcde778f-8f1a-4c1a-a497-6c66733d54ff-workload-socket\") pod \"e2e-shared-a-w6-9b17b9-openshift-default-598d5bf87-rr64l\" (UID: \"fcde778f-8f1a-4c1a-a497-6c66733d54ff\") " pod="openshift-ingress/e2e-shared-a-w6-9b17b9-openshift-default-598d5bf87-rr64l"
Oct 31 12:43:46.503157 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:46.477390    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcde778f-8f1a-4c1a-a497-6c66733d54ff-istio-envoy\") pod \"e2e-shared-a-w6-9b17b9-openshift-default-598d5bf87-rr64l\" (UID: \"fcde778f-8f1a-4c1a-a497-6c66733d54ff\") " pod="openshift-ingress/e2e-shared-a-w6-9b17b9-openshift-default-598d5bf87-rr64l"
Oct 31 12:43:46.503157 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:46.477574    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fcde778f-8f1a-4c1a-a497-6c66733d54ff-istio-token\") pod \"e2e-shared-a-w6-9b17b9-openshift-default-598d5bf87-rr64l\" (UID: \"fcde778f-8f1a-4c1a-a497-6c66733d54ff\") " pod="openshift-ingress/e2e-shared-a-w6-9b17b9-openshift-default-598d5bf87-rr64l"
Oct 31 12:43:46.503157 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:46.477652    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcde778f-8f1a-4c1a-a497-6c66733d54ff-credential-socket\") pod \"e2e-shared-a-w6-9b17b9-openshift-default-598d5bf87-rr64l\" (UID: \"fcde778f-8f1a-4c1a-a497-6c66733d54ff\") " pod="openshift-ingress/e2e-shared-a-w6-9b17b9-openshift-default-598d5bf87-rr64l"
Oct 31 12:43:46.503157 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:46.477734    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcde778f-8f1a-4c1a-a497-6c66733d54ff-istio-podinfo\") pod \"e2e-shared-a-w6-9b17b9-openshift-default-598d5bf87-rr64l\" (UID: \"fcde778f-8f1a-4c1a-a497-6c66733d54ff\") " pod="openshift-ingress/e2e-shared-a-w6-9b17b9-openshift-default-598d5bf87-rr64l"
Oct 31 12:43:46.503157 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:46.477856    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcde778f-8f1a-4c1a-a497-6c66733d54ff-workload-certs\") pod \"e2e-shared-a-w6-9b17b9-openshift-default-598d5bf87-rr64l\" (UID: \"fcde778f-8f1a-4c1a-a497-6c66733d54ff\") " pod="openshift-ingress/e2e-shared-a-w6-9b17b9-openshift-default-598d5bf87-rr64l"
Oct 31 12:43:46.503157 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:46.477924    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcde778f-8f1a-4c1a-a497-6c66733d54ff-istio-data\") pod \"e2e-shared-a-w6-9b17b9-openshift-default-598d5bf87-rr64l\" (UID: \"fcde778f-8f1a-4c1a-a497-6c66733d54ff\") " pod="openshift-ingress/e2e-shared-a-w6-9b17b9-openshift-default-598d5bf87-rr64l"
Oct 31 12:43:46.503157 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:46.478008    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcde778f-8f1a-4c1a-a497-6c66733d54ff-istiod-ca-cert\") pod \"e2e-shared-a-w6-9b17b9-openshift-default-598d5bf87-rr64l\" (UID: \"fcde778f-8f1a-4c1a-a497-6c66733d54ff\") " pod="openshift-ingress/e2e-shared-a-w6-9b17b9-openshift-default-598d5bf87-rr64l"
Oct 31 12:43:46.592222 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:46.585281    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcde778f-8f1a-4c1a-a497-6c66733d54ff-workload-socket\") pod \"e2e-shared-a-w6-9b17b9-openshift-default-598d5bf87-rr64l\" (UID: \"fcde778f-8f1a-4c1a-a497-6c66733d54ff\") " pod="openshift-ingress/e2e-shared-a-w6-9b17b9-openshift-default-598d5bf87-rr64l"
Oct 31 12:43:46.592222 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:46.591719    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcde778f-8f1a-4c1a-a497-6c66733d54ff-istio-data\") pod \"e2e-shared-a-w6-9b17b9-openshift-default-598d5bf87-rr64l\" (UID: \"fcde778f-8f1a-4c1a-a497-6c66733d54ff\") " pod="openshift-ingress/e2e-shared-a-w6-9b17b9-openshift-default-598d5bf87-rr64l"
Oct 31 12:43:46.592222 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:46.591956    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcde778f-8f1a-4c1a-a497-6c66733d54ff-credential-socket\") pod \"e2e-shared-a-w6-9b17b9-openshift-default-598d5bf87-rr64l\" (UID: \"fcde778f-8f1a-4c1a-a497-6c66733d54ff\") " pod="openshift-ingress/e2e-shared-a-w6-9b17b9-openshift-default-598d5bf87-rr64l"
Oct 31 12:43:46.598055 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:46.598023    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcde778f-8f1a-4c1a-a497-6c66733d54ff-istiod-ca-cert\") pod \"e2e-shared-a-w6-9b17b9-openshift-default-598d5bf87-rr64l\" (UID: \"fcde778f-8f1a-4c1a-a497-6c66733d54ff\") " pod="openshift-ingress/e2e-shared-a-w6-9b17b9-openshift-default-598d5bf87-rr64l"
Oct 31 12:43:46.693548 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:46.686740    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcde778f-8f1a-4c1a-a497-6c66733d54ff-workload-certs\") pod \"e2e-shared-a-w6-9b17b9-openshift-default-598d5bf87-rr64l\" (UID: \"fcde778f-8f1a-4c1a-a497-6c66733d54ff\") " pod="openshift-ingress/e2e-shared-a-w6-9b17b9-openshift-default-598d5bf87-rr64l"
Oct 31 12:43:47.194027 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:47.111446    5060 patch_prober.go:29] interesting pod/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-sc6ff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 12:43:47.194027 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:47.111563    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-sc6ff" podUID="360843c7-795f-4f22-84cd-76317addf9c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:43:47.194027 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:47.144956    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gb8hv\" (UniqueName: \"kubernetes.io/projected/fcde778f-8f1a-4c1a-a497-6c66733d54ff-kube-api-access-gb8hv\") pod \"e2e-shared-a-w6-9b17b9-openshift-default-598d5bf87-rr64l\" (UID: \"fcde778f-8f1a-4c1a-a497-6c66733d54ff\") " pod="openshift-ingress/e2e-shared-a-w6-9b17b9-openshift-default-598d5bf87-rr64l"
Oct 31 12:43:47.194027 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:47.145012    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcde778f-8f1a-4c1a-a497-6c66733d54ff-istio-podinfo\") pod \"e2e-shared-a-w6-9b17b9-openshift-default-598d5bf87-rr64l\" (UID: \"fcde778f-8f1a-4c1a-a497-6c66733d54ff\") " pod="openshift-ingress/e2e-shared-a-w6-9b17b9-openshift-default-598d5bf87-rr64l"
Oct 31 12:43:47.194027 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:47.145218    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcde778f-8f1a-4c1a-a497-6c66733d54ff-istio-envoy\") pod \"e2e-shared-a-w6-9b17b9-openshift-default-598d5bf87-rr64l\" (UID: \"fcde778f-8f1a-4c1a-a497-6c66733d54ff\") " pod="openshift-ingress/e2e-shared-a-w6-9b17b9-openshift-default-598d5bf87-rr64l"
Oct 31 12:43:47.206153 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:47.206079    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fcde778f-8f1a-4c1a-a497-6c66733d54ff-istio-token\") pod \"e2e-shared-a-w6-9b17b9-openshift-default-598d5bf87-rr64l\" (UID: \"fcde778f-8f1a-4c1a-a497-6c66733d54ff\") " pod="openshift-ingress/e2e-shared-a-w6-9b17b9-openshift-default-598d5bf87-rr64l"
Oct 31 12:43:47.545852 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:47.545805    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/premium-w2-e6c79d-kserve-6468bf46d4-h66f8" podUID="cd6a266f-8d12-4089-83ee-c75c716bf26c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:43:47.572466 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:47.549939    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-9b17b9-openshift-default-598d5bf87-rr64l"
Oct 31 12:43:47.720181 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:47.687987    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-2-w5-b25d3b-kserve-76f798d59f-vvzhg" podUID="394ed76c-84ac-4612-9acb-a7e36cb68d89" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": context deadline exceeded"
Oct 31 12:43:48.335527 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:48.334771    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x2qzv" podUID="c247ed2a-aa90-424b-9b7e-4f2936d67e3f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded"
Oct 31 12:43:48.429401 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:48.429354    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/simulator-w4-c67de2-kserve-868679cf9b-gqjbx" podUID="e104d7da-1fb2-4c99-a480-b4ae84c4f273" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:43:48.631753 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:48.631643    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-rm7cl" podUID="8292eaa8-0a02-46be-8654-3c4136ff604f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:43:48.777242 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:48.777195    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/premium-w2-e6c79d-kserve-6468bf46d4-h66f8" podUID="cd6a266f-8d12-4089-83ee-c75c716bf26c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:43:48.850516 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:43:48.838096    5060 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.754s"
Oct 31 12:43:48.850516 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:48.838276    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-sc6ff"
Oct 31 12:43:48.850516 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:48.838382    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e6c79d/premium-w2-e6c79d-kserve-6468bf46d4-h66f8"
Oct 31 12:43:48.850516 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:48.838404    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d5e8c3f2-pz9b6"]
Oct 31 12:43:48.850516 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:48.838441    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d5e8c3f2-pz9b6"]
Oct 31 12:43:48.850516 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:48.838652    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-d5e8c3f2-openshift-default-5dc86f657c-j4wrx"]
Oct 31 12:43:48.850516 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:48.838702    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b849a4c4-s8d6j"]
Oct 31 12:43:48.850516 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:48.838943    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-d5e8c3f2-openshift-default-5dc86f657c-j4wrx" podUID="f135ca99-10b3-4413-8d5f-1541b2aae5cf" containerName="istio-proxy" containerID="cri-o://a3ec3fae13ce05f9dfd02a3b10d4fc37bc7f4f5c366ef5d2f09480420a60f3eb" gracePeriod=30
Oct 31 12:43:48.966514 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:48.966201    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-2-w5-b25d3b-kserve-76f798d59f-vvzhg" podUID="394ed76c-84ac-4612-9acb-a7e36cb68d89" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:43:48.971912 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:48.967587    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-2-w5-b25d3b-kserve-76f798d59f-vvzhg"
Oct 31 12:43:49.063514 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:49.063088    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-43c7f8da-vdb55" event={"ID":"f1e03045-95b9-48cc-925f-d54675bf1e7b","Type":"ContainerStarted","Data":"8cb07de612b2bc92d1780daf393195acb445a1c1eda2cea0a134626b8b26ba58"}
Oct 31 12:43:49.454247 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:49.454210    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3126c9e8-c907-422c-94b7-0c9baea649c9" path="/var/lib/kubelet/pods/3126c9e8-c907-422c-94b7-0c9baea649c9/volumes"
Oct 31 12:43:49.462514 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:49.454817    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-sc6ff"
Oct 31 12:43:49.462514 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:49.454885    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-2-w5-b25d3b-kserve-76f798d59f-vvzhg"
Oct 31 12:43:49.462514 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:49.454917    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e6c79d/premium-w2-e6c79d-kserve-6468bf46d4-h66f8"
Oct 31 12:43:49.462514 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:49.454936    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b849a4c4-s8d6j"]
Oct 31 12:43:49.689811 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:49.688555    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-b849a4c4-openshift-default-76d4f9446c-blrcp"]
Oct 31 12:43:49.689811 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:49.688847    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-b849a4c4-openshift-default-76d4f9446c-blrcp" podUID="fd869a1f-8f33-4d47-86a9-b01a13d2c088" containerName="istio-proxy" containerID="cri-o://1dde22342826df50733b8414dca10bcbb0a90199ec0164bb228c2c3528e2e25b" gracePeriod=30
Oct 31 12:43:50.549522 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:50.511980    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:43:50.549522 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:50.512108    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:43:50.549522 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:50.512159    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:43:50.726695 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:50.726660    5060 generic.go:356] "Generic (PLEG): container finished" podID="f1e03045-95b9-48cc-925f-d54675bf1e7b" containerID="8cb07de612b2bc92d1780daf393195acb445a1c1eda2cea0a134626b8b26ba58" exitCode=0
Oct 31 12:43:50.729257 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:50.729222    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-43c7f8da-vdb55" event={"ID":"f1e03045-95b9-48cc-925f-d54675bf1e7b","Type":"ContainerDied","Data":"8cb07de612b2bc92d1780daf393195acb445a1c1eda2cea0a134626b8b26ba58"}
Oct 31 12:43:50.732708 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:43:50.732670    5060 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1e03045_95b9_48cc_925f_d54675bf1e7b.slice/crio-8cb07de612b2bc92d1780daf393195acb445a1c1eda2cea0a134626b8b26ba58.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1e03045_95b9_48cc_925f_d54675bf1e7b.slice/crio-conmon-8cb07de612b2bc92d1780daf393195acb445a1c1eda2cea0a134626b8b26ba58.scope\": RecentStats: unable to find data in memory cache]"
Oct 31 12:43:50.820613 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:50.820575    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-9b17b9-openshift-default-598d5bf87-rr64l"]
Oct 31 12:43:51.060409 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:51.059860    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ed56313-9c9d-4e0f-9764-c8b9dacdfda2" path="/var/lib/kubelet/pods/6ed56313-9c9d-4e0f-9764-c8b9dacdfda2/volumes"
Oct 31 12:43:51.157516 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:51.153680    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-43c7f8da-vdb55"
Oct 31 12:43:51.321519 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:51.321272    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1e03045-95b9-48cc-925f-d54675bf1e7b-kube-api-access-hbznn\" (UniqueName: \"kubernetes.io/projected/f1e03045-95b9-48cc-925f-d54675bf1e7b-kube-api-access-hbznn\") pod \"f1e03045-95b9-48cc-925f-d54675bf1e7b\" (UID: \"f1e03045-95b9-48cc-925f-d54675bf1e7b\") "
Oct 31 12:43:51.321519 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:51.321373    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f1e03045-95b9-48cc-925f-d54675bf1e7b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1e03045-95b9-48cc-925f-d54675bf1e7b-maas-api-xks-ca\") pod \"f1e03045-95b9-48cc-925f-d54675bf1e7b\" (UID: \"f1e03045-95b9-48cc-925f-d54675bf1e7b\") "
Oct 31 12:43:51.321519 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:51.321425    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1e03045-95b9-48cc-925f-d54675bf1e7b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1e03045-95b9-48cc-925f-d54675bf1e7b-maas-api-openshift-ca\") pod \"f1e03045-95b9-48cc-925f-d54675bf1e7b\" (UID: \"f1e03045-95b9-48cc-925f-d54675bf1e7b\") "
Oct 31 12:43:51.341162 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:51.337135    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1e03045-95b9-48cc-925f-d54675bf1e7b-maas-api-openshift-ca" pod "f1e03045-95b9-48cc-925f-d54675bf1e7b" (UID: "f1e03045-95b9-48cc-925f-d54675bf1e7b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:43:51.366384 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:51.362677    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1e03045-95b9-48cc-925f-d54675bf1e7b-maas-api-xks-ca" pod "f1e03045-95b9-48cc-925f-d54675bf1e7b" (UID: "f1e03045-95b9-48cc-925f-d54675bf1e7b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:43:51.418519 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:51.417142    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1e03045-95b9-48cc-925f-d54675bf1e7b-kube-api-access-hbznn" pod "f1e03045-95b9-48cc-925f-d54675bf1e7b" (UID: "f1e03045-95b9-48cc-925f-d54675bf1e7b"). InnerVolumeSpecName "kube-api-access-hbznn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:43:51.442517 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:51.438750    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbznn\" (UniqueName: \"kubernetes.io/projected/f1e03045-95b9-48cc-925f-d54675bf1e7b-kube-api-access-hbznn\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:43:51.442517 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:51.438790    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1e03045-95b9-48cc-925f-d54675bf1e7b-maas-api-xks-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:43:51.442517 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:51.438808    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1e03045-95b9-48cc-925f-d54675bf1e7b-maas-api-openshift-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:43:51.896621 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:51.896543    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-9b17b9-openshift-default-598d5bf87-rr64l" event={"ID":"fcde778f-8f1a-4c1a-a497-6c66733d54ff","Type":"ContainerStarted","Data":"be48cd4e14df08136b036625e90d6dac400cba28c17756e71993064162ace3e4"}
Oct 31 12:43:51.897248 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:51.897220    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-9b17b9-openshift-default-598d5bf87-rr64l" event={"ID":"fcde778f-8f1a-4c1a-a497-6c66733d54ff","Type":"ContainerStarted","Data":"05c06bd58510d98086307a019cd0561d638b4de7e331357b49880153f1df33ee"}
Oct 31 12:43:51.901539 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:51.901510    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-43c7f8da-vdb55" event={"ID":"f1e03045-95b9-48cc-925f-d54675bf1e7b","Type":"ContainerDied","Data":"e1949d45f298544e521497a010ec142ac5f59c0ff4dbff226b94813390c564bc"}
Oct 31 12:43:51.901761 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:51.901745    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e1949d45f298544e521497a010ec142ac5f59c0ff4dbff226b94813390c564bc"
Oct 31 12:43:51.901951 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:51.901936    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-43c7f8da-vdb55"
Oct 31 12:43:52.198581 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:52.195704    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w6-9b17b9-openshift-default-598d5bf87-rr64l" podStartSLOduration=7.195683993 podStartE2EDuration="7.195683993s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:43:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:43:51.99698685 +0000 UTC m=+995.100054566" watchObservedRunningTime="2026-10-31 12:43:52.195683993 +0000 UTC m=+995.298751705"
Oct 31 12:43:52.598300 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:52.598260    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-9b17b9-openshift-default-598d5bf87-rr64l"
Oct 31 12:43:52.604896 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:52.604867    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-9b17b9-openshift-default-598d5bf87-rr64l"
Oct 31 12:43:52.913946 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:52.913863    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-9b17b9-openshift-default-598d5bf87-rr64l"
Oct 31 12:43:52.915039 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:52.915016    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-9b17b9-openshift-default-598d5bf87-rr64l"
Oct 31 12:43:59.892726 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:59.887533    5060 patch_prober.go:29] interesting pod/e2e-ait-b849a4c4-openshift-default-76d4f9446c-blrcp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": read tcp 10.132.0.2:34278->10.132.0.154:15021: read: connection reset by peer" start-of-body=
Oct 31 12:43:59.892726 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:43:59.887624    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-b849a4c4-openshift-default-76d4f9446c-blrcp" podUID="fd869a1f-8f33-4d47-86a9-b01a13d2c088" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": read tcp 10.132.0.2:34278->10.132.0.154:15021: read: connection reset by peer"
Oct 31 12:43:59.971261 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:43:59.971225    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7phb" containerName="extractor"
Oct 31 12:44:00.242253 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:00.165045    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b849a4c4-openshift-default-76d4f9446c-blrcp"
Oct 31 12:44:00.288409 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:00.288374    5060 generic.go:356] "Generic (PLEG): container finished" podID="fd869a1f-8f33-4d47-86a9-b01a13d2c088" containerID="1dde22342826df50733b8414dca10bcbb0a90199ec0164bb228c2c3528e2e25b" exitCode=0
Oct 31 12:44:00.288637 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:00.288504    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b849a4c4-openshift-default-76d4f9446c-blrcp"
Oct 31 12:44:00.288637 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:00.288517    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b849a4c4-openshift-default-76d4f9446c-blrcp" event={"ID":"fd869a1f-8f33-4d47-86a9-b01a13d2c088","Type":"ContainerDied","Data":"1dde22342826df50733b8414dca10bcbb0a90199ec0164bb228c2c3528e2e25b"}
Oct 31 12:44:00.288637 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:00.288576    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b849a4c4-openshift-default-76d4f9446c-blrcp" event={"ID":"fd869a1f-8f33-4d47-86a9-b01a13d2c088","Type":"ContainerDied","Data":"3c97f87061c81292d08d86726d3daab6252884446997656305e9f1d56ee4ee16"}
Oct 31 12:44:00.288637 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:00.288615    5060 scope.go:122] "RemoveContainer" containerID="1dde22342826df50733b8414dca10bcbb0a90199ec0164bb228c2c3528e2e25b"
Oct 31 12:44:00.309827 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:00.309803    5060 scope.go:122] "RemoveContainer" containerID="1dde22342826df50733b8414dca10bcbb0a90199ec0164bb228c2c3528e2e25b"
Oct 31 12:44:00.310180 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:44:00.310158    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1dde22342826df50733b8414dca10bcbb0a90199ec0164bb228c2c3528e2e25b\": container with ID starting with 1dde22342826df50733b8414dca10bcbb0a90199ec0164bb228c2c3528e2e25b not found: ID does not exist" containerID="1dde22342826df50733b8414dca10bcbb0a90199ec0164bb228c2c3528e2e25b"
Oct 31 12:44:00.310257 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:00.310188    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1dde22342826df50733b8414dca10bcbb0a90199ec0164bb228c2c3528e2e25b"} err="failed to get container status \"1dde22342826df50733b8414dca10bcbb0a90199ec0164bb228c2c3528e2e25b\": rpc error: code = NotFound desc = could not find container \"1dde22342826df50733b8414dca10bcbb0a90199ec0164bb228c2c3528e2e25b\": container with ID starting with 1dde22342826df50733b8414dca10bcbb0a90199ec0164bb228c2c3528e2e25b not found: ID does not exist"
Oct 31 12:44:00.314726 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:00.314706    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd869a1f-8f33-4d47-86a9-b01a13d2c088-istio-token\" (UniqueName: \"kubernetes.io/projected/fd869a1f-8f33-4d47-86a9-b01a13d2c088-istio-token\") pod \"fd869a1f-8f33-4d47-86a9-b01a13d2c088\" (UID: \"fd869a1f-8f33-4d47-86a9-b01a13d2c088\") "
Oct 31 12:44:00.314843 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:00.314754    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd869a1f-8f33-4d47-86a9-b01a13d2c088-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd869a1f-8f33-4d47-86a9-b01a13d2c088-credential-socket\") pod \"fd869a1f-8f33-4d47-86a9-b01a13d2c088\" (UID: \"fd869a1f-8f33-4d47-86a9-b01a13d2c088\") "
Oct 31 12:44:00.314843 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:00.314773    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd869a1f-8f33-4d47-86a9-b01a13d2c088-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd869a1f-8f33-4d47-86a9-b01a13d2c088-istio-data\") pod \"fd869a1f-8f33-4d47-86a9-b01a13d2c088\" (UID: \"fd869a1f-8f33-4d47-86a9-b01a13d2c088\") "
Oct 31 12:44:00.314974 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:00.314886    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd869a1f-8f33-4d47-86a9-b01a13d2c088-kube-api-access-p66xk\" (UniqueName: \"kubernetes.io/projected/fd869a1f-8f33-4d47-86a9-b01a13d2c088-kube-api-access-p66xk\") pod \"fd869a1f-8f33-4d47-86a9-b01a13d2c088\" (UID: \"fd869a1f-8f33-4d47-86a9-b01a13d2c088\") "
Oct 31 12:44:00.315053 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:00.315026    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd869a1f-8f33-4d47-86a9-b01a13d2c088-credential-socket" pod "fd869a1f-8f33-4d47-86a9-b01a13d2c088" (UID: "fd869a1f-8f33-4d47-86a9-b01a13d2c088"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:44:00.315103 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:00.315034    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd869a1f-8f33-4d47-86a9-b01a13d2c088-istio-data" pod "fd869a1f-8f33-4d47-86a9-b01a13d2c088" (UID: "fd869a1f-8f33-4d47-86a9-b01a13d2c088"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:44:00.315245 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:00.315225    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd869a1f-8f33-4d47-86a9-b01a13d2c088-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd869a1f-8f33-4d47-86a9-b01a13d2c088-workload-socket\") pod \"fd869a1f-8f33-4d47-86a9-b01a13d2c088\" (UID: \"fd869a1f-8f33-4d47-86a9-b01a13d2c088\") "
Oct 31 12:44:00.315313 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:00.315262    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fd869a1f-8f33-4d47-86a9-b01a13d2c088-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd869a1f-8f33-4d47-86a9-b01a13d2c088-istio-podinfo\") pod \"fd869a1f-8f33-4d47-86a9-b01a13d2c088\" (UID: \"fd869a1f-8f33-4d47-86a9-b01a13d2c088\") "
Oct 31 12:44:00.315374 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:00.315322    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd869a1f-8f33-4d47-86a9-b01a13d2c088-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd869a1f-8f33-4d47-86a9-b01a13d2c088-istio-envoy\") pod \"fd869a1f-8f33-4d47-86a9-b01a13d2c088\" (UID: \"fd869a1f-8f33-4d47-86a9-b01a13d2c088\") "
Oct 31 12:44:00.315475 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:00.315384    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd869a1f-8f33-4d47-86a9-b01a13d2c088-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd869a1f-8f33-4d47-86a9-b01a13d2c088-workload-certs\") pod \"fd869a1f-8f33-4d47-86a9-b01a13d2c088\" (UID: \"fd869a1f-8f33-4d47-86a9-b01a13d2c088\") "
Oct 31 12:44:00.315475 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:00.315421    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd869a1f-8f33-4d47-86a9-b01a13d2c088-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd869a1f-8f33-4d47-86a9-b01a13d2c088-istiod-ca-cert\") pod \"fd869a1f-8f33-4d47-86a9-b01a13d2c088\" (UID: \"fd869a1f-8f33-4d47-86a9-b01a13d2c088\") "
Oct 31 12:44:00.315643 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:00.315483    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd869a1f-8f33-4d47-86a9-b01a13d2c088-workload-socket" pod "fd869a1f-8f33-4d47-86a9-b01a13d2c088" (UID: "fd869a1f-8f33-4d47-86a9-b01a13d2c088"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:44:00.315753 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:00.315726    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd869a1f-8f33-4d47-86a9-b01a13d2c088-workload-certs" pod "fd869a1f-8f33-4d47-86a9-b01a13d2c088" (UID: "fd869a1f-8f33-4d47-86a9-b01a13d2c088"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:44:00.315882 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:00.315861    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd869a1f-8f33-4d47-86a9-b01a13d2c088-istiod-ca-cert" pod "fd869a1f-8f33-4d47-86a9-b01a13d2c088" (UID: "fd869a1f-8f33-4d47-86a9-b01a13d2c088"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:44:00.316196 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:00.316178    5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd869a1f-8f33-4d47-86a9-b01a13d2c088-workload-socket\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:44:00.316274 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:00.316203    5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd869a1f-8f33-4d47-86a9-b01a13d2c088-workload-certs\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:44:00.316274 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:00.316216    5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd869a1f-8f33-4d47-86a9-b01a13d2c088-istiod-ca-cert\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:44:00.316274 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:00.316230    5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd869a1f-8f33-4d47-86a9-b01a13d2c088-credential-socket\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:44:00.316274 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:00.316243    5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd869a1f-8f33-4d47-86a9-b01a13d2c088-istio-data\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:44:00.317423 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:00.317397    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd869a1f-8f33-4d47-86a9-b01a13d2c088-kube-api-access-p66xk" pod "fd869a1f-8f33-4d47-86a9-b01a13d2c088" (UID: "fd869a1f-8f33-4d47-86a9-b01a13d2c088"). InnerVolumeSpecName "kube-api-access-p66xk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:44:00.317935 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:00.317910    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd869a1f-8f33-4d47-86a9-b01a13d2c088-istio-token" pod "fd869a1f-8f33-4d47-86a9-b01a13d2c088" (UID: "fd869a1f-8f33-4d47-86a9-b01a13d2c088"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:44:00.318027 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:00.317986    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd869a1f-8f33-4d47-86a9-b01a13d2c088-istio-envoy" pod "fd869a1f-8f33-4d47-86a9-b01a13d2c088" (UID: "fd869a1f-8f33-4d47-86a9-b01a13d2c088"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:44:00.318122 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:00.318104    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fd869a1f-8f33-4d47-86a9-b01a13d2c088-istio-podinfo" pod "fd869a1f-8f33-4d47-86a9-b01a13d2c088" (UID: "fd869a1f-8f33-4d47-86a9-b01a13d2c088"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 12:44:00.417926 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:00.417845    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p66xk\" (UniqueName: \"kubernetes.io/projected/fd869a1f-8f33-4d47-86a9-b01a13d2c088-kube-api-access-p66xk\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:44:00.417926 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:00.417883    5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd869a1f-8f33-4d47-86a9-b01a13d2c088-istio-podinfo\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:44:00.417926 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:00.417899    5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd869a1f-8f33-4d47-86a9-b01a13d2c088-istio-envoy\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:44:00.417926 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:00.417915    5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd869a1f-8f33-4d47-86a9-b01a13d2c088-istio-token\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:44:00.571919 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:00.571871    5060 patch_prober.go:29] interesting pod/e2e-derive-d5e8c3f2-openshift-default-5dc86f657c-j4wrx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": read tcp 10.132.0.2:43842->10.132.0.153:15021: read: connection reset by peer" start-of-body=
Oct 31 12:44:00.572074 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:00.571958    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-d5e8c3f2-openshift-default-5dc86f657c-j4wrx" podUID="f135ca99-10b3-4413-8d5f-1541b2aae5cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": read tcp 10.132.0.2:43842->10.132.0.153:15021: read: connection reset by peer"
Oct 31 12:44:00.663000 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:00.662966    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-b849a4c4-openshift-default-76d4f9446c-blrcp"]
Oct 31 12:44:00.670232 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:00.670204    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-b849a4c4-openshift-default-76d4f9446c-blrcp"]
Oct 31 12:44:00.808612 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:00.808583    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-d5e8c3f2-openshift-default-5dc86f657c-j4wrx"
Oct 31 12:44:00.926019 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:00.925932    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f135ca99-10b3-4413-8d5f-1541b2aae5cf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f135ca99-10b3-4413-8d5f-1541b2aae5cf-istio-podinfo\") pod \"f135ca99-10b3-4413-8d5f-1541b2aae5cf\" (UID: \"f135ca99-10b3-4413-8d5f-1541b2aae5cf\") "
Oct 31 12:44:00.926019 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:00.925975    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f135ca99-10b3-4413-8d5f-1541b2aae5cf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f135ca99-10b3-4413-8d5f-1541b2aae5cf-istio-data\") pod \"f135ca99-10b3-4413-8d5f-1541b2aae5cf\" (UID: \"f135ca99-10b3-4413-8d5f-1541b2aae5cf\") "
Oct 31 12:44:00.926580 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:00.926065    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f135ca99-10b3-4413-8d5f-1541b2aae5cf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f135ca99-10b3-4413-8d5f-1541b2aae5cf-credential-socket\") pod \"f135ca99-10b3-4413-8d5f-1541b2aae5cf\" (UID: \"f135ca99-10b3-4413-8d5f-1541b2aae5cf\") "
Oct 31 12:44:00.926580 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:00.926193    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f135ca99-10b3-4413-8d5f-1541b2aae5cf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f135ca99-10b3-4413-8d5f-1541b2aae5cf-istio-envoy\") pod \"f135ca99-10b3-4413-8d5f-1541b2aae5cf\" (UID: \"f135ca99-10b3-4413-8d5f-1541b2aae5cf\") "
Oct 31 12:44:00.926580 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:00.926222    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f135ca99-10b3-4413-8d5f-1541b2aae5cf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f135ca99-10b3-4413-8d5f-1541b2aae5cf-workload-socket\") pod \"f135ca99-10b3-4413-8d5f-1541b2aae5cf\" (UID: \"f135ca99-10b3-4413-8d5f-1541b2aae5cf\") "
Oct 31 12:44:00.926580 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:00.926309    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f135ca99-10b3-4413-8d5f-1541b2aae5cf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f135ca99-10b3-4413-8d5f-1541b2aae5cf-workload-certs\") pod \"f135ca99-10b3-4413-8d5f-1541b2aae5cf\" (UID: \"f135ca99-10b3-4413-8d5f-1541b2aae5cf\") "
Oct 31 12:44:00.926580 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:00.926359    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f135ca99-10b3-4413-8d5f-1541b2aae5cf-istio-data" pod "f135ca99-10b3-4413-8d5f-1541b2aae5cf" (UID: "f135ca99-10b3-4413-8d5f-1541b2aae5cf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:44:00.926580 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:00.926388    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f135ca99-10b3-4413-8d5f-1541b2aae5cf-credential-socket" pod "f135ca99-10b3-4413-8d5f-1541b2aae5cf" (UID: "f135ca99-10b3-4413-8d5f-1541b2aae5cf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:44:00.926580 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:00.926397    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f135ca99-10b3-4413-8d5f-1541b2aae5cf-kube-api-access-v5j6b\" (UniqueName: \"kubernetes.io/projected/f135ca99-10b3-4413-8d5f-1541b2aae5cf-kube-api-access-v5j6b\") pod \"f135ca99-10b3-4413-8d5f-1541b2aae5cf\" (UID: \"f135ca99-10b3-4413-8d5f-1541b2aae5cf\") "
Oct 31 12:44:00.926580 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:00.926521    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f135ca99-10b3-4413-8d5f-1541b2aae5cf-istio-token\" (UniqueName: \"kubernetes.io/projected/f135ca99-10b3-4413-8d5f-1541b2aae5cf-istio-token\") pod \"f135ca99-10b3-4413-8d5f-1541b2aae5cf\" (UID: \"f135ca99-10b3-4413-8d5f-1541b2aae5cf\") "
Oct 31 12:44:00.926580 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:00.926575    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f135ca99-10b3-4413-8d5f-1541b2aae5cf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f135ca99-10b3-4413-8d5f-1541b2aae5cf-istiod-ca-cert\") pod \"f135ca99-10b3-4413-8d5f-1541b2aae5cf\" (UID: \"f135ca99-10b3-4413-8d5f-1541b2aae5cf\") "
Oct 31 12:44:00.927054 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:00.926639    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f135ca99-10b3-4413-8d5f-1541b2aae5cf-workload-socket" pod "f135ca99-10b3-4413-8d5f-1541b2aae5cf" (UID: "f135ca99-10b3-4413-8d5f-1541b2aae5cf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:44:00.927054 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:00.926655    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f135ca99-10b3-4413-8d5f-1541b2aae5cf-workload-certs" pod "f135ca99-10b3-4413-8d5f-1541b2aae5cf" (UID: "f135ca99-10b3-4413-8d5f-1541b2aae5cf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:44:00.927721 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:00.927413    5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f135ca99-10b3-4413-8d5f-1541b2aae5cf-istio-data\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:44:00.927721 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:00.927441    5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f135ca99-10b3-4413-8d5f-1541b2aae5cf-credential-socket\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:44:00.927721 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:00.927456    5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f135ca99-10b3-4413-8d5f-1541b2aae5cf-workload-socket\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:44:00.927721 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:00.927469    5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f135ca99-10b3-4413-8d5f-1541b2aae5cf-workload-certs\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:44:00.927721 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:00.927686    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f135ca99-10b3-4413-8d5f-1541b2aae5cf-istiod-ca-cert" pod "f135ca99-10b3-4413-8d5f-1541b2aae5cf" (UID: "f135ca99-10b3-4413-8d5f-1541b2aae5cf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:44:00.929173 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:00.929132    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f135ca99-10b3-4413-8d5f-1541b2aae5cf-kube-api-access-v5j6b" pod "f135ca99-10b3-4413-8d5f-1541b2aae5cf" (UID: "f135ca99-10b3-4413-8d5f-1541b2aae5cf"). InnerVolumeSpecName "kube-api-access-v5j6b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:44:00.929373 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:00.929347    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f135ca99-10b3-4413-8d5f-1541b2aae5cf-istio-envoy" pod "f135ca99-10b3-4413-8d5f-1541b2aae5cf" (UID: "f135ca99-10b3-4413-8d5f-1541b2aae5cf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:44:00.929448 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:00.929433    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f135ca99-10b3-4413-8d5f-1541b2aae5cf-istio-podinfo" pod "f135ca99-10b3-4413-8d5f-1541b2aae5cf" (UID: "f135ca99-10b3-4413-8d5f-1541b2aae5cf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 12:44:00.934893 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:00.934864    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f135ca99-10b3-4413-8d5f-1541b2aae5cf-istio-token" pod "f135ca99-10b3-4413-8d5f-1541b2aae5cf" (UID: "f135ca99-10b3-4413-8d5f-1541b2aae5cf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:44:01.029124 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:01.029091    5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f135ca99-10b3-4413-8d5f-1541b2aae5cf-istio-envoy\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:44:01.029124 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:01.029122    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v5j6b\" (UniqueName: \"kubernetes.io/projected/f135ca99-10b3-4413-8d5f-1541b2aae5cf-kube-api-access-v5j6b\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:44:01.029124 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:01.029133    5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f135ca99-10b3-4413-8d5f-1541b2aae5cf-istio-token\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:44:01.029368 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:01.029142    5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f135ca99-10b3-4413-8d5f-1541b2aae5cf-istiod-ca-cert\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:44:01.029368 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:01.029151    5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f135ca99-10b3-4413-8d5f-1541b2aae5cf-istio-podinfo\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:44:01.050318 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:01.050283    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd869a1f-8f33-4d47-86a9-b01a13d2c088" path="/var/lib/kubelet/pods/fd869a1f-8f33-4d47-86a9-b01a13d2c088/volumes"
Oct 31 12:44:01.318172 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:01.318131    5060 generic.go:356] "Generic (PLEG): container finished" podID="f135ca99-10b3-4413-8d5f-1541b2aae5cf" containerID="a3ec3fae13ce05f9dfd02a3b10d4fc37bc7f4f5c366ef5d2f09480420a60f3eb" exitCode=0
Oct 31 12:44:01.318360 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:01.318208    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-d5e8c3f2-openshift-default-5dc86f657c-j4wrx"
Oct 31 12:44:01.318430 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:01.318266    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-d5e8c3f2-openshift-default-5dc86f657c-j4wrx" event={"ID":"f135ca99-10b3-4413-8d5f-1541b2aae5cf","Type":"ContainerDied","Data":"a3ec3fae13ce05f9dfd02a3b10d4fc37bc7f4f5c366ef5d2f09480420a60f3eb"}
Oct 31 12:44:01.318568 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:01.318543    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-d5e8c3f2-openshift-default-5dc86f657c-j4wrx" event={"ID":"f135ca99-10b3-4413-8d5f-1541b2aae5cf","Type":"ContainerDied","Data":"6c175cf4dacc8bbfd2208d8e8e1ba9e8225838e6ddaf430f07e1b07d386a550b"}
Oct 31 12:44:01.318651 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:01.318587    5060 scope.go:122] "RemoveContainer" containerID="a3ec3fae13ce05f9dfd02a3b10d4fc37bc7f4f5c366ef5d2f09480420a60f3eb"
Oct 31 12:44:01.334472 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:01.333516    5060 scope.go:122] "RemoveContainer" containerID="a3ec3fae13ce05f9dfd02a3b10d4fc37bc7f4f5c366ef5d2f09480420a60f3eb"
Oct 31 12:44:01.335059 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:44:01.334957    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3ec3fae13ce05f9dfd02a3b10d4fc37bc7f4f5c366ef5d2f09480420a60f3eb\": container with ID starting with a3ec3fae13ce05f9dfd02a3b10d4fc37bc7f4f5c366ef5d2f09480420a60f3eb not found: ID does not exist" containerID="a3ec3fae13ce05f9dfd02a3b10d4fc37bc7f4f5c366ef5d2f09480420a60f3eb"
Oct 31 12:44:01.335059 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:01.335001    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3ec3fae13ce05f9dfd02a3b10d4fc37bc7f4f5c366ef5d2f09480420a60f3eb"} err="failed to get container status \"a3ec3fae13ce05f9dfd02a3b10d4fc37bc7f4f5c366ef5d2f09480420a60f3eb\": rpc error: code = NotFound desc = could not find container \"a3ec3fae13ce05f9dfd02a3b10d4fc37bc7f4f5c366ef5d2f09480420a60f3eb\": container with ID starting with a3ec3fae13ce05f9dfd02a3b10d4fc37bc7f4f5c366ef5d2f09480420a60f3eb not found: ID does not exist"
Oct 31 12:44:01.347444 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:01.347417    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-d5e8c3f2-openshift-default-5dc86f657c-j4wrx"]
Oct 31 12:44:01.356714 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:01.356688    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-d5e8c3f2-openshift-default-5dc86f657c-j4wrx"]
Oct 31 12:44:02.008521 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:02.002757    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-43c7f8da-vdb55"]
Oct 31 12:44:02.022510 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:02.019700    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-43c7f8da-vdb55"]
Oct 31 12:44:02.191514 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:02.184535    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-43c7f8da-gw-openshift-default-69d4944d9c-mxmg5"]
Oct 31 12:44:02.191514 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:02.184825    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-43c7f8da-gw-openshift-default-69d4944d9c-mxmg5" podUID="77382c0a-8546-4b30-bbc6-e21a77328be8" containerName="istio-proxy" containerID="cri-o://613fc368611ab8081f078d3777d92a99d93ea6301fb0bcd19a3baf52f1b22cc9" gracePeriod=30
Oct 31 12:44:03.067045 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:03.067006    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f135ca99-10b3-4413-8d5f-1541b2aae5cf" path="/var/lib/kubelet/pods/f135ca99-10b3-4413-8d5f-1541b2aae5cf/volumes"
Oct 31 12:44:03.087151 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:03.086289    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1e03045-95b9-48cc-925f-d54675bf1e7b" path="/var/lib/kubelet/pods/f1e03045-95b9-48cc-925f-d54675bf1e7b/volumes"
Oct 31 12:44:06.177896 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:06.177861    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3bf1e1f-lbw5g"]
Oct 31 12:44:06.178698 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:06.178679    5060 cpu_manager.go:414] "containerMap: removing container" podUID="f135ca99-10b3-4413-8d5f-1541b2aae5cf" containerName="istio-proxy"
Oct 31 12:44:06.178698 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:06.178697    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f135ca99-10b3-4413-8d5f-1541b2aae5cf" containerName="istio-proxy"
Oct 31 12:44:06.178861 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:06.178711    5060 cpu_manager.go:414] "containerMap: removing container" podUID="fd869a1f-8f33-4d47-86a9-b01a13d2c088" containerName="istio-proxy"
Oct 31 12:44:06.178861 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:06.178717    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd869a1f-8f33-4d47-86a9-b01a13d2c088" containerName="istio-proxy"
Oct 31 12:44:06.178861 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:06.178741    5060 cpu_manager.go:414] "containerMap: removing container" podUID="f1e03045-95b9-48cc-925f-d54675bf1e7b" containerName="revoke-keys"
Oct 31 12:44:06.178861 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:06.178747    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1e03045-95b9-48cc-925f-d54675bf1e7b" containerName="revoke-keys"
Oct 31 12:44:06.178861 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:06.178858    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="f135ca99-10b3-4413-8d5f-1541b2aae5cf" containerName="istio-proxy"
Oct 31 12:44:06.179113 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:06.178866    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd869a1f-8f33-4d47-86a9-b01a13d2c088" containerName="istio-proxy"
Oct 31 12:44:06.179113 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:06.178873    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1e03045-95b9-48cc-925f-d54675bf1e7b" containerName="revoke-keys"
Oct 31 12:44:06.183338 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:06.183071    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3bf1e1f-lbw5g"
Oct 31 12:44:06.186972 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:06.186941    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 12:44:06.187850 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:06.187823    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 12:44:06.196632 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:06.196605    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3bf1e1f-lbw5g"]
Oct 31 12:44:06.245025 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:06.244993    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 12:44:06.245196 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:06.245054    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7wsmb\""
Oct 31 12:44:06.265394 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:06.265359    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc3210d5-cef9-47d8-b4d1-986c62846911-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c3bf1e1f-lbw5g\" (UID: \"bc3210d5-cef9-47d8-b4d1-986c62846911\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3bf1e1f-lbw5g"
Oct 31 12:44:06.265563 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:06.265510    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc3210d5-cef9-47d8-b4d1-986c62846911-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c3bf1e1f-lbw5g\" (UID: \"bc3210d5-cef9-47d8-b4d1-986c62846911\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3bf1e1f-lbw5g"
Oct 31 12:44:06.265563 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:06.265542    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wxjw\" (UniqueName: \"kubernetes.io/projected/bc3210d5-cef9-47d8-b4d1-986c62846911-kube-api-access-6wxjw\") pod \"maas-api-revoke-sub-c3bf1e1f-lbw5g\" (UID: \"bc3210d5-cef9-47d8-b4d1-986c62846911\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3bf1e1f-lbw5g"
Oct 31 12:44:06.367755 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:06.367722    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc3210d5-cef9-47d8-b4d1-986c62846911-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c3bf1e1f-lbw5g\" (UID: \"bc3210d5-cef9-47d8-b4d1-986c62846911\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3bf1e1f-lbw5g"
Oct 31 12:44:06.367923 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:06.367808    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc3210d5-cef9-47d8-b4d1-986c62846911-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c3bf1e1f-lbw5g\" (UID: \"bc3210d5-cef9-47d8-b4d1-986c62846911\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3bf1e1f-lbw5g"
Oct 31 12:44:06.367923 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:06.367829    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wxjw\" (UniqueName: \"kubernetes.io/projected/bc3210d5-cef9-47d8-b4d1-986c62846911-kube-api-access-6wxjw\") pod \"maas-api-revoke-sub-c3bf1e1f-lbw5g\" (UID: \"bc3210d5-cef9-47d8-b4d1-986c62846911\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3bf1e1f-lbw5g"
Oct 31 12:44:06.368377 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:06.368351    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc3210d5-cef9-47d8-b4d1-986c62846911-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c3bf1e1f-lbw5g\" (UID: \"bc3210d5-cef9-47d8-b4d1-986c62846911\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3bf1e1f-lbw5g"
Oct 31 12:44:06.370244 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:06.370223    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc3210d5-cef9-47d8-b4d1-986c62846911-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c3bf1e1f-lbw5g\" (UID: \"bc3210d5-cef9-47d8-b4d1-986c62846911\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3bf1e1f-lbw5g"
Oct 31 12:44:06.380132 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:06.380101    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wxjw\" (UniqueName: \"kubernetes.io/projected/bc3210d5-cef9-47d8-b4d1-986c62846911-kube-api-access-6wxjw\") pod \"maas-api-revoke-sub-c3bf1e1f-lbw5g\" (UID: \"bc3210d5-cef9-47d8-b4d1-986c62846911\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3bf1e1f-lbw5g"
Oct 31 12:44:06.505644 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:06.505550    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3bf1e1f-lbw5g"
Oct 31 12:44:06.854888 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:44:06.854851    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc3210d5_cef9_47d8_b4d1_986c62846911.slice/crio-395df6301948b26dd9d964a805ab8f444aff562e308c1df1c37dea96e69c6f21 WatchSource:0}: Error finding container 395df6301948b26dd9d964a805ab8f444aff562e308c1df1c37dea96e69c6f21: Status 404 returned error can't find the container with id 395df6301948b26dd9d964a805ab8f444aff562e308c1df1c37dea96e69c6f21
Oct 31 12:44:06.858815 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:06.858785    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3bf1e1f-lbw5g"]
Oct 31 12:44:07.546549 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:07.545903    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-43c7f8da-gw-openshift-default-69d4944d9c-mxmg5"
Oct 31 12:44:07.666217 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:07.666172    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3bf1e1f-lbw5g" event={"ID":"bc3210d5-cef9-47d8-b4d1-986c62846911","Type":"ContainerStarted","Data":"713e0be1c0ac6a852d5adac9b6e465a1a779e5d5953887093396407ffb0a8215"}
Oct 31 12:44:07.666399 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:07.666243    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3bf1e1f-lbw5g" event={"ID":"bc3210d5-cef9-47d8-b4d1-986c62846911","Type":"ContainerStarted","Data":"395df6301948b26dd9d964a805ab8f444aff562e308c1df1c37dea96e69c6f21"}
Oct 31 12:44:07.669676 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:07.669635    5060 generic.go:356] "Generic (PLEG): container finished" podID="77382c0a-8546-4b30-bbc6-e21a77328be8" containerID="613fc368611ab8081f078d3777d92a99d93ea6301fb0bcd19a3baf52f1b22cc9" exitCode=0
Oct 31 12:44:07.669849 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:07.669689    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-43c7f8da-gw-openshift-default-69d4944d9c-mxmg5" event={"ID":"77382c0a-8546-4b30-bbc6-e21a77328be8","Type":"ContainerDied","Data":"613fc368611ab8081f078d3777d92a99d93ea6301fb0bcd19a3baf52f1b22cc9"}
Oct 31 12:44:07.669849 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:07.669790    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-43c7f8da-gw-openshift-default-69d4944d9c-mxmg5" event={"ID":"77382c0a-8546-4b30-bbc6-e21a77328be8","Type":"ContainerDied","Data":"92d0a54a15c14ec846ade8db47e50b92b59e99ae7a2e870dc88ab7500913e0b3"}
Oct 31 12:44:07.669979 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:07.669854    5060 scope.go:122] "RemoveContainer" containerID="613fc368611ab8081f078d3777d92a99d93ea6301fb0bcd19a3baf52f1b22cc9"
Oct 31 12:44:07.670748 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:07.670721    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-43c7f8da-gw-openshift-default-69d4944d9c-mxmg5"
Oct 31 12:44:07.690996 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:07.690959    5060 scope.go:122] "RemoveContainer" containerID="613fc368611ab8081f078d3777d92a99d93ea6301fb0bcd19a3baf52f1b22cc9"
Oct 31 12:44:07.692327 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:44:07.691228    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"613fc368611ab8081f078d3777d92a99d93ea6301fb0bcd19a3baf52f1b22cc9\": container with ID starting with 613fc368611ab8081f078d3777d92a99d93ea6301fb0bcd19a3baf52f1b22cc9 not found: ID does not exist" containerID="613fc368611ab8081f078d3777d92a99d93ea6301fb0bcd19a3baf52f1b22cc9"
Oct 31 12:44:07.692327 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:07.691294    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"613fc368611ab8081f078d3777d92a99d93ea6301fb0bcd19a3baf52f1b22cc9"} err="failed to get container status \"613fc368611ab8081f078d3777d92a99d93ea6301fb0bcd19a3baf52f1b22cc9\": rpc error: code = NotFound desc = could not find container \"613fc368611ab8081f078d3777d92a99d93ea6301fb0bcd19a3baf52f1b22cc9\": container with ID starting with 613fc368611ab8081f078d3777d92a99d93ea6301fb0bcd19a3baf52f1b22cc9 not found: ID does not exist"
Oct 31 12:44:07.695208 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:07.695165    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3bf1e1f-lbw5g" podStartSLOduration=1.695152652 podStartE2EDuration="1.695152652s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:44:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:44:07.690346709 +0000 UTC m=+1010.793414420" watchObservedRunningTime="2026-10-31 12:44:07.695152652 +0000 UTC m=+1010.798220365"
Oct 31 12:44:07.696678 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:07.696650    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77382c0a-8546-4b30-bbc6-e21a77328be8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77382c0a-8546-4b30-bbc6-e21a77328be8-credential-socket\") pod \"77382c0a-8546-4b30-bbc6-e21a77328be8\" (UID: \"77382c0a-8546-4b30-bbc6-e21a77328be8\") "
Oct 31 12:44:07.696791 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:07.696713    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77382c0a-8546-4b30-bbc6-e21a77328be8-istio-token\" (UniqueName: \"kubernetes.io/projected/77382c0a-8546-4b30-bbc6-e21a77328be8-istio-token\") pod \"77382c0a-8546-4b30-bbc6-e21a77328be8\" (UID: \"77382c0a-8546-4b30-bbc6-e21a77328be8\") "
Oct 31 12:44:07.696791 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:07.696758    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77382c0a-8546-4b30-bbc6-e21a77328be8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77382c0a-8546-4b30-bbc6-e21a77328be8-workload-certs\") pod \"77382c0a-8546-4b30-bbc6-e21a77328be8\" (UID: \"77382c0a-8546-4b30-bbc6-e21a77328be8\") "
Oct 31 12:44:07.696920 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:07.696829    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/77382c0a-8546-4b30-bbc6-e21a77328be8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77382c0a-8546-4b30-bbc6-e21a77328be8-istio-podinfo\") pod \"77382c0a-8546-4b30-bbc6-e21a77328be8\" (UID: \"77382c0a-8546-4b30-bbc6-e21a77328be8\") "
Oct 31 12:44:07.696920 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:07.696855    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77382c0a-8546-4b30-bbc6-e21a77328be8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77382c0a-8546-4b30-bbc6-e21a77328be8-istio-data\") pod \"77382c0a-8546-4b30-bbc6-e21a77328be8\" (UID: \"77382c0a-8546-4b30-bbc6-e21a77328be8\") "
Oct 31 12:44:07.697029 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:07.697001    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77382c0a-8546-4b30-bbc6-e21a77328be8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77382c0a-8546-4b30-bbc6-e21a77328be8-istiod-ca-cert\") pod \"77382c0a-8546-4b30-bbc6-e21a77328be8\" (UID: \"77382c0a-8546-4b30-bbc6-e21a77328be8\") "
Oct 31 12:44:07.697086 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:07.697041    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77382c0a-8546-4b30-bbc6-e21a77328be8-kube-api-access-bsnkz\" (UniqueName: \"kubernetes.io/projected/77382c0a-8546-4b30-bbc6-e21a77328be8-kube-api-access-bsnkz\") pod \"77382c0a-8546-4b30-bbc6-e21a77328be8\" (UID: \"77382c0a-8546-4b30-bbc6-e21a77328be8\") "
Oct 31 12:44:07.697159 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:07.697142    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77382c0a-8546-4b30-bbc6-e21a77328be8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77382c0a-8546-4b30-bbc6-e21a77328be8-workload-socket\") pod \"77382c0a-8546-4b30-bbc6-e21a77328be8\" (UID: \"77382c0a-8546-4b30-bbc6-e21a77328be8\") "
Oct 31 12:44:07.697228 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:07.697214    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77382c0a-8546-4b30-bbc6-e21a77328be8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77382c0a-8546-4b30-bbc6-e21a77328be8-istio-envoy\") pod \"77382c0a-8546-4b30-bbc6-e21a77328be8\" (UID: \"77382c0a-8546-4b30-bbc6-e21a77328be8\") "
Oct 31 12:44:07.713974 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:07.713936    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77382c0a-8546-4b30-bbc6-e21a77328be8-istiod-ca-cert" pod "77382c0a-8546-4b30-bbc6-e21a77328be8" (UID: "77382c0a-8546-4b30-bbc6-e21a77328be8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:44:07.714103 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:07.714042    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77382c0a-8546-4b30-bbc6-e21a77328be8-workload-socket" pod "77382c0a-8546-4b30-bbc6-e21a77328be8" (UID: "77382c0a-8546-4b30-bbc6-e21a77328be8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:44:07.714327 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:07.714302    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77382c0a-8546-4b30-bbc6-e21a77328be8-workload-certs" pod "77382c0a-8546-4b30-bbc6-e21a77328be8" (UID: "77382c0a-8546-4b30-bbc6-e21a77328be8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:44:07.714478 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:07.714454    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77382c0a-8546-4b30-bbc6-e21a77328be8-istio-data" pod "77382c0a-8546-4b30-bbc6-e21a77328be8" (UID: "77382c0a-8546-4b30-bbc6-e21a77328be8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:44:07.714727 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:07.714700    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77382c0a-8546-4b30-bbc6-e21a77328be8-credential-socket" pod "77382c0a-8546-4b30-bbc6-e21a77328be8" (UID: "77382c0a-8546-4b30-bbc6-e21a77328be8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:44:07.723511 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:07.722967    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77382c0a-8546-4b30-bbc6-e21a77328be8-istio-token" pod "77382c0a-8546-4b30-bbc6-e21a77328be8" (UID: "77382c0a-8546-4b30-bbc6-e21a77328be8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:44:07.725538 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:07.724549    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/77382c0a-8546-4b30-bbc6-e21a77328be8-istio-podinfo" pod "77382c0a-8546-4b30-bbc6-e21a77328be8" (UID: "77382c0a-8546-4b30-bbc6-e21a77328be8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 12:44:07.728155 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:07.728129    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77382c0a-8546-4b30-bbc6-e21a77328be8-istio-envoy" pod "77382c0a-8546-4b30-bbc6-e21a77328be8" (UID: "77382c0a-8546-4b30-bbc6-e21a77328be8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:44:07.729839 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:07.729817    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77382c0a-8546-4b30-bbc6-e21a77328be8-kube-api-access-bsnkz" pod "77382c0a-8546-4b30-bbc6-e21a77328be8" (UID: "77382c0a-8546-4b30-bbc6-e21a77328be8"). InnerVolumeSpecName "kube-api-access-bsnkz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:44:07.799800 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:07.799752    5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77382c0a-8546-4b30-bbc6-e21a77328be8-istio-podinfo\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:44:07.799800 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:07.799790    5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77382c0a-8546-4b30-bbc6-e21a77328be8-istio-data\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:44:07.799800 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:07.799806    5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77382c0a-8546-4b30-bbc6-e21a77328be8-istiod-ca-cert\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:44:07.800089 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:07.799821    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bsnkz\" (UniqueName: \"kubernetes.io/projected/77382c0a-8546-4b30-bbc6-e21a77328be8-kube-api-access-bsnkz\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:44:07.800089 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:07.799837    5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77382c0a-8546-4b30-bbc6-e21a77328be8-workload-socket\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:44:07.800089 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:07.799851    5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77382c0a-8546-4b30-bbc6-e21a77328be8-istio-envoy\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:44:07.800089 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:07.799865    5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77382c0a-8546-4b30-bbc6-e21a77328be8-credential-socket\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:44:07.800089 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:07.799879    5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77382c0a-8546-4b30-bbc6-e21a77328be8-istio-token\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:44:07.800089 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:07.799892    5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77382c0a-8546-4b30-bbc6-e21a77328be8-workload-certs\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:44:08.034942 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:08.034909    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-43c7f8da-gw-openshift-default-69d4944d9c-mxmg5"]
Oct 31 12:44:08.041726 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:08.041695    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-43c7f8da-gw-openshift-default-69d4944d9c-mxmg5"]
Oct 31 12:44:08.093721 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:08.093631    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-665v2"]
Oct 31 12:44:08.094733 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:08.094709    5060 cpu_manager.go:414] "containerMap: removing container" podUID="77382c0a-8546-4b30-bbc6-e21a77328be8" containerName="istio-proxy"
Oct 31 12:44:08.094733 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:08.094730    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77382c0a-8546-4b30-bbc6-e21a77328be8" containerName="istio-proxy"
Oct 31 12:44:08.094945 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:08.094930    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="77382c0a-8546-4b30-bbc6-e21a77328be8" containerName="istio-proxy"
Oct 31 12:44:08.099344 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:08.099326    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-665v2"
Oct 31 12:44:08.114025 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:08.113998    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-665v2"]
Oct 31 12:44:08.229734 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:08.229614    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmzjq\" (UniqueName: \"kubernetes.io/projected/982df9b0-91aa-407d-a838-85d26b8fd39f-kube-api-access-dmzjq\") pod \"istiod-openshift-gateway-7ddd9cfdc8-665v2\" (UID: \"982df9b0-91aa-407d-a838-85d26b8fd39f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-665v2"
Oct 31 12:44:08.229955 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:08.229929    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/982df9b0-91aa-407d-a838-85d26b8fd39f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-665v2\" (UID: \"982df9b0-91aa-407d-a838-85d26b8fd39f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-665v2"
Oct 31 12:44:08.230098 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:08.230000    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/982df9b0-91aa-407d-a838-85d26b8fd39f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-665v2\" (UID: \"982df9b0-91aa-407d-a838-85d26b8fd39f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-665v2"
Oct 31 12:44:08.230256 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:08.230232    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/982df9b0-91aa-407d-a838-85d26b8fd39f-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-665v2\" (UID: \"982df9b0-91aa-407d-a838-85d26b8fd39f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-665v2"
Oct 31 12:44:08.230350 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:08.230294    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/982df9b0-91aa-407d-a838-85d26b8fd39f-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-665v2\" (UID: \"982df9b0-91aa-407d-a838-85d26b8fd39f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-665v2"
Oct 31 12:44:08.230350 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:08.230341    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/982df9b0-91aa-407d-a838-85d26b8fd39f-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-665v2\" (UID: \"982df9b0-91aa-407d-a838-85d26b8fd39f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-665v2"
Oct 31 12:44:08.230555 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:08.230479    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/982df9b0-91aa-407d-a838-85d26b8fd39f-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-665v2\" (UID: \"982df9b0-91aa-407d-a838-85d26b8fd39f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-665v2"
Oct 31 12:44:08.332950 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:08.332907    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/982df9b0-91aa-407d-a838-85d26b8fd39f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-665v2\" (UID: \"982df9b0-91aa-407d-a838-85d26b8fd39f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-665v2"
Oct 31 12:44:08.333164 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:08.332994    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/982df9b0-91aa-407d-a838-85d26b8fd39f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-665v2\" (UID: \"982df9b0-91aa-407d-a838-85d26b8fd39f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-665v2"
Oct 31 12:44:08.333255 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:08.333128    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/982df9b0-91aa-407d-a838-85d26b8fd39f-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-665v2\" (UID: \"982df9b0-91aa-407d-a838-85d26b8fd39f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-665v2"
Oct 31 12:44:08.333322 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:08.333265    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/982df9b0-91aa-407d-a838-85d26b8fd39f-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-665v2\" (UID: \"982df9b0-91aa-407d-a838-85d26b8fd39f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-665v2"
Oct 31 12:44:08.333322 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:08.333312    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/982df9b0-91aa-407d-a838-85d26b8fd39f-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-665v2\" (UID: \"982df9b0-91aa-407d-a838-85d26b8fd39f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-665v2"
Oct 31 12:44:08.333438 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:08.333354    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/982df9b0-91aa-407d-a838-85d26b8fd39f-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-665v2\" (UID: \"982df9b0-91aa-407d-a838-85d26b8fd39f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-665v2"
Oct 31 12:44:08.333542 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:08.333481    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmzjq\" (UniqueName: \"kubernetes.io/projected/982df9b0-91aa-407d-a838-85d26b8fd39f-kube-api-access-dmzjq\") pod \"istiod-openshift-gateway-7ddd9cfdc8-665v2\" (UID: \"982df9b0-91aa-407d-a838-85d26b8fd39f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-665v2"
Oct 31 12:44:08.333824 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:08.333788    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/982df9b0-91aa-407d-a838-85d26b8fd39f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-665v2\" (UID: \"982df9b0-91aa-407d-a838-85d26b8fd39f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-665v2"
Oct 31 12:44:08.336015 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:08.335981    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/982df9b0-91aa-407d-a838-85d26b8fd39f-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-665v2\" (UID: \"982df9b0-91aa-407d-a838-85d26b8fd39f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-665v2"
Oct 31 12:44:08.336119 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:08.336077    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/982df9b0-91aa-407d-a838-85d26b8fd39f-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-665v2\" (UID: \"982df9b0-91aa-407d-a838-85d26b8fd39f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-665v2"
Oct 31 12:44:08.336119 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:08.336089    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/982df9b0-91aa-407d-a838-85d26b8fd39f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-665v2\" (UID: \"982df9b0-91aa-407d-a838-85d26b8fd39f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-665v2"
Oct 31 12:44:08.336341 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:08.336325    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/982df9b0-91aa-407d-a838-85d26b8fd39f-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-665v2\" (UID: \"982df9b0-91aa-407d-a838-85d26b8fd39f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-665v2"
Oct 31 12:44:08.345887 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:08.345821    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/982df9b0-91aa-407d-a838-85d26b8fd39f-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-665v2\" (UID: \"982df9b0-91aa-407d-a838-85d26b8fd39f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-665v2"
Oct 31 12:44:08.346531 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:08.346465    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmzjq\" (UniqueName: \"kubernetes.io/projected/982df9b0-91aa-407d-a838-85d26b8fd39f-kube-api-access-dmzjq\") pod \"istiod-openshift-gateway-7ddd9cfdc8-665v2\" (UID: \"982df9b0-91aa-407d-a838-85d26b8fd39f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-665v2"
Oct 31 12:44:08.418803 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:08.418773    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-665v2"
Oct 31 12:44:08.686282 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:08.686194    5060 generic.go:356] "Generic (PLEG): container finished" podID="bc3210d5-cef9-47d8-b4d1-986c62846911" containerID="713e0be1c0ac6a852d5adac9b6e465a1a779e5d5953887093396407ffb0a8215" exitCode=0
Oct 31 12:44:08.686791 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:08.686275    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3bf1e1f-lbw5g" event={"ID":"bc3210d5-cef9-47d8-b4d1-986c62846911","Type":"ContainerDied","Data":"713e0be1c0ac6a852d5adac9b6e465a1a779e5d5953887093396407ffb0a8215"}
Oct 31 12:44:08.739520 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:08.731370    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-665v2"]
Oct 31 12:44:08.819806 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:08.804711    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:44:08.819806 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:08.804808    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:44:09.148347 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:09.148313    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77382c0a-8546-4b30-bbc6-e21a77328be8" path="/var/lib/kubelet/pods/77382c0a-8546-4b30-bbc6-e21a77328be8/volumes"
Oct 31 12:44:09.713214 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:09.713145    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-665v2" event={"ID":"982df9b0-91aa-407d-a838-85d26b8fd39f","Type":"ContainerStarted","Data":"893e20820bd1cd47f78958ec0fd43f8a3c42451ebd6fe386b909888acbfb000c"}
Oct 31 12:44:09.714166 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:09.713222    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-665v2" event={"ID":"982df9b0-91aa-407d-a838-85d26b8fd39f","Type":"ContainerStarted","Data":"978c81e2c75664f4231650d452d36cb12ceec8ab8309e31fb2f5d5869b331bfb"}
Oct 31 12:44:09.714166 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:09.713266    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-665v2"
Oct 31 12:44:09.747100 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:09.747046    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-665v2" podStartSLOduration=1.7470326200000001 podStartE2EDuration="1.74703262s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:44:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:44:09.739438377 +0000 UTC m=+1012.842506089" watchObservedRunningTime="2026-10-31 12:44:09.74703262 +0000 UTC m=+1012.850100331"
Oct 31 12:44:10.121353 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:10.121318    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3bf1e1f-lbw5g"
Oct 31 12:44:10.172369 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:10.170143    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc3210d5-cef9-47d8-b4d1-986c62846911-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc3210d5-cef9-47d8-b4d1-986c62846911-maas-api-openshift-ca\") pod \"bc3210d5-cef9-47d8-b4d1-986c62846911\" (UID: \"bc3210d5-cef9-47d8-b4d1-986c62846911\") "
Oct 31 12:44:10.172369 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:10.170666    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc3210d5-cef9-47d8-b4d1-986c62846911-maas-api-openshift-ca" pod "bc3210d5-cef9-47d8-b4d1-986c62846911" (UID: "bc3210d5-cef9-47d8-b4d1-986c62846911"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:44:10.172369 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:10.170866    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bc3210d5-cef9-47d8-b4d1-986c62846911-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc3210d5-cef9-47d8-b4d1-986c62846911-maas-api-xks-ca\") pod \"bc3210d5-cef9-47d8-b4d1-986c62846911\" (UID: \"bc3210d5-cef9-47d8-b4d1-986c62846911\") "
Oct 31 12:44:10.172369 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:10.170986    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc3210d5-cef9-47d8-b4d1-986c62846911-kube-api-access-6wxjw\" (UniqueName: \"kubernetes.io/projected/bc3210d5-cef9-47d8-b4d1-986c62846911-kube-api-access-6wxjw\") pod \"bc3210d5-cef9-47d8-b4d1-986c62846911\" (UID: \"bc3210d5-cef9-47d8-b4d1-986c62846911\") "
Oct 31 12:44:10.172369 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:10.171716    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc3210d5-cef9-47d8-b4d1-986c62846911-maas-api-openshift-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:44:10.175721 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:10.175694    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc3210d5-cef9-47d8-b4d1-986c62846911-maas-api-xks-ca" pod "bc3210d5-cef9-47d8-b4d1-986c62846911" (UID: "bc3210d5-cef9-47d8-b4d1-986c62846911"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:44:10.190325 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:10.190281    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc3210d5-cef9-47d8-b4d1-986c62846911-kube-api-access-6wxjw" pod "bc3210d5-cef9-47d8-b4d1-986c62846911" (UID: "bc3210d5-cef9-47d8-b4d1-986c62846911"). InnerVolumeSpecName "kube-api-access-6wxjw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:44:10.273796 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:10.273759    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6wxjw\" (UniqueName: \"kubernetes.io/projected/bc3210d5-cef9-47d8-b4d1-986c62846911-kube-api-access-6wxjw\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:44:10.273796 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:10.273791    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc3210d5-cef9-47d8-b4d1-986c62846911-maas-api-xks-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:44:10.723861 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:10.723832    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3bf1e1f-lbw5g"
Oct 31 12:44:10.724298 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:10.723872    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3bf1e1f-lbw5g" event={"ID":"bc3210d5-cef9-47d8-b4d1-986c62846911","Type":"ContainerDied","Data":"395df6301948b26dd9d964a805ab8f444aff562e308c1df1c37dea96e69c6f21"}
Oct 31 12:44:10.724298 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:10.723938    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="395df6301948b26dd9d964a805ab8f444aff562e308c1df1c37dea96e69c6f21"
Oct 31 12:44:10.726170 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:10.726152    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-665v2"
Oct 31 12:44:12.040844 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:12.040808    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04484e50-2g9bs"]
Oct 31 12:44:12.047837 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:12.047811    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04484e50-2g9bs"]
Oct 31 12:44:13.063242 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:13.063202    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ef9f970-7daf-4b8f-98d0-5b4315ef9504" path="/var/lib/kubelet/pods/4ef9f970-7daf-4b8f-98d0-5b4315ef9504/volumes"
Oct 31 12:44:13.910529 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:13.910460    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d702046e-fjp65"]
Oct 31 12:44:13.911253 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:13.911238    5060 cpu_manager.go:414] "containerMap: removing container" podUID="bc3210d5-cef9-47d8-b4d1-986c62846911" containerName="revoke-keys"
Oct 31 12:44:13.911298 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:13.911256    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc3210d5-cef9-47d8-b4d1-986c62846911" containerName="revoke-keys"
Oct 31 12:44:13.911414 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:13.911406    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc3210d5-cef9-47d8-b4d1-986c62846911" containerName="revoke-keys"
Oct 31 12:44:13.916146 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:13.916116    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d702046e-fjp65"
Oct 31 12:44:13.921516 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:13.921456    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 12:44:13.921653 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:13.921517    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7wsmb\""
Oct 31 12:44:13.921653 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:13.921535    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 12:44:13.921789 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:13.921473    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 12:44:13.922394 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:13.922369    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d702046e-fjp65"]
Oct 31 12:44:13.968163 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:13.968113    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8gzv\" (UniqueName: \"kubernetes.io/projected/1e65fb36-fe66-483a-8482-b9cc8b5922f4-kube-api-access-h8gzv\") pod \"maas-api-revoke-keys-e2e-ait-d702046e-fjp65\" (UID: \"1e65fb36-fe66-483a-8482-b9cc8b5922f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d702046e-fjp65"
Oct 31 12:44:13.968447 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:13.968419    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e65fb36-fe66-483a-8482-b9cc8b5922f4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-d702046e-fjp65\" (UID: \"1e65fb36-fe66-483a-8482-b9cc8b5922f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d702046e-fjp65"
Oct 31 12:44:13.968612 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:13.968469    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e65fb36-fe66-483a-8482-b9cc8b5922f4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-d702046e-fjp65\" (UID: \"1e65fb36-fe66-483a-8482-b9cc8b5922f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d702046e-fjp65"
Oct 31 12:44:14.070593 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:14.070549    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e65fb36-fe66-483a-8482-b9cc8b5922f4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-d702046e-fjp65\" (UID: \"1e65fb36-fe66-483a-8482-b9cc8b5922f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d702046e-fjp65"
Oct 31 12:44:14.071029 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:14.070607    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e65fb36-fe66-483a-8482-b9cc8b5922f4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-d702046e-fjp65\" (UID: \"1e65fb36-fe66-483a-8482-b9cc8b5922f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d702046e-fjp65"
Oct 31 12:44:14.071029 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:14.070811    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8gzv\" (UniqueName: \"kubernetes.io/projected/1e65fb36-fe66-483a-8482-b9cc8b5922f4-kube-api-access-h8gzv\") pod \"maas-api-revoke-keys-e2e-ait-d702046e-fjp65\" (UID: \"1e65fb36-fe66-483a-8482-b9cc8b5922f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d702046e-fjp65"
Oct 31 12:44:14.071333 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:14.071306    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e65fb36-fe66-483a-8482-b9cc8b5922f4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-d702046e-fjp65\" (UID: \"1e65fb36-fe66-483a-8482-b9cc8b5922f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d702046e-fjp65"
Oct 31 12:44:14.073216 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:14.073191    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e65fb36-fe66-483a-8482-b9cc8b5922f4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-d702046e-fjp65\" (UID: \"1e65fb36-fe66-483a-8482-b9cc8b5922f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d702046e-fjp65"
Oct 31 12:44:14.082904 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:14.082875    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8gzv\" (UniqueName: \"kubernetes.io/projected/1e65fb36-fe66-483a-8482-b9cc8b5922f4-kube-api-access-h8gzv\") pod \"maas-api-revoke-keys-e2e-ait-d702046e-fjp65\" (UID: \"1e65fb36-fe66-483a-8482-b9cc8b5922f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d702046e-fjp65"
Oct 31 12:44:14.243224 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:14.242839    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d702046e-fjp65"
Oct 31 12:44:14.661805 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:44:14.661770    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e65fb36_fe66_483a_8482_b9cc8b5922f4.slice/crio-b29fd6069fdc9290d04bf35318f8b0dbfb042cd3e0c6f9b0e622719774f126f8 WatchSource:0}: Error finding container b29fd6069fdc9290d04bf35318f8b0dbfb042cd3e0c6f9b0e622719774f126f8: Status 404 returned error can't find the container with id b29fd6069fdc9290d04bf35318f8b0dbfb042cd3e0c6f9b0e622719774f126f8
Oct 31 12:44:14.668537 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:14.668441    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d702046e-fjp65"]
Oct 31 12:44:14.857941 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:14.857905    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d702046e-fjp65" event={"ID":"1e65fb36-fe66-483a-8482-b9cc8b5922f4","Type":"ContainerStarted","Data":"b29fd6069fdc9290d04bf35318f8b0dbfb042cd3e0c6f9b0e622719774f126f8"}
Oct 31 12:44:15.886426 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:15.886385    5060 generic.go:356] "Generic (PLEG): container finished" podID="1e65fb36-fe66-483a-8482-b9cc8b5922f4" containerID="5e6f756002afe25183885ebe3601efc4d4cf83bf2f6d0a0f523c7bf4669a04d9" exitCode=0
Oct 31 12:44:15.887209 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:15.887162    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d702046e-fjp65" event={"ID":"1e65fb36-fe66-483a-8482-b9cc8b5922f4","Type":"ContainerDied","Data":"5e6f756002afe25183885ebe3601efc4d4cf83bf2f6d0a0f523c7bf4669a04d9"}
Oct 31 12:44:17.101297 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:17.101265    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d702046e-fjp65"
Oct 31 12:44:17.147606 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:17.147569    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e65fb36-fe66-483a-8482-b9cc8b5922f4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e65fb36-fe66-483a-8482-b9cc8b5922f4-maas-api-openshift-ca\") pod \"1e65fb36-fe66-483a-8482-b9cc8b5922f4\" (UID: \"1e65fb36-fe66-483a-8482-b9cc8b5922f4\") "
Oct 31 12:44:17.147770 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:17.147658    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1e65fb36-fe66-483a-8482-b9cc8b5922f4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e65fb36-fe66-483a-8482-b9cc8b5922f4-maas-api-xks-ca\") pod \"1e65fb36-fe66-483a-8482-b9cc8b5922f4\" (UID: \"1e65fb36-fe66-483a-8482-b9cc8b5922f4\") "
Oct 31 12:44:17.147902 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:17.147877    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e65fb36-fe66-483a-8482-b9cc8b5922f4-kube-api-access-h8gzv\" (UniqueName: \"kubernetes.io/projected/1e65fb36-fe66-483a-8482-b9cc8b5922f4-kube-api-access-h8gzv\") pod \"1e65fb36-fe66-483a-8482-b9cc8b5922f4\" (UID: \"1e65fb36-fe66-483a-8482-b9cc8b5922f4\") "
Oct 31 12:44:17.148014 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:17.147981    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e65fb36-fe66-483a-8482-b9cc8b5922f4-maas-api-openshift-ca" pod "1e65fb36-fe66-483a-8482-b9cc8b5922f4" (UID: "1e65fb36-fe66-483a-8482-b9cc8b5922f4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:44:17.148864 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:17.148756    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e65fb36-fe66-483a-8482-b9cc8b5922f4-maas-api-openshift-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:44:17.150106 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:17.150082    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e65fb36-fe66-483a-8482-b9cc8b5922f4-maas-api-xks-ca" pod "1e65fb36-fe66-483a-8482-b9cc8b5922f4" (UID: "1e65fb36-fe66-483a-8482-b9cc8b5922f4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:44:17.150942 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:17.150912    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e65fb36-fe66-483a-8482-b9cc8b5922f4-kube-api-access-h8gzv" pod "1e65fb36-fe66-483a-8482-b9cc8b5922f4" (UID: "1e65fb36-fe66-483a-8482-b9cc8b5922f4"). InnerVolumeSpecName "kube-api-access-h8gzv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:44:17.266635 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:17.259834    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8gzv\" (UniqueName: \"kubernetes.io/projected/1e65fb36-fe66-483a-8482-b9cc8b5922f4-kube-api-access-h8gzv\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:44:17.266635 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:17.259886    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e65fb36-fe66-483a-8482-b9cc8b5922f4-maas-api-xks-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:44:17.936248 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:17.936216    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d702046e-fjp65"
Oct 31 12:44:17.936248 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:17.936228    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d702046e-fjp65" event={"ID":"1e65fb36-fe66-483a-8482-b9cc8b5922f4","Type":"ContainerDied","Data":"b29fd6069fdc9290d04bf35318f8b0dbfb042cd3e0c6f9b0e622719774f126f8"}
Oct 31 12:44:17.936450 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:17.936298    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b29fd6069fdc9290d04bf35318f8b0dbfb042cd3e0c6f9b0e622719774f126f8"
Oct 31 12:44:21.348024 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:21.347968    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/simulator-w4-c67de2-kserve-868679cf9b-gqjbx" podUID="e104d7da-1fb2-4c99-a480-b4ae84c4f273" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:44:21.348753 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:21.348029    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-2-w5-b25d3b-kserve-76f798d59f-vvzhg" podUID="394ed76c-84ac-4612-9acb-a7e36cb68d89" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:44:21.349677 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:21.349648    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/premium-w2-e6c79d-kserve-6468bf46d4-h66f8" podUID="cd6a266f-8d12-4089-83ee-c75c716bf26c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Oct 31 12:44:22.426527 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:22.425769    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-2-w5-b25d3b-kserve-76f798d59f-vvzhg" podUID="394ed76c-84ac-4612-9acb-a7e36cb68d89" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:44:22.640531 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:22.614301    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-2-w5-b25d3b-kserve-76f798d59f-vvzhg"
Oct 31 12:44:22.694570 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:22.670314    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/premium-w2-e6c79d-kserve-6468bf46d4-h66f8" podUID="cd6a266f-8d12-4089-83ee-c75c716bf26c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:44:22.759957 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:22.746208    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x2qzv" podUID="c247ed2a-aa90-424b-9b7e-4f2936d67e3f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded"
Oct 31 12:44:22.790785 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:22.771645    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e6c79d/premium-w2-e6c79d-kserve-6468bf46d4-h66f8"
Oct 31 12:44:22.834997 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:22.834939    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/simulator-w4-c67de2-kserve-868679cf9b-gqjbx" podUID="e104d7da-1fb2-4c99-a480-b4ae84c4f273" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:44:23.276521 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:23.260881    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-rbn8q" podUID="dea929f7-61c3-4921-9bdb-106f763e067e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:44:23.347521 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:23.278610    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c67de2/simulator-w4-c67de2-kserve-868679cf9b-gqjbx"
Oct 31 12:44:23.445870 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:23.378864    5060 patch_prober.go:29] interesting pod/e2e-worker-w3-b218d3-openshift-default-67596dd89f-2fn6t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 12:44:23.445870 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:23.378940    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-2fn6t" podUID="60d395e9-e4e9-4c68-ab10-9ef5371df1b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:44:23.445870 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:23.379018    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-rm7cl" podUID="8292eaa8-0a02-46be-8654-3c4136ff604f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded"
Oct 31 12:44:23.704153 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:23.704044    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-2-w5-b25d3b-kserve-76f798d59f-vvzhg" podUID="394ed76c-84ac-4612-9acb-a7e36cb68d89" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:44:23.955719 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:23.817237    5060 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x2qzv" podUID="c247ed2a-aa90-424b-9b7e-4f2936d67e3f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": context deadline exceeded"
Oct 31 12:44:24.020529 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:23.954893    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/premium-w2-e6c79d-kserve-6468bf46d4-h66f8" podUID="cd6a266f-8d12-4089-83ee-c75c716bf26c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Oct 31 12:44:24.025069 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:24.021562    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/simulator-w4-c67de2-kserve-868679cf9b-gqjbx" podUID="e104d7da-1fb2-4c99-a480-b4ae84c4f273" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:44:24.025069 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:24.021838    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x2qzv" podUID="c247ed2a-aa90-424b-9b7e-4f2936d67e3f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded"
Oct 31 12:44:24.143197 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:24.135578    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cdc017bf-9swb5"]
Oct 31 12:44:24.143197 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:24.135669    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x2qzv"
Oct 31 12:44:24.447606 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:24.447563    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cdc017bf-9swb5"]
Oct 31 12:44:24.489930 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:24.489890    5060 patch_prober.go:29] interesting pod/e2e-worker-w3-b218d3-openshift-default-67596dd89f-28t2l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 12:44:24.497617 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:24.497569    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-28t2l" podUID="66295870-6cfc-4f48-b353-566fe73a6bf0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:44:24.959747 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:24.959686    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-2-w5-b25d3b-kserve-76f798d59f-vvzhg" podUID="394ed76c-84ac-4612-9acb-a7e36cb68d89" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": context deadline exceeded"
Oct 31 12:44:25.073548 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:25.019262    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/premium-w2-e6c79d-kserve-6468bf46d4-h66f8" podUID="cd6a266f-8d12-4089-83ee-c75c716bf26c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:44:25.159550 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:25.148461    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x2qzv" podUID="c247ed2a-aa90-424b-9b7e-4f2936d67e3f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:44:25.239664 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:25.185125    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/simulator-w4-c67de2-kserve-868679cf9b-gqjbx" podUID="e104d7da-1fb2-4c99-a480-b4ae84c4f273" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded"
Oct 31 12:44:25.520610 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:25.520516    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8a3573f-fa00-4b9d-9ea5-b961e6efd4f6" path="/var/lib/kubelet/pods/a8a3573f-fa00-4b9d-9ea5-b961e6efd4f6/volumes"
Oct 31 12:44:25.526731 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:25.526704    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9545da26-bbzlv"]
Oct 31 12:44:25.986518 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:25.979722    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-2-w5-b25d3b-kserve-76f798d59f-vvzhg" podUID="394ed76c-84ac-4612-9acb-a7e36cb68d89" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": context deadline exceeded"
Oct 31 12:44:26.027633 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:26.027583    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/premium-w2-e6c79d-kserve-6468bf46d4-h66f8" podUID="cd6a266f-8d12-4089-83ee-c75c716bf26c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Oct 31 12:44:26.099516 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:26.050107    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9545da26-bbzlv"]
Oct 31 12:44:26.152703 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:26.152654    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x2qzv" podUID="c247ed2a-aa90-424b-9b7e-4f2936d67e3f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded"
Oct 31 12:44:26.203631 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:26.203577    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/simulator-w4-c67de2-kserve-868679cf9b-gqjbx" podUID="e104d7da-1fb2-4c99-a480-b4ae84c4f273" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:44:26.577970 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:26.577834    5060 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-2-w5-b25d3b-kserve-76f798d59f-vvzhg" podUID="394ed76c-84ac-4612-9acb-a7e36cb68d89" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/health\": context deadline exceeded"
Oct 31 12:44:26.674288 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:26.674215    5060 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-c67de2/simulator-w4-c67de2-kserve-868679cf9b-gqjbx" podUID="e104d7da-1fb2-4c99-a480-b4ae84c4f273" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:44:27.573515 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:27.569839    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/premium-w2-e6c79d-kserve-6468bf46d4-h66f8" podUID="cd6a266f-8d12-4089-83ee-c75c716bf26c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Oct 31 12:44:27.579517 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:27.575884    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x2qzv" podUID="c247ed2a-aa90-424b-9b7e-4f2936d67e3f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded"
Oct 31 12:44:27.849112 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:27.772927    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-2-w5-b25d3b-kserve-76f798d59f-vvzhg" podUID="394ed76c-84ac-4612-9acb-a7e36cb68d89" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:44:27.956847 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:27.835137    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/simulator-w4-c67de2-kserve-868679cf9b-gqjbx" podUID="e104d7da-1fb2-4c99-a480-b4ae84c4f273" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded"
Oct 31 12:44:28.759515 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:28.667032    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/premium-w2-e6c79d-kserve-6468bf46d4-h66f8" podUID="cd6a266f-8d12-4089-83ee-c75c716bf26c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:44:29.066006 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:28.795979    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x2qzv" podUID="c247ed2a-aa90-424b-9b7e-4f2936d67e3f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded"
Oct 31 12:44:29.082544 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:28.920849    5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-645cf464d7-7psf5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 12:44:29.082544 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:29.075401    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-7psf5" podUID="37aec2da-59cb-41ed-9e6b-a443f6ff84c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:44:29.082544 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:28.944286    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/simulator-w4-c67de2-kserve-868679cf9b-gqjbx" podUID="e104d7da-1fb2-4c99-a480-b4ae84c4f273" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:44:29.082544 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:28.929972    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-2-w5-b25d3b-kserve-76f798d59f-vvzhg" podUID="394ed76c-84ac-4612-9acb-a7e36cb68d89" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": context deadline exceeded"
Oct 31 12:44:29.082544 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:29.064481    5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-645cf464d7-rz7mt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 12:44:29.082544 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:29.076952    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-rz7mt" podUID="978e5cf9-a89e-4fc9-b4ba-fe0ee2de401b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:44:29.082544 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:29.063984    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5206e1f-8f34-4035-b26f-3ba2cc158edd" path="/var/lib/kubelet/pods/d5206e1f-8f34-4035-b26f-3ba2cc158edd/volumes"
Oct 31 12:44:29.082544 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:44:29.077603    5060 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.554s"
Oct 31 12:44:29.084968 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:29.084924    5060 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-e6c79d/premium-w2-e6c79d-kserve-6468bf46d4-h66f8" podUID="cd6a266f-8d12-4089-83ee-c75c716bf26c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:44:29.849818 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:29.849731    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-2-w5-b25d3b-kserve-76f798d59f-vvzhg"
Oct 31 12:44:30.033573 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:29.990524    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x2qzv"
Oct 31 12:44:30.078545 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:30.071913    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/premium-w2-e6c79d-kserve-6468bf46d4-h66f8" podUID="cd6a266f-8d12-4089-83ee-c75c716bf26c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Oct 31 12:44:30.078545 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:30.075614    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/simulator-w4-c67de2-kserve-868679cf9b-gqjbx" podUID="e104d7da-1fb2-4c99-a480-b4ae84c4f273" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:44:30.478513 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:30.477747    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-361536d3-gk9cp"]
Oct 31 12:44:30.487337 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:30.487301    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-361536d3-gk9cp"]
Oct 31 12:44:30.552860 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:30.552824    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e6c79d/premium-w2-e6c79d-kserve-6468bf46d4-h66f8"
Oct 31 12:44:30.555927 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:30.555897    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c67de2/simulator-w4-c67de2-kserve-868679cf9b-gqjbx"
Oct 31 12:44:30.634879 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:44:30.634798    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7phb" containerName="extractor"
Oct 31 12:44:31.165547 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:31.146792    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f993e3b5-0533-45de-bf21-1797b50ac34d" path="/var/lib/kubelet/pods/f993e3b5-0533-45de-bf21-1797b50ac34d/volumes"
Oct 31 12:44:32.744879 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:32.744779    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d702046e-fjp65"]
Oct 31 12:44:32.772851 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:32.772797    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d702046e-fjp65"]
Oct 31 12:44:33.162540 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:33.157112    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e65fb36-fe66-483a-8482-b9cc8b5922f4" path="/var/lib/kubelet/pods/1e65fb36-fe66-483a-8482-b9cc8b5922f4/volumes"
Oct 31 12:44:33.162540 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:33.157781    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e2af189-ntpnr"]
Oct 31 12:44:33.162540 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:33.161604    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e2af189-ntpnr"]
Oct 31 12:44:35.095786 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:35.095747    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2aad7d1b-a9a7-41ba-8016-6a66f72f4cb3" path="/var/lib/kubelet/pods/2aad7d1b-a9a7-41ba-8016-6a66f72f4cb3/volumes"
Oct 31 12:44:36.038036 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:36.037982    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08422da2-298fq"]
Oct 31 12:44:36.070253 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:36.070189    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9dde715b-q58kx"]
Oct 31 12:44:36.136562 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:36.131708    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08422da2-298fq"]
Oct 31 12:44:36.165566 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:36.165517    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9dde715b-q58kx"]
Oct 31 12:44:37.092972 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:37.092933    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29aae726-4558-4b6d-ab74-770c96cac741" path="/var/lib/kubelet/pods/29aae726-4558-4b6d-ab74-770c96cac741/volumes"
Oct 31 12:44:37.093532 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:37.093508    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed4c92cc-3b80-4016-bdb3-6092fb0ecfdd" path="/var/lib/kubelet/pods/ed4c92cc-3b80-4016-bdb3-6092fb0ecfdd/volumes"
Oct 31 12:44:38.039048 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:38.039006    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9384727b-z9rvm"]
Oct 31 12:44:38.051109 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:38.051062    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9384727b-z9rvm"]
Oct 31 12:44:38.467586 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:38.466896    5060 patch_prober.go:29] interesting pod/e2e-mt-ed25a690-openshift-default-d54cbd5df-cr49v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 12:44:38.467586 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:38.466971    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-cr49v" podUID="f957e25b-5bf0-4259-afbd-fcd50a9364d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:44:38.955697 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:38.955653    5060 patch_prober.go:29] interesting pod/e2e-shared-a-w6-9b17b9-openshift-default-598d5bf87-rr64l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 12:44:38.955874 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:38.955731    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w6-9b17b9-openshift-default-598d5bf87-rr64l" podUID="fcde778f-8f1a-4c1a-a497-6c66733d54ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:44:39.056394 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:39.056341    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66856f9b-041f-4251-997c-be59be05384c" path="/var/lib/kubelet/pods/66856f9b-041f-4251-997c-be59be05384c/volumes"
Oct 31 12:44:41.041002 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:41.040944    5060 generic.go:356] "Generic (PLEG): container finished" podID="f957e25b-5bf0-4259-afbd-fcd50a9364d5" containerID="3fdffae8b53e3c69a2867c4800bf0be6233f9ec417c3386499390379583398f6" exitCode=0
Oct 31 12:44:41.044138 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:41.044098    5060 generic.go:356] "Generic (PLEG): container finished" podID="fcde778f-8f1a-4c1a-a497-6c66733d54ff" containerID="be48cd4e14df08136b036625e90d6dac400cba28c17756e71993064162ace3e4" exitCode=0
Oct 31 12:44:41.053829 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:41.053799    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-cr49v" event={"ID":"f957e25b-5bf0-4259-afbd-fcd50a9364d5","Type":"ContainerDied","Data":"3fdffae8b53e3c69a2867c4800bf0be6233f9ec417c3386499390379583398f6"}
Oct 31 12:44:41.053965 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:41.053855    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-9b17b9-openshift-default-598d5bf87-rr64l" event={"ID":"fcde778f-8f1a-4c1a-a497-6c66733d54ff","Type":"ContainerDied","Data":"be48cd4e14df08136b036625e90d6dac400cba28c17756e71993064162ace3e4"}
Oct 31 12:44:41.054289 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:41.054270    5060 scope.go:122] "RemoveContainer" containerID="be48cd4e14df08136b036625e90d6dac400cba28c17756e71993064162ace3e4"
Oct 31 12:44:41.054635 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:41.054608    5060 scope.go:122] "RemoveContainer" containerID="3fdffae8b53e3c69a2867c4800bf0be6233f9ec417c3386499390379583398f6"
Oct 31 12:44:41.056060 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:41.056022    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:44:41.056152 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:41.056116    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:44:41.056212 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:41.056158    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:44:41.056212 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:41.056116    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:44:41.056308 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:41.056252    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:44:41.056308 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:41.056294    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:44:42.068518 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:42.067911    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-9b17b9-openshift-default-598d5bf87-rr64l" event={"ID":"fcde778f-8f1a-4c1a-a497-6c66733d54ff","Type":"ContainerStarted","Data":"5a4614f248369d23f04985ae910ff69be39a38a7d5d1959fd4ff018ca7e272ab"}
Oct 31 12:44:42.077029 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:42.076988    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-cr49v" event={"ID":"f957e25b-5bf0-4259-afbd-fcd50a9364d5","Type":"ContainerStarted","Data":"5626dc1a2d0125d74428281ca91d1a94e7eddb74d5d5b713dd45ea1ed1aa7227"}
Oct 31 12:44:42.418558 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:42.417936    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-cr49v"
Oct 31 12:44:42.418558 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:42.418073    5060 patch_prober.go:29] interesting pod/e2e-mt-ed25a690-openshift-default-d54cbd5df-cr49v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body=
Oct 31 12:44:42.418558 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:42.418121    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-cr49v" podUID="f957e25b-5bf0-4259-afbd-fcd50a9364d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused"
Oct 31 12:44:42.637215 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:42.637181    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-9b17b9-openshift-default-598d5bf87-rr64l"
Oct 31 12:44:42.637927 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:42.637894    5060 patch_prober.go:29] interesting pod/e2e-shared-a-w6-9b17b9-openshift-default-598d5bf87-rr64l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" start-of-body=
Oct 31 12:44:42.638078 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:42.637959    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-9b17b9-openshift-default-598d5bf87-rr64l" podUID="fcde778f-8f1a-4c1a-a497-6c66733d54ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused"
Oct 31 12:44:43.465731 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:43.465681    5060 patch_prober.go:29] interesting pod/e2e-mt-ed25a690-openshift-default-d54cbd5df-cr49v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body=
Oct 31 12:44:43.466364 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:43.466316    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-cr49v" podUID="f957e25b-5bf0-4259-afbd-fcd50a9364d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused"
Oct 31 12:44:43.600996 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:43.600955    5060 patch_prober.go:29] interesting pod/e2e-shared-a-w6-9b17b9-openshift-default-598d5bf87-rr64l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" start-of-body=
Oct 31 12:44:43.604637 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:43.604050    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-9b17b9-openshift-default-598d5bf87-rr64l" podUID="fcde778f-8f1a-4c1a-a497-6c66733d54ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused"
Oct 31 12:44:44.170591 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:44.170550    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-174b5a58-b42hp"]
Oct 31 12:44:44.223463 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:44.223422    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-174b5a58-b42hp"]
Oct 31 12:44:44.414993 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:44.414952    5060 patch_prober.go:29] interesting pod/e2e-mt-ed25a690-openshift-default-d54cbd5df-cr49v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body=
Oct 31 12:44:44.415174 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:44.415023    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-cr49v" podUID="f957e25b-5bf0-4259-afbd-fcd50a9364d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused"
Oct 31 12:44:44.605515 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:44.604873    5060 patch_prober.go:29] interesting pod/e2e-shared-a-w6-9b17b9-openshift-default-598d5bf87-rr64l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" start-of-body=
Oct 31 12:44:44.605515 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:44.604935    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-9b17b9-openshift-default-598d5bf87-rr64l" podUID="fcde778f-8f1a-4c1a-a497-6c66733d54ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused"
Oct 31 12:44:45.080558 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:45.070041    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63d60111-09da-4dc5-9c83-4b715c93d5aa" path="/var/lib/kubelet/pods/63d60111-09da-4dc5-9c83-4b715c93d5aa/volumes"
Oct 31 12:44:45.080969 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:45.080935    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38082411-ccdx6"]
Oct 31 12:44:45.105598 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:45.105561    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86358a2b-26tlt"]
Oct 31 12:44:45.132722 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:45.132164    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38082411-ccdx6"]
Oct 31 12:44:45.139114 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:45.139086    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86358a2b-26tlt"]
Oct 31 12:44:45.417021 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:45.416921    5060 patch_prober.go:29] interesting pod/e2e-mt-ed25a690-openshift-default-d54cbd5df-cr49v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body=
Oct 31 12:44:45.417021 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:45.416985    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-cr49v" podUID="f957e25b-5bf0-4259-afbd-fcd50a9364d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused"
Oct 31 12:44:45.606906 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:45.606865    5060 patch_prober.go:29] interesting pod/e2e-shared-a-w6-9b17b9-openshift-default-598d5bf87-rr64l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" start-of-body=
Oct 31 12:44:45.607513 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:45.607466    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-9b17b9-openshift-default-598d5bf87-rr64l" podUID="fcde778f-8f1a-4c1a-a497-6c66733d54ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused"
Oct 31 12:44:46.123948 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:46.123910    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e50d2485-49m8z"]
Oct 31 12:44:46.240527 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:46.235422    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e50d2485-49m8z"]
Oct 31 12:44:46.438977 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:46.426313    5060 patch_prober.go:29] interesting pod/e2e-mt-ed25a690-openshift-default-d54cbd5df-cr49v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body=
Oct 31 12:44:46.438977 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:46.426390    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-cr49v" podUID="f957e25b-5bf0-4259-afbd-fcd50a9364d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused"
Oct 31 12:44:46.600052 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:46.600014    5060 patch_prober.go:29] interesting pod/e2e-shared-a-w6-9b17b9-openshift-default-598d5bf87-rr64l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" start-of-body=
Oct 31 12:44:46.600315 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:46.600281    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-9b17b9-openshift-default-598d5bf87-rr64l" podUID="fcde778f-8f1a-4c1a-a497-6c66733d54ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused"
Oct 31 12:44:47.102295 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:47.102252    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="060d9865-c13f-4247-b639-e6b1260314a2" path="/var/lib/kubelet/pods/060d9865-c13f-4247-b639-e6b1260314a2/volumes"
Oct 31 12:44:47.103873 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:47.103848    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18cf4a1e-a2da-4f86-871c-8da52d904b8c" path="/var/lib/kubelet/pods/18cf4a1e-a2da-4f86-871c-8da52d904b8c/volumes"
Oct 31 12:44:47.117520 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:47.117447    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de3e8b02-e9f6-4a9d-90c1-ea2a3d901fbc" path="/var/lib/kubelet/pods/de3e8b02-e9f6-4a9d-90c1-ea2a3d901fbc/volumes"
Oct 31 12:44:47.439582 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:47.439471    5060 patch_prober.go:29] interesting pod/e2e-mt-ed25a690-openshift-default-d54cbd5df-cr49v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body=
Oct 31 12:44:47.439867 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:47.439831    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-cr49v" podUID="f957e25b-5bf0-4259-afbd-fcd50a9364d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused"
Oct 31 12:44:47.613795 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:47.613757    5060 patch_prober.go:29] interesting pod/e2e-shared-a-w6-9b17b9-openshift-default-598d5bf87-rr64l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" start-of-body=
Oct 31 12:44:47.614111 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:47.614079    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-9b17b9-openshift-default-598d5bf87-rr64l" podUID="fcde778f-8f1a-4c1a-a497-6c66733d54ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused"
Oct 31 12:44:48.176843 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:48.163076    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8138d6b-7r46x"]
Oct 31 12:44:48.380526 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:48.380153    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8138d6b-7r46x"]
Oct 31 12:44:48.423513 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:48.418138    5060 patch_prober.go:29] interesting pod/e2e-mt-ed25a690-openshift-default-d54cbd5df-cr49v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body=
Oct 31 12:44:48.423513 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:48.418216    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-cr49v" podUID="f957e25b-5bf0-4259-afbd-fcd50a9364d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused"
Oct 31 12:44:48.601416 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:48.601379    5060 patch_prober.go:29] interesting pod/e2e-shared-a-w6-9b17b9-openshift-default-598d5bf87-rr64l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" start-of-body=
Oct 31 12:44:48.601714 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:48.601683    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-9b17b9-openshift-default-598d5bf87-rr64l" podUID="fcde778f-8f1a-4c1a-a497-6c66733d54ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused"
Oct 31 12:44:49.154897 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:49.154847    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e805d884-0f36-4b61-b17d-b829c423ce95" path="/var/lib/kubelet/pods/e805d884-0f36-4b61-b17d-b829c423ce95/volumes"
Oct 31 12:44:49.418957 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:49.418873    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-cr49v"
Oct 31 12:44:49.419359 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:49.419340    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-cr49v"
Oct 31 12:44:49.419983 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:49.419958    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-cr49v"
Oct 31 12:44:49.608654 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:49.608623    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-9b17b9-openshift-default-598d5bf87-rr64l"
Oct 31 12:44:49.608937 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:49.608916    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-9b17b9-openshift-default-598d5bf87-rr64l"
Oct 31 12:44:49.609893 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:49.609872    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-9b17b9-openshift-default-598d5bf87-rr64l"
Oct 31 12:44:55.029620 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:55.029573    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f6427de-hg4lt"]
Oct 31 12:44:55.036048 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:55.036014    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f6427de-hg4lt"]
Oct 31 12:44:55.049922 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:55.049894    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32d226db-5345-467d-9a0b-12920ae3e0e3" path="/var/lib/kubelet/pods/32d226db-5345-467d-9a0b-12920ae3e0e3/volumes"
Oct 31 12:44:59.062560 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:59.062447    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-8faad6-openshift-default-9ff4fdc49-fc7fk"]
Oct 31 12:44:59.063623 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:59.063581    5060 cpu_manager.go:414] "containerMap: removing container" podUID="1e65fb36-fe66-483a-8482-b9cc8b5922f4" containerName="revoke-keys"
Oct 31 12:44:59.063833 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:59.063817    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e65fb36-fe66-483a-8482-b9cc8b5922f4" containerName="revoke-keys"
Oct 31 12:44:59.064201 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:59.064186    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e65fb36-fe66-483a-8482-b9cc8b5922f4" containerName="revoke-keys"
Oct 31 12:44:59.085632 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:59.085603    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e622406c-mq2c9"]
Oct 31 12:44:59.085632 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:59.085638    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-8faad6-openshift-default-9ff4fdc49-fc7fk"]
Oct 31 12:44:59.085839 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:59.085720    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-8faad6-openshift-default-9ff4fdc49-fc7fk"
Oct 31 12:44:59.089763 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:59.089734    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e622406c-mq2c9"]
Oct 31 12:44:59.141313 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:59.141261    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-8faad6-openshift-default-dockercfg-7rmp4\""
Oct 31 12:44:59.267515 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:59.264766    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d1ddd1b-8dc4-4706-911f-18376b4b2b68-istio-podinfo\") pod \"e2e-shared-b-w4-8faad6-openshift-default-9ff4fdc49-fc7fk\" (UID: \"4d1ddd1b-8dc4-4706-911f-18376b4b2b68\") " pod="openshift-ingress/e2e-shared-b-w4-8faad6-openshift-default-9ff4fdc49-fc7fk"
Oct 31 12:44:59.267515 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:59.264829    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d1ddd1b-8dc4-4706-911f-18376b4b2b68-credential-socket\") pod \"e2e-shared-b-w4-8faad6-openshift-default-9ff4fdc49-fc7fk\" (UID: \"4d1ddd1b-8dc4-4706-911f-18376b4b2b68\") " pod="openshift-ingress/e2e-shared-b-w4-8faad6-openshift-default-9ff4fdc49-fc7fk"
Oct 31 12:44:59.267515 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:59.264952    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d1ddd1b-8dc4-4706-911f-18376b4b2b68-workload-certs\") pod \"e2e-shared-b-w4-8faad6-openshift-default-9ff4fdc49-fc7fk\" (UID: \"4d1ddd1b-8dc4-4706-911f-18376b4b2b68\") " pod="openshift-ingress/e2e-shared-b-w4-8faad6-openshift-default-9ff4fdc49-fc7fk"
Oct 31 12:44:59.267515 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:59.264991    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnq7x\" (UniqueName: \"kubernetes.io/projected/4d1ddd1b-8dc4-4706-911f-18376b4b2b68-kube-api-access-bnq7x\") pod \"e2e-shared-b-w4-8faad6-openshift-default-9ff4fdc49-fc7fk\" (UID: \"4d1ddd1b-8dc4-4706-911f-18376b4b2b68\") " pod="openshift-ingress/e2e-shared-b-w4-8faad6-openshift-default-9ff4fdc49-fc7fk"
Oct 31 12:44:59.267515 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:59.265035    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d1ddd1b-8dc4-4706-911f-18376b4b2b68-istio-data\") pod \"e2e-shared-b-w4-8faad6-openshift-default-9ff4fdc49-fc7fk\" (UID: \"4d1ddd1b-8dc4-4706-911f-18376b4b2b68\") " pod="openshift-ingress/e2e-shared-b-w4-8faad6-openshift-default-9ff4fdc49-fc7fk"
Oct 31 12:44:59.267515 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:59.265059    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d1ddd1b-8dc4-4706-911f-18376b4b2b68-istio-token\") pod \"e2e-shared-b-w4-8faad6-openshift-default-9ff4fdc49-fc7fk\" (UID: \"4d1ddd1b-8dc4-4706-911f-18376b4b2b68\") " pod="openshift-ingress/e2e-shared-b-w4-8faad6-openshift-default-9ff4fdc49-fc7fk"
Oct 31 12:44:59.267515 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:59.265106    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d1ddd1b-8dc4-4706-911f-18376b4b2b68-workload-socket\") pod \"e2e-shared-b-w4-8faad6-openshift-default-9ff4fdc49-fc7fk\" (UID: \"4d1ddd1b-8dc4-4706-911f-18376b4b2b68\") " pod="openshift-ingress/e2e-shared-b-w4-8faad6-openshift-default-9ff4fdc49-fc7fk"
Oct 31 12:44:59.267515 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:59.265223    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d1ddd1b-8dc4-4706-911f-18376b4b2b68-istiod-ca-cert\") pod \"e2e-shared-b-w4-8faad6-openshift-default-9ff4fdc49-fc7fk\" (UID: \"4d1ddd1b-8dc4-4706-911f-18376b4b2b68\") " pod="openshift-ingress/e2e-shared-b-w4-8faad6-openshift-default-9ff4fdc49-fc7fk"
Oct 31 12:44:59.267515 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:59.265388    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d1ddd1b-8dc4-4706-911f-18376b4b2b68-istio-envoy\") pod \"e2e-shared-b-w4-8faad6-openshift-default-9ff4fdc49-fc7fk\" (UID: \"4d1ddd1b-8dc4-4706-911f-18376b4b2b68\") " pod="openshift-ingress/e2e-shared-b-w4-8faad6-openshift-default-9ff4fdc49-fc7fk"
Oct 31 12:44:59.371562 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:59.368664    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d1ddd1b-8dc4-4706-911f-18376b4b2b68-istio-envoy\") pod \"e2e-shared-b-w4-8faad6-openshift-default-9ff4fdc49-fc7fk\" (UID: \"4d1ddd1b-8dc4-4706-911f-18376b4b2b68\") " pod="openshift-ingress/e2e-shared-b-w4-8faad6-openshift-default-9ff4fdc49-fc7fk"
Oct 31 12:44:59.371562 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:59.368869    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d1ddd1b-8dc4-4706-911f-18376b4b2b68-istio-podinfo\") pod \"e2e-shared-b-w4-8faad6-openshift-default-9ff4fdc49-fc7fk\" (UID: \"4d1ddd1b-8dc4-4706-911f-18376b4b2b68\") " pod="openshift-ingress/e2e-shared-b-w4-8faad6-openshift-default-9ff4fdc49-fc7fk"
Oct 31 12:44:59.371562 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:59.368915    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d1ddd1b-8dc4-4706-911f-18376b4b2b68-credential-socket\") pod \"e2e-shared-b-w4-8faad6-openshift-default-9ff4fdc49-fc7fk\" (UID: \"4d1ddd1b-8dc4-4706-911f-18376b4b2b68\") " pod="openshift-ingress/e2e-shared-b-w4-8faad6-openshift-default-9ff4fdc49-fc7fk"
Oct 31 12:44:59.371562 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:59.368955    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d1ddd1b-8dc4-4706-911f-18376b4b2b68-workload-certs\") pod \"e2e-shared-b-w4-8faad6-openshift-default-9ff4fdc49-fc7fk\" (UID: \"4d1ddd1b-8dc4-4706-911f-18376b4b2b68\") " pod="openshift-ingress/e2e-shared-b-w4-8faad6-openshift-default-9ff4fdc49-fc7fk"
Oct 31 12:44:59.371562 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:59.368988    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnq7x\" (UniqueName: \"kubernetes.io/projected/4d1ddd1b-8dc4-4706-911f-18376b4b2b68-kube-api-access-bnq7x\") pod \"e2e-shared-b-w4-8faad6-openshift-default-9ff4fdc49-fc7fk\" (UID: \"4d1ddd1b-8dc4-4706-911f-18376b4b2b68\") " pod="openshift-ingress/e2e-shared-b-w4-8faad6-openshift-default-9ff4fdc49-fc7fk"
Oct 31 12:44:59.371562 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:59.369029    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d1ddd1b-8dc4-4706-911f-18376b4b2b68-istio-data\") pod \"e2e-shared-b-w4-8faad6-openshift-default-9ff4fdc49-fc7fk\" (UID: \"4d1ddd1b-8dc4-4706-911f-18376b4b2b68\") " pod="openshift-ingress/e2e-shared-b-w4-8faad6-openshift-default-9ff4fdc49-fc7fk"
Oct 31 12:44:59.371562 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:59.369052    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d1ddd1b-8dc4-4706-911f-18376b4b2b68-istio-token\") pod \"e2e-shared-b-w4-8faad6-openshift-default-9ff4fdc49-fc7fk\" (UID: \"4d1ddd1b-8dc4-4706-911f-18376b4b2b68\") " pod="openshift-ingress/e2e-shared-b-w4-8faad6-openshift-default-9ff4fdc49-fc7fk"
Oct 31 12:44:59.371562 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:59.369101    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d1ddd1b-8dc4-4706-911f-18376b4b2b68-workload-socket\") pod \"e2e-shared-b-w4-8faad6-openshift-default-9ff4fdc49-fc7fk\" (UID: \"4d1ddd1b-8dc4-4706-911f-18376b4b2b68\") " pod="openshift-ingress/e2e-shared-b-w4-8faad6-openshift-default-9ff4fdc49-fc7fk"
Oct 31 12:44:59.371562 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:59.369224    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d1ddd1b-8dc4-4706-911f-18376b4b2b68-istiod-ca-cert\") pod \"e2e-shared-b-w4-8faad6-openshift-default-9ff4fdc49-fc7fk\" (UID: \"4d1ddd1b-8dc4-4706-911f-18376b4b2b68\") " pod="openshift-ingress/e2e-shared-b-w4-8faad6-openshift-default-9ff4fdc49-fc7fk"
Oct 31 12:44:59.371562 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:59.370231    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d1ddd1b-8dc4-4706-911f-18376b4b2b68-istiod-ca-cert\") pod \"e2e-shared-b-w4-8faad6-openshift-default-9ff4fdc49-fc7fk\" (UID: \"4d1ddd1b-8dc4-4706-911f-18376b4b2b68\") " pod="openshift-ingress/e2e-shared-b-w4-8faad6-openshift-default-9ff4fdc49-fc7fk"
Oct 31 12:44:59.371562 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:59.371547    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d1ddd1b-8dc4-4706-911f-18376b4b2b68-credential-socket\") pod \"e2e-shared-b-w4-8faad6-openshift-default-9ff4fdc49-fc7fk\" (UID: \"4d1ddd1b-8dc4-4706-911f-18376b4b2b68\") " pod="openshift-ingress/e2e-shared-b-w4-8faad6-openshift-default-9ff4fdc49-fc7fk"
Oct 31 12:44:59.372247 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:59.371798    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d1ddd1b-8dc4-4706-911f-18376b4b2b68-workload-certs\") pod \"e2e-shared-b-w4-8faad6-openshift-default-9ff4fdc49-fc7fk\" (UID: \"4d1ddd1b-8dc4-4706-911f-18376b4b2b68\") " pod="openshift-ingress/e2e-shared-b-w4-8faad6-openshift-default-9ff4fdc49-fc7fk"
Oct 31 12:44:59.372312 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:59.372258    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d1ddd1b-8dc4-4706-911f-18376b4b2b68-istio-data\") pod \"e2e-shared-b-w4-8faad6-openshift-default-9ff4fdc49-fc7fk\" (UID: \"4d1ddd1b-8dc4-4706-911f-18376b4b2b68\") " pod="openshift-ingress/e2e-shared-b-w4-8faad6-openshift-default-9ff4fdc49-fc7fk"
Oct 31 12:44:59.395669 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:59.387793    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d1ddd1b-8dc4-4706-911f-18376b4b2b68-workload-socket\") pod \"e2e-shared-b-w4-8faad6-openshift-default-9ff4fdc49-fc7fk\" (UID: \"4d1ddd1b-8dc4-4706-911f-18376b4b2b68\") " pod="openshift-ingress/e2e-shared-b-w4-8faad6-openshift-default-9ff4fdc49-fc7fk"
Oct 31 12:44:59.396401 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:59.396310    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d1ddd1b-8dc4-4706-911f-18376b4b2b68-istio-token\") pod \"e2e-shared-b-w4-8faad6-openshift-default-9ff4fdc49-fc7fk\" (UID: \"4d1ddd1b-8dc4-4706-911f-18376b4b2b68\") " pod="openshift-ingress/e2e-shared-b-w4-8faad6-openshift-default-9ff4fdc49-fc7fk"
Oct 31 12:44:59.396809 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:59.396783    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d1ddd1b-8dc4-4706-911f-18376b4b2b68-istio-envoy\") pod \"e2e-shared-b-w4-8faad6-openshift-default-9ff4fdc49-fc7fk\" (UID: \"4d1ddd1b-8dc4-4706-911f-18376b4b2b68\") " pod="openshift-ingress/e2e-shared-b-w4-8faad6-openshift-default-9ff4fdc49-fc7fk"
Oct 31 12:44:59.406278 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:59.406217    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnq7x\" (UniqueName: \"kubernetes.io/projected/4d1ddd1b-8dc4-4706-911f-18376b4b2b68-kube-api-access-bnq7x\") pod \"e2e-shared-b-w4-8faad6-openshift-default-9ff4fdc49-fc7fk\" (UID: \"4d1ddd1b-8dc4-4706-911f-18376b4b2b68\") " pod="openshift-ingress/e2e-shared-b-w4-8faad6-openshift-default-9ff4fdc49-fc7fk"
Oct 31 12:44:59.435592 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:59.427107    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d1ddd1b-8dc4-4706-911f-18376b4b2b68-istio-podinfo\") pod \"e2e-shared-b-w4-8faad6-openshift-default-9ff4fdc49-fc7fk\" (UID: \"4d1ddd1b-8dc4-4706-911f-18376b4b2b68\") " pod="openshift-ingress/e2e-shared-b-w4-8faad6-openshift-default-9ff4fdc49-fc7fk"
Oct 31 12:44:59.702309 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:44:59.702221    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-8faad6-openshift-default-9ff4fdc49-fc7fk"
Oct 31 12:45:00.029741 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:45:00.029699    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7phb" containerName="extractor"
Oct 31 12:45:00.067408 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:00.067371    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-8faad6-openshift-default-9ff4fdc49-fc7fk"]
Oct 31 12:45:00.068129 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:00.068096    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:45:00.068206 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:00.068164    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:45:00.068206 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:00.068193    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:45:00.094826 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:00.094796    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f07afdef-x7w7t"]
Oct 31 12:45:00.104917 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:00.104895    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-97464201-lbnl5"]
Oct 31 12:45:00.111337 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:00.111313    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f07afdef-x7w7t"]
Oct 31 12:45:00.117250 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:00.117228    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-97464201-lbnl5"]
Oct 31 12:45:00.154920 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:00.154891    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-2d6201-29890845-tm2bd"]
Oct 31 12:45:00.161428 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:00.161403    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-2d6201-29890845-tm2bd"
Oct 31 12:45:00.164888 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:00.164555    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fb0343-29890845-cxhdt"]
Oct 31 12:45:00.177538 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:00.177463    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 12:45:00.177668 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:00.177569    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7wsmb\""
Oct 31 12:45:00.177668 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:00.177632    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 12:45:00.183892 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:00.183785    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-62734c-29890845-tbvmk"]
Oct 31 12:45:00.184036 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:00.183939    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fb0343-29890845-cxhdt"
Oct 31 12:45:00.199339 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:00.198294    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-62734c-29890845-tbvmk"
Oct 31 12:45:00.206262 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:00.206235    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890845-92l2w"]
Oct 31 12:45:00.219705 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:00.219682    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890845-92l2w"
Oct 31 12:45:00.252692 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:00.252540    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghnqm\" (UniqueName: \"kubernetes.io/projected/de4cc1ae-6e6f-4124-a0cd-464549508f9e-kube-api-access-ghnqm\") pod \"maas-api-key-cleanup-e2e-worker-w0-fb0343-29890845-cxhdt\" (UID: \"de4cc1ae-6e6f-4124-a0cd-464549508f9e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fb0343-29890845-cxhdt"
Oct 31 12:45:00.252692 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:00.252660    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ph6qp\" (UniqueName: \"kubernetes.io/projected/5361e3ad-5107-41cc-8afa-f0c64e8236b1-kube-api-access-ph6qp\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-62734c-29890845-tbvmk\" (UID: \"5361e3ad-5107-41cc-8afa-f0c64e8236b1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-62734c-29890845-tbvmk"
Oct 31 12:45:00.254118 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:00.252801    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x577d\" (UniqueName: \"kubernetes.io/projected/dd0d4341-08cb-40e3-9062-c24256031504-kube-api-access-x577d\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-2d6201-29890845-tm2bd\" (UID: \"dd0d4341-08cb-40e3-9062-c24256031504\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-2d6201-29890845-tm2bd"
Oct 31 12:45:00.254118 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:00.252952    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5361e3ad-5107-41cc-8afa-f0c64e8236b1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-62734c-29890845-tbvmk\" (UID: \"5361e3ad-5107-41cc-8afa-f0c64e8236b1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-62734c-29890845-tbvmk"
Oct 31 12:45:00.254118 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:00.253068    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd0d4341-08cb-40e3-9062-c24256031504-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-2d6201-29890845-tm2bd\" (UID: \"dd0d4341-08cb-40e3-9062-c24256031504\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-2d6201-29890845-tm2bd"
Oct 31 12:45:00.254118 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:00.253167    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/de4cc1ae-6e6f-4124-a0cd-464549508f9e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-fb0343-29890845-cxhdt\" (UID: \"de4cc1ae-6e6f-4124-a0cd-464549508f9e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fb0343-29890845-cxhdt"
Oct 31 12:45:00.255167 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:00.254751    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-2d6201-29890845-tm2bd"]
Oct 31 12:45:00.279002 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:00.278958    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fb0343-29890845-cxhdt"]
Oct 31 12:45:00.322267 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:00.322236    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-62734c-29890845-tbvmk"]
Oct 31 12:45:00.336818 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:00.336791    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-ed25a690-29890845-pvxsp"]
Oct 31 12:45:00.345661 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:00.345630    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-ed25a690-29890845-pvxsp"
Oct 31 12:45:00.349534 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:00.348757    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e6c79d-29890845-9lxh7"]
Oct 31 12:45:00.357537 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:00.354933    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cq7ng\" (UniqueName: \"kubernetes.io/projected/d87b9890-61ac-4018-a312-c02787e3c371-kube-api-access-cq7ng\") pod \"maas-api-key-cleanup-29890845-92l2w\" (UID: \"d87b9890-61ac-4018-a312-c02787e3c371\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890845-92l2w"
Oct 31 12:45:00.357537 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:00.354990    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghnqm\" (UniqueName: \"kubernetes.io/projected/de4cc1ae-6e6f-4124-a0cd-464549508f9e-kube-api-access-ghnqm\") pod \"maas-api-key-cleanup-e2e-worker-w0-fb0343-29890845-cxhdt\" (UID: \"de4cc1ae-6e6f-4124-a0cd-464549508f9e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fb0343-29890845-cxhdt"
Oct 31 12:45:00.357537 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:00.355089    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ph6qp\" (UniqueName: \"kubernetes.io/projected/5361e3ad-5107-41cc-8afa-f0c64e8236b1-kube-api-access-ph6qp\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-62734c-29890845-tbvmk\" (UID: \"5361e3ad-5107-41cc-8afa-f0c64e8236b1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-62734c-29890845-tbvmk"
Oct 31 12:45:00.357537 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:00.355166    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x577d\" (UniqueName: \"kubernetes.io/projected/dd0d4341-08cb-40e3-9062-c24256031504-kube-api-access-x577d\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-2d6201-29890845-tm2bd\" (UID: \"dd0d4341-08cb-40e3-9062-c24256031504\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-2d6201-29890845-tm2bd"
Oct 31 12:45:00.357537 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:00.355231    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d87b9890-61ac-4018-a312-c02787e3c371-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890845-92l2w\" (UID: \"d87b9890-61ac-4018-a312-c02787e3c371\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890845-92l2w"
Oct 31 12:45:00.357537 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:00.355341    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5361e3ad-5107-41cc-8afa-f0c64e8236b1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-62734c-29890845-tbvmk\" (UID: \"5361e3ad-5107-41cc-8afa-f0c64e8236b1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-62734c-29890845-tbvmk"
Oct 31 12:45:00.357537 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:00.355455    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd0d4341-08cb-40e3-9062-c24256031504-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-2d6201-29890845-tm2bd\" (UID: \"dd0d4341-08cb-40e3-9062-c24256031504\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-2d6201-29890845-tm2bd"
Oct 31 12:45:00.357537 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:00.355571    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/de4cc1ae-6e6f-4124-a0cd-464549508f9e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-fb0343-29890845-cxhdt\" (UID: \"de4cc1ae-6e6f-4124-a0cd-464549508f9e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fb0343-29890845-cxhdt"
Oct 31 12:45:00.361584 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:00.359895    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5361e3ad-5107-41cc-8afa-f0c64e8236b1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-62734c-29890845-tbvmk\" (UID: \"5361e3ad-5107-41cc-8afa-f0c64e8236b1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-62734c-29890845-tbvmk"
Oct 31 12:45:00.371792 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:00.371761    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/de4cc1ae-6e6f-4124-a0cd-464549508f9e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-fb0343-29890845-cxhdt\" (UID: \"de4cc1ae-6e6f-4124-a0cd-464549508f9e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fb0343-29890845-cxhdt"
Oct 31 12:45:00.376043 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:00.375992    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghnqm\" (UniqueName: \"kubernetes.io/projected/de4cc1ae-6e6f-4124-a0cd-464549508f9e-kube-api-access-ghnqm\") pod \"maas-api-key-cleanup-e2e-worker-w0-fb0343-29890845-cxhdt\" (UID: \"de4cc1ae-6e6f-4124-a0cd-464549508f9e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fb0343-29890845-cxhdt"
Oct 31 12:45:00.378830 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:00.378785    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ph6qp\" (UniqueName: \"kubernetes.io/projected/5361e3ad-5107-41cc-8afa-f0c64e8236b1-kube-api-access-ph6qp\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-62734c-29890845-tbvmk\" (UID: \"5361e3ad-5107-41cc-8afa-f0c64e8236b1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-62734c-29890845-tbvmk"
Oct 31 12:45:00.382219 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:00.382176    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x577d\" (UniqueName: \"kubernetes.io/projected/dd0d4341-08cb-40e3-9062-c24256031504-kube-api-access-x577d\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-2d6201-29890845-tm2bd\" (UID: \"dd0d4341-08cb-40e3-9062-c24256031504\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-2d6201-29890845-tm2bd"
Oct 31 12:45:00.396027 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:00.396008    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e6c79d-29890845-9lxh7"
Oct 31 12:45:00.397588 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:00.397569    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-ed25a690-29890845-pvxsp"]
Oct 31 12:45:00.397757 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:00.397744    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e6c79d-29890845-9lxh7"]
Oct 31 12:45:00.400662 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:00.400591    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890845-92l2w"]
Oct 31 12:45:00.402069 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:00.402047    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd0d4341-08cb-40e3-9062-c24256031504-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-2d6201-29890845-tm2bd\" (UID: \"dd0d4341-08cb-40e3-9062-c24256031504\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-2d6201-29890845-tm2bd"
Oct 31 12:45:00.408999 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:00.408975    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-67d410-29890845-6m5qx"]
Oct 31 12:45:00.416899 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:00.416881    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-67d410-29890845-6m5qx"
Oct 31 12:45:00.418630 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:00.418611    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-67d410-29890845-6m5qx"]
Oct 31 12:45:00.456672 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:00.456641    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkf85\" (UniqueName: \"kubernetes.io/projected/352baef6-da71-44a4-b1cc-f0960db22901-kube-api-access-vkf85\") pod \"maas-api-key-cleanup-e2e-mt-ed25a690-29890845-pvxsp\" (UID: \"352baef6-da71-44a4-b1cc-f0960db22901\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-ed25a690-29890845-pvxsp"
Oct 31 12:45:00.456895 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:00.456874    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cq7ng\" (UniqueName: \"kubernetes.io/projected/d87b9890-61ac-4018-a312-c02787e3c371-kube-api-access-cq7ng\") pod \"maas-api-key-cleanup-29890845-92l2w\" (UID: \"d87b9890-61ac-4018-a312-c02787e3c371\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890845-92l2w"
Oct 31 12:45:00.457003 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:00.456943    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09c33526-ddf7-41df-8b1a-bf098b046b84-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-e6c79d-29890845-9lxh7\" (UID: \"09c33526-ddf7-41df-8b1a-bf098b046b84\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e6c79d-29890845-9lxh7"
Oct 31 12:45:00.457171 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:00.457147    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2t5g\" (UniqueName: \"kubernetes.io/projected/09c33526-ddf7-41df-8b1a-bf098b046b84-kube-api-access-v2t5g\") pod \"maas-api-key-cleanup-e2e-worker-w2-e6c79d-29890845-9lxh7\" (UID: \"09c33526-ddf7-41df-8b1a-bf098b046b84\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e6c79d-29890845-9lxh7"
Oct 31 12:45:00.457275 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:00.457253    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/352baef6-da71-44a4-b1cc-f0960db22901-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-ed25a690-29890845-pvxsp\" (UID: \"352baef6-da71-44a4-b1cc-f0960db22901\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-ed25a690-29890845-pvxsp"
Oct 31 12:45:00.457408 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:00.457390    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d87b9890-61ac-4018-a312-c02787e3c371-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890845-92l2w\" (UID: \"d87b9890-61ac-4018-a312-c02787e3c371\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890845-92l2w"
Oct 31 12:45:00.458850 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:00.458824    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d87b9890-61ac-4018-a312-c02787e3c371-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890845-92l2w\" (UID: \"d87b9890-61ac-4018-a312-c02787e3c371\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890845-92l2w"
Oct 31 12:45:00.469366 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:00.469334    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cq7ng\" (UniqueName: \"kubernetes.io/projected/d87b9890-61ac-4018-a312-c02787e3c371-kube-api-access-cq7ng\") pod \"maas-api-key-cleanup-29890845-92l2w\" (UID: \"d87b9890-61ac-4018-a312-c02787e3c371\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890845-92l2w"
Oct 31 12:45:00.562010 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:00.561404    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkf85\" (UniqueName: \"kubernetes.io/projected/352baef6-da71-44a4-b1cc-f0960db22901-kube-api-access-vkf85\") pod \"maas-api-key-cleanup-e2e-mt-ed25a690-29890845-pvxsp\" (UID: \"352baef6-da71-44a4-b1cc-f0960db22901\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-ed25a690-29890845-pvxsp"
Oct 31 12:45:00.562010 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:00.561511    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8x8z9\" (UniqueName: \"kubernetes.io/projected/f1b5757d-78b6-4614-b359-5398628f7913-kube-api-access-8x8z9\") pod \"maas-api-key-cleanup-e2e-worker-w6-67d410-29890845-6m5qx\" (UID: \"f1b5757d-78b6-4614-b359-5398628f7913\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-67d410-29890845-6m5qx"
Oct 31 12:45:00.562010 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:00.561682    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09c33526-ddf7-41df-8b1a-bf098b046b84-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-e6c79d-29890845-9lxh7\" (UID: \"09c33526-ddf7-41df-8b1a-bf098b046b84\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e6c79d-29890845-9lxh7"
Oct 31 12:45:00.562010 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:00.561708    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2t5g\" (UniqueName: \"kubernetes.io/projected/09c33526-ddf7-41df-8b1a-bf098b046b84-kube-api-access-v2t5g\") pod \"maas-api-key-cleanup-e2e-worker-w2-e6c79d-29890845-9lxh7\" (UID: \"09c33526-ddf7-41df-8b1a-bf098b046b84\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e6c79d-29890845-9lxh7"
Oct 31 12:45:00.562010 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:00.561785    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/352baef6-da71-44a4-b1cc-f0960db22901-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-ed25a690-29890845-pvxsp\" (UID: \"352baef6-da71-44a4-b1cc-f0960db22901\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-ed25a690-29890845-pvxsp"
Oct 31 12:45:00.562010 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:00.561818    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1b5757d-78b6-4614-b359-5398628f7913-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-67d410-29890845-6m5qx\" (UID: \"f1b5757d-78b6-4614-b359-5398628f7913\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-67d410-29890845-6m5qx"
Oct 31 12:45:00.568588 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:00.563345    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09c33526-ddf7-41df-8b1a-bf098b046b84-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-e6c79d-29890845-9lxh7\" (UID: \"09c33526-ddf7-41df-8b1a-bf098b046b84\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e6c79d-29890845-9lxh7"
Oct 31 12:45:00.568588 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:00.566636    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/352baef6-da71-44a4-b1cc-f0960db22901-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-ed25a690-29890845-pvxsp\" (UID: \"352baef6-da71-44a4-b1cc-f0960db22901\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-ed25a690-29890845-pvxsp"
Oct 31 12:45:00.568588 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:00.567038    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-2d6201-29890845-tm2bd"
Oct 31 12:45:00.606067 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:00.592578    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkf85\" (UniqueName: \"kubernetes.io/projected/352baef6-da71-44a4-b1cc-f0960db22901-kube-api-access-vkf85\") pod \"maas-api-key-cleanup-e2e-mt-ed25a690-29890845-pvxsp\" (UID: \"352baef6-da71-44a4-b1cc-f0960db22901\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-ed25a690-29890845-pvxsp"
Oct 31 12:45:00.606067 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:00.596239    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2t5g\" (UniqueName: \"kubernetes.io/projected/09c33526-ddf7-41df-8b1a-bf098b046b84-kube-api-access-v2t5g\") pod \"maas-api-key-cleanup-e2e-worker-w2-e6c79d-29890845-9lxh7\" (UID: \"09c33526-ddf7-41df-8b1a-bf098b046b84\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e6c79d-29890845-9lxh7"
Oct 31 12:45:00.647925 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:00.647883    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fb0343-29890845-cxhdt"
Oct 31 12:45:00.648485 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:00.648460    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-62734c-29890845-tbvmk"
Oct 31 12:45:00.675928 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:00.669240    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8x8z9\" (UniqueName: \"kubernetes.io/projected/f1b5757d-78b6-4614-b359-5398628f7913-kube-api-access-8x8z9\") pod \"maas-api-key-cleanup-e2e-worker-w6-67d410-29890845-6m5qx\" (UID: \"f1b5757d-78b6-4614-b359-5398628f7913\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-67d410-29890845-6m5qx"
Oct 31 12:45:00.675928 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:00.669699    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1b5757d-78b6-4614-b359-5398628f7913-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-67d410-29890845-6m5qx\" (UID: \"f1b5757d-78b6-4614-b359-5398628f7913\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-67d410-29890845-6m5qx"
Oct 31 12:45:00.675928 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:00.670631    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1b5757d-78b6-4614-b359-5398628f7913-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-67d410-29890845-6m5qx\" (UID: \"f1b5757d-78b6-4614-b359-5398628f7913\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-67d410-29890845-6m5qx"
Oct 31 12:45:00.684853 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:00.684763    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8x8z9\" (UniqueName: \"kubernetes.io/projected/f1b5757d-78b6-4614-b359-5398628f7913-kube-api-access-8x8z9\") pod \"maas-api-key-cleanup-e2e-worker-w6-67d410-29890845-6m5qx\" (UID: \"f1b5757d-78b6-4614-b359-5398628f7913\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-67d410-29890845-6m5qx"
Oct 31 12:45:00.697040 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:00.696791    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890845-92l2w"
Oct 31 12:45:00.698322 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:00.698252    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-ed25a690-29890845-pvxsp"
Oct 31 12:45:00.737921 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:00.737888    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e6c79d-29890845-9lxh7"
Oct 31 12:45:00.744834 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:00.744656    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-67d410-29890845-6m5qx"
Oct 31 12:45:00.745818 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:00.745654    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-8faad6-openshift-default-9ff4fdc49-fc7fk" event={"ID":"4d1ddd1b-8dc4-4706-911f-18376b4b2b68","Type":"ContainerStarted","Data":"d7d9f76e56891d8d11d5191df4ed997977d680b6bdc8f71963300232ed4a9340"}
Oct 31 12:45:00.745818 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:00.745716    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-8faad6-openshift-default-9ff4fdc49-fc7fk" event={"ID":"4d1ddd1b-8dc4-4706-911f-18376b4b2b68","Type":"ContainerStarted","Data":"b51a18b2b9cb14b4b74d2d6477a11fd1deb5fe4dc1f6fd729c471c8687e857dd"}
Oct 31 12:45:00.786544 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:00.785707    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-8faad6-openshift-default-9ff4fdc49-fc7fk" podStartSLOduration=1.785686307 podStartE2EDuration="1.785686307s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:44:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:45:00.780295368 +0000 UTC m=+1063.883363081" watchObservedRunningTime="2026-10-31 12:45:00.785686307 +0000 UTC m=+1063.888754021"
Oct 31 12:45:01.098093 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:45:01.098051    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd0d4341_08cb_40e3_9062_c24256031504.slice/crio-200620095c53c5e095fd0a54c496b9cb86e860be744ec6d875f4b28ebf6f784a WatchSource:0}: Error finding container 200620095c53c5e095fd0a54c496b9cb86e860be744ec6d875f4b28ebf6f784a: Status 404 returned error can't find the container with id 200620095c53c5e095fd0a54c496b9cb86e860be744ec6d875f4b28ebf6f784a
Oct 31 12:45:01.138102 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:01.138029    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1449a717-8c6d-4eed-b3e0-4475a92e75c3" path="/var/lib/kubelet/pods/1449a717-8c6d-4eed-b3e0-4475a92e75c3/volumes"
Oct 31 12:45:01.139100 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:01.139072    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b44cd3d-8334-452f-937f-a3dd8904d3f2" path="/var/lib/kubelet/pods/7b44cd3d-8334-452f-937f-a3dd8904d3f2/volumes"
Oct 31 12:45:01.139757 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:01.139733    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c469bb8f-7712-4c3d-8bd4-af409971ef16" path="/var/lib/kubelet/pods/c469bb8f-7712-4c3d-8bd4-af409971ef16/volumes"
Oct 31 12:45:01.144319 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:01.144267    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-2d6201-29890845-tm2bd"]
Oct 31 12:45:01.511039 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:01.511001    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fb0343-29890845-cxhdt"]
Oct 31 12:45:01.539093 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:01.537564    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-62734c-29890845-tbvmk"]
Oct 31 12:45:01.560240 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:45:01.560195    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde4cc1ae_6e6f_4124_a0cd_464549508f9e.slice/crio-9c1918e697137e873f372e51bd8dfb89191acc77747f3980028d94a30e053544 WatchSource:0}: Error finding container 9c1918e697137e873f372e51bd8dfb89191acc77747f3980028d94a30e053544: Status 404 returned error can't find the container with id 9c1918e697137e873f372e51bd8dfb89191acc77747f3980028d94a30e053544
Oct 31 12:45:01.565810 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:45:01.565633    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5361e3ad_5107_41cc_8afa_f0c64e8236b1.slice/crio-67b2aad281fc70873a3fc6a3e21b4024f4d4075eef105abbb7ca9f5a3a433429 WatchSource:0}: Error finding container 67b2aad281fc70873a3fc6a3e21b4024f4d4075eef105abbb7ca9f5a3a433429: Status 404 returned error can't find the container with id 67b2aad281fc70873a3fc6a3e21b4024f4d4075eef105abbb7ca9f5a3a433429
Oct 31 12:45:01.566146 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:45:01.566120    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd87b9890_61ac_4018_a312_c02787e3c371.slice/crio-1981eb813b2628a6cb36dd904b88c52d130f458f5848df57dfc7494f7ba6a509 WatchSource:0}: Error finding container 1981eb813b2628a6cb36dd904b88c52d130f458f5848df57dfc7494f7ba6a509: Status 404 returned error can't find the container with id 1981eb813b2628a6cb36dd904b88c52d130f458f5848df57dfc7494f7ba6a509
Oct 31 12:45:01.591251 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:45:01.591228    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1b5757d_78b6_4614_b359_5398628f7913.slice/crio-8f7a93440ff80d46e51702bf118ea949090067121b359f98e6548c97ff55c48b WatchSource:0}: Error finding container 8f7a93440ff80d46e51702bf118ea949090067121b359f98e6548c97ff55c48b: Status 404 returned error can't find the container with id 8f7a93440ff80d46e51702bf118ea949090067121b359f98e6548c97ff55c48b
Oct 31 12:45:01.619564 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:01.615457    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890845-92l2w"]
Oct 31 12:45:01.631656 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:01.631571    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-67d410-29890845-6m5qx"]
Oct 31 12:45:01.634815 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:45:01.634584    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09c33526_ddf7_41df_8b1a_bf098b046b84.slice/crio-3baf4733f3a6679d07b12678e9b9dc51a5756fef7e17110a79a82c7b4332c27c WatchSource:0}: Error finding container 3baf4733f3a6679d07b12678e9b9dc51a5756fef7e17110a79a82c7b4332c27c: Status 404 returned error can't find the container with id 3baf4733f3a6679d07b12678e9b9dc51a5756fef7e17110a79a82c7b4332c27c
Oct 31 12:45:01.639117 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:01.639051    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e6c79d-29890845-9lxh7"]
Oct 31 12:45:01.763231 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:01.737937    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-8faad6-openshift-default-9ff4fdc49-fc7fk"
Oct 31 12:45:02.010045 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:01.999795    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-8faad6-openshift-default-9ff4fdc49-fc7fk"
Oct 31 12:45:02.059547 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:02.039385    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-ed25a690-29890845-pvxsp"]
Oct 31 12:45:02.318259 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:02.318175    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-67d410-29890845-6m5qx" event={"ID":"f1b5757d-78b6-4614-b359-5398628f7913","Type":"ContainerStarted","Data":"8f7a93440ff80d46e51702bf118ea949090067121b359f98e6548c97ff55c48b"}
Oct 31 12:45:02.327743 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:02.327709    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-ed25a690-29890845-pvxsp" event={"ID":"352baef6-da71-44a4-b1cc-f0960db22901","Type":"ContainerStarted","Data":"38a21b8a4f2a23460d5e80338e9d0cbe0bb2958b79db0f2d2b8e327e5898f869"}
Oct 31 12:45:02.349673 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:02.349633    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fb0343-29890845-cxhdt" event={"ID":"de4cc1ae-6e6f-4124-a0cd-464549508f9e","Type":"ContainerStarted","Data":"9c1918e697137e873f372e51bd8dfb89191acc77747f3980028d94a30e053544"}
Oct 31 12:45:02.352975 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:02.352944    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e6c79d-29890845-9lxh7" event={"ID":"09c33526-ddf7-41df-8b1a-bf098b046b84","Type":"ContainerStarted","Data":"3baf4733f3a6679d07b12678e9b9dc51a5756fef7e17110a79a82c7b4332c27c"}
Oct 31 12:45:02.361843 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:02.361807    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-62734c-29890845-tbvmk" event={"ID":"5361e3ad-5107-41cc-8afa-f0c64e8236b1","Type":"ContainerStarted","Data":"67b2aad281fc70873a3fc6a3e21b4024f4d4075eef105abbb7ca9f5a3a433429"}
Oct 31 12:45:02.363587 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:02.363562    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890845-92l2w" event={"ID":"d87b9890-61ac-4018-a312-c02787e3c371","Type":"ContainerStarted","Data":"1981eb813b2628a6cb36dd904b88c52d130f458f5848df57dfc7494f7ba6a509"}
Oct 31 12:45:02.368213 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:02.368186    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-2d6201-29890845-tm2bd" event={"ID":"dd0d4341-08cb-40e3-9062-c24256031504","Type":"ContainerStarted","Data":"8a2fe4ac6f281cc10ed8736a271d8b5fde4803e8685236813c97bdf77fa4d99b"}
Oct 31 12:45:02.368344 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:02.368239    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-2d6201-29890845-tm2bd" event={"ID":"dd0d4341-08cb-40e3-9062-c24256031504","Type":"ContainerStarted","Data":"200620095c53c5e095fd0a54c496b9cb86e860be744ec6d875f4b28ebf6f784a"}
Oct 31 12:45:02.368845 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:02.368819    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-8faad6-openshift-default-9ff4fdc49-fc7fk"
Oct 31 12:45:02.370891 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:02.370699    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-8faad6-openshift-default-9ff4fdc49-fc7fk"
Oct 31 12:45:02.371690 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:02.371626    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fb0343-29890845-cxhdt" podStartSLOduration=2.371607157 podStartE2EDuration="2.371607157s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:45:02.370201747 +0000 UTC m=+1065.473269461" watchObservedRunningTime="2026-10-31 12:45:02.371607157 +0000 UTC m=+1065.474674873"
Oct 31 12:45:02.405068 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:02.405007    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-2d6201-29890845-tm2bd" podStartSLOduration=2.404988439 podStartE2EDuration="2.404988439s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:45:02.398886392 +0000 UTC m=+1065.501954105" watchObservedRunningTime="2026-10-31 12:45:02.404988439 +0000 UTC m=+1065.508056153"
Oct 31 12:45:03.050231 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:03.050201    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c51ce5e-t4xfg"]
Oct 31 12:45:03.050231 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:03.050239    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c51ce5e-t4xfg"]
Oct 31 12:45:03.386380 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:03.386340    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-67d410-29890845-6m5qx" event={"ID":"f1b5757d-78b6-4614-b359-5398628f7913","Type":"ContainerStarted","Data":"0aaaec949983d26e2264f6c453c770f101f7fa114b0e02f0a931db96a61f8ef1"}
Oct 31 12:45:03.388915 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:03.388760    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-ed25a690-29890845-pvxsp" event={"ID":"352baef6-da71-44a4-b1cc-f0960db22901","Type":"ContainerStarted","Data":"6c5b11607f61d3ae3126eb929c458acb9f1b1a241f27c32b84840080c9abea12"}
Oct 31 12:45:03.391656 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:03.391629    5060 generic.go:356] "Generic (PLEG): container finished" podID="de4cc1ae-6e6f-4124-a0cd-464549508f9e" containerID="fe9d9c18ff042fb66d83fb676bbb4bfa2b961945e4d3fe09fef99fe1cc756b87" exitCode=60
Oct 31 12:45:03.391804 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:03.391704    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fb0343-29890845-cxhdt" event={"ID":"de4cc1ae-6e6f-4124-a0cd-464549508f9e","Type":"ContainerDied","Data":"fe9d9c18ff042fb66d83fb676bbb4bfa2b961945e4d3fe09fef99fe1cc756b87"}
Oct 31 12:45:03.392037 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:03.392019    5060 scope.go:122] "RemoveContainer" containerID="fe9d9c18ff042fb66d83fb676bbb4bfa2b961945e4d3fe09fef99fe1cc756b87"
Oct 31 12:45:03.393837 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:03.393811    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e6c79d-29890845-9lxh7" event={"ID":"09c33526-ddf7-41df-8b1a-bf098b046b84","Type":"ContainerStarted","Data":"b093363e94f05433bfd08698e236d68a50886bc4afe3aa9f6320dd72a6dccba6"}
Oct 31 12:45:03.396371 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:03.396350    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-62734c-29890845-tbvmk" event={"ID":"5361e3ad-5107-41cc-8afa-f0c64e8236b1","Type":"ContainerStarted","Data":"3d444cf5cdf7dc9e294458adead651fc185b727476ff09e28ddf244f30354ffd"}
Oct 31 12:45:03.398481 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:03.398453    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890845-92l2w" event={"ID":"d87b9890-61ac-4018-a312-c02787e3c371","Type":"ContainerStarted","Data":"e0092fe7d928e0cd52241dace262309fccf9a6523a05cd6e78cefd72dee391f9"}
Oct 31 12:45:03.400443 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:03.400417    5060 generic.go:356] "Generic (PLEG): container finished" podID="dd0d4341-08cb-40e3-9062-c24256031504" containerID="8a2fe4ac6f281cc10ed8736a271d8b5fde4803e8685236813c97bdf77fa4d99b" exitCode=60
Oct 31 12:45:03.400600 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:03.400537    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-2d6201-29890845-tm2bd" event={"ID":"dd0d4341-08cb-40e3-9062-c24256031504","Type":"ContainerDied","Data":"8a2fe4ac6f281cc10ed8736a271d8b5fde4803e8685236813c97bdf77fa4d99b"}
Oct 31 12:45:03.400736 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:03.400720    5060 scope.go:122] "RemoveContainer" containerID="8a2fe4ac6f281cc10ed8736a271d8b5fde4803e8685236813c97bdf77fa4d99b"
Oct 31 12:45:03.411780 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:03.411737    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-67d410-29890845-6m5qx" podStartSLOduration=3.411720769 podStartE2EDuration="3.411720769s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:45:03.409293121 +0000 UTC m=+1066.512360844" watchObservedRunningTime="2026-10-31 12:45:03.411720769 +0000 UTC m=+1066.514788510"
Oct 31 12:45:03.532872 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:03.532053    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e6c79d-29890845-9lxh7" podStartSLOduration=3.532030389 podStartE2EDuration="3.532030389s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:45:03.449003319 +0000 UTC m=+1066.552071062" watchObservedRunningTime="2026-10-31 12:45:03.532030389 +0000 UTC m=+1066.635098103"
Oct 31 12:45:03.587070 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:03.585919    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890845-92l2w" podStartSLOduration=3.585896651 podStartE2EDuration="3.585896651s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:45:03.540970888 +0000 UTC m=+1066.644038602" watchObservedRunningTime="2026-10-31 12:45:03.585896651 +0000 UTC m=+1066.688964365"
Oct 31 12:45:03.604968 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:03.600904    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-62734c-29890845-tbvmk" podStartSLOduration=3.600884617 podStartE2EDuration="3.600884617s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:45:03.593347858 +0000 UTC m=+1066.696415569" watchObservedRunningTime="2026-10-31 12:45:03.600884617 +0000 UTC m=+1066.703952330"
Oct 31 12:45:03.639271 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:03.639051    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-ed25a690-29890845-pvxsp" podStartSLOduration=3.639035572 podStartE2EDuration="3.639035572s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:45:03.622880688 +0000 UTC m=+1066.725948399" watchObservedRunningTime="2026-10-31 12:45:03.639035572 +0000 UTC m=+1066.742103284"
Oct 31 12:45:04.587611 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:04.586882    5060 generic.go:356] "Generic (PLEG): container finished" podID="f1b5757d-78b6-4614-b359-5398628f7913" containerID="0aaaec949983d26e2264f6c453c770f101f7fa114b0e02f0a931db96a61f8ef1" exitCode=60
Oct 31 12:45:04.587611 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:04.587045    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-67d410-29890845-6m5qx" event={"ID":"f1b5757d-78b6-4614-b359-5398628f7913","Type":"ContainerDied","Data":"0aaaec949983d26e2264f6c453c770f101f7fa114b0e02f0a931db96a61f8ef1"}
Oct 31 12:45:04.588624 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:04.588592    5060 scope.go:122] "RemoveContainer" containerID="0aaaec949983d26e2264f6c453c770f101f7fa114b0e02f0a931db96a61f8ef1"
Oct 31 12:45:04.673180 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:04.670999    5060 generic.go:356] "Generic (PLEG): container finished" podID="352baef6-da71-44a4-b1cc-f0960db22901" containerID="6c5b11607f61d3ae3126eb929c458acb9f1b1a241f27c32b84840080c9abea12" exitCode=60
Oct 31 12:45:04.673180 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:04.671161    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-ed25a690-29890845-pvxsp" event={"ID":"352baef6-da71-44a4-b1cc-f0960db22901","Type":"ContainerDied","Data":"6c5b11607f61d3ae3126eb929c458acb9f1b1a241f27c32b84840080c9abea12"}
Oct 31 12:45:04.673180 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:04.671935    5060 scope.go:122] "RemoveContainer" containerID="6c5b11607f61d3ae3126eb929c458acb9f1b1a241f27c32b84840080c9abea12"
Oct 31 12:45:04.734894 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:04.699991    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fb0343-29890845-cxhdt" event={"ID":"de4cc1ae-6e6f-4124-a0cd-464549508f9e","Type":"ContainerStarted","Data":"2a1cf6ca4dd02f62f5689c3dbe9da74ddc83c40744557cdd05a3201ad5f357e5"}
Oct 31 12:45:04.734894 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:04.707128    5060 generic.go:356] "Generic (PLEG): container finished" podID="09c33526-ddf7-41df-8b1a-bf098b046b84" containerID="b093363e94f05433bfd08698e236d68a50886bc4afe3aa9f6320dd72a6dccba6" exitCode=60
Oct 31 12:45:04.734894 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:04.707285    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e6c79d-29890845-9lxh7" event={"ID":"09c33526-ddf7-41df-8b1a-bf098b046b84","Type":"ContainerDied","Data":"b093363e94f05433bfd08698e236d68a50886bc4afe3aa9f6320dd72a6dccba6"}
Oct 31 12:45:04.734894 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:04.707842    5060 scope.go:122] "RemoveContainer" containerID="b093363e94f05433bfd08698e236d68a50886bc4afe3aa9f6320dd72a6dccba6"
Oct 31 12:45:04.734894 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:04.732720    5060 generic.go:356] "Generic (PLEG): container finished" podID="5361e3ad-5107-41cc-8afa-f0c64e8236b1" containerID="3d444cf5cdf7dc9e294458adead651fc185b727476ff09e28ddf244f30354ffd" exitCode=60
Oct 31 12:45:04.734894 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:04.733004    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-62734c-29890845-tbvmk" event={"ID":"5361e3ad-5107-41cc-8afa-f0c64e8236b1","Type":"ContainerDied","Data":"3d444cf5cdf7dc9e294458adead651fc185b727476ff09e28ddf244f30354ffd"}
Oct 31 12:45:04.734894 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:04.733579    5060 scope.go:122] "RemoveContainer" containerID="3d444cf5cdf7dc9e294458adead651fc185b727476ff09e28ddf244f30354ffd"
Oct 31 12:45:04.737862 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:04.737826    5060 generic.go:356] "Generic (PLEG): container finished" podID="d87b9890-61ac-4018-a312-c02787e3c371" containerID="e0092fe7d928e0cd52241dace262309fccf9a6523a05cd6e78cefd72dee391f9" exitCode=60
Oct 31 12:45:04.738039 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:04.737923    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890845-92l2w" event={"ID":"d87b9890-61ac-4018-a312-c02787e3c371","Type":"ContainerDied","Data":"e0092fe7d928e0cd52241dace262309fccf9a6523a05cd6e78cefd72dee391f9"}
Oct 31 12:45:04.738406 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:04.738384    5060 scope.go:122] "RemoveContainer" containerID="e0092fe7d928e0cd52241dace262309fccf9a6523a05cd6e78cefd72dee391f9"
Oct 31 12:45:04.772740 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:04.772703    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-2d6201-29890845-tm2bd" event={"ID":"dd0d4341-08cb-40e3-9062-c24256031504","Type":"ContainerStarted","Data":"6dcf2ea6b61540a89880d45c677894e4986b913bb7dd0e6e64bfffe0a9215600"}
Oct 31 12:45:05.051570 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:05.051539    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c3204bd-0aa1-4adb-877f-e435f0a1725a" path="/var/lib/kubelet/pods/3c3204bd-0aa1-4adb-877f-e435f0a1725a/volumes"
Oct 31 12:45:05.788752 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:05.788708    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890845-92l2w" event={"ID":"d87b9890-61ac-4018-a312-c02787e3c371","Type":"ContainerStarted","Data":"3ab544abb92df8ed94885c493ea81134ae67c9b04bb9712e34d1f006e4c6031c"}
Oct 31 12:45:05.791117 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:05.791081    5060 generic.go:356] "Generic (PLEG): container finished" podID="dd0d4341-08cb-40e3-9062-c24256031504" containerID="6dcf2ea6b61540a89880d45c677894e4986b913bb7dd0e6e64bfffe0a9215600" exitCode=60
Oct 31 12:45:05.791246 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:05.791165    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-2d6201-29890845-tm2bd" event={"ID":"dd0d4341-08cb-40e3-9062-c24256031504","Type":"ContainerDied","Data":"6dcf2ea6b61540a89880d45c677894e4986b913bb7dd0e6e64bfffe0a9215600"}
Oct 31 12:45:05.791246 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:05.791229    5060 scope.go:122] "RemoveContainer" containerID="8a2fe4ac6f281cc10ed8736a271d8b5fde4803e8685236813c97bdf77fa4d99b"
Oct 31 12:45:05.791480 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:05.791460    5060 scope.go:122] "RemoveContainer" containerID="6dcf2ea6b61540a89880d45c677894e4986b913bb7dd0e6e64bfffe0a9215600"
Oct 31 12:45:05.792046 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:45:05.791869    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-2d6201-29890845-tm2bd_odh-ai-gateway-infra(dd0d4341-08cb-40e3-9062-c24256031504)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-2d6201-29890845-tm2bd" podUID="dd0d4341-08cb-40e3-9062-c24256031504"
Oct 31 12:45:05.793919 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:05.793887    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-67d410-29890845-6m5qx" event={"ID":"f1b5757d-78b6-4614-b359-5398628f7913","Type":"ContainerStarted","Data":"ac287b2eaf2128ed22fa26a0ae28f59a166006b12fd77e383aa957ce3339e30a"}
Oct 31 12:45:05.796294 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:05.796266    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-ed25a690-29890845-pvxsp" event={"ID":"352baef6-da71-44a4-b1cc-f0960db22901","Type":"ContainerStarted","Data":"5a9d979681590812140679756bf0833174fceb811e9e730755c1e11275e1335e"}
Oct 31 12:45:05.798637 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:05.798613    5060 generic.go:356] "Generic (PLEG): container finished" podID="de4cc1ae-6e6f-4124-a0cd-464549508f9e" containerID="2a1cf6ca4dd02f62f5689c3dbe9da74ddc83c40744557cdd05a3201ad5f357e5" exitCode=60
Oct 31 12:45:05.798732 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:05.798693    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fb0343-29890845-cxhdt" event={"ID":"de4cc1ae-6e6f-4124-a0cd-464549508f9e","Type":"ContainerDied","Data":"2a1cf6ca4dd02f62f5689c3dbe9da74ddc83c40744557cdd05a3201ad5f357e5"}
Oct 31 12:45:05.798949 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:05.798931    5060 scope.go:122] "RemoveContainer" containerID="2a1cf6ca4dd02f62f5689c3dbe9da74ddc83c40744557cdd05a3201ad5f357e5"
Oct 31 12:45:05.799287 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:45:05.799262    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-fb0343-29890845-cxhdt_odh-ai-gateway-infra(de4cc1ae-6e6f-4124-a0cd-464549508f9e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fb0343-29890845-cxhdt" podUID="de4cc1ae-6e6f-4124-a0cd-464549508f9e"
Oct 31 12:45:05.801034 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:05.800970    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e6c79d-29890845-9lxh7" event={"ID":"09c33526-ddf7-41df-8b1a-bf098b046b84","Type":"ContainerStarted","Data":"07d3a57bd3fe730ec0d7a60dbe432c87f73c6466dd95ec6ab817d6f875b7264b"}
Oct 31 12:45:05.803544 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:05.803519    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-62734c-29890845-tbvmk" event={"ID":"5361e3ad-5107-41cc-8afa-f0c64e8236b1","Type":"ContainerStarted","Data":"805b2810854b16c730d546248fc36bc3d28a3c3ba63214343f7abf818910d9af"}
Oct 31 12:45:05.813531 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:05.811538    5060 scope.go:122] "RemoveContainer" containerID="fe9d9c18ff042fb66d83fb676bbb4bfa2b961945e4d3fe09fef99fe1cc756b87"
Oct 31 12:45:06.928740 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:06.928476    5060 generic.go:356] "Generic (PLEG): container finished" podID="f1b5757d-78b6-4614-b359-5398628f7913" containerID="ac287b2eaf2128ed22fa26a0ae28f59a166006b12fd77e383aa957ce3339e30a" exitCode=60
Oct 31 12:45:06.949524 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:06.930635    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-67d410-29890845-6m5qx" event={"ID":"f1b5757d-78b6-4614-b359-5398628f7913","Type":"ContainerDied","Data":"ac287b2eaf2128ed22fa26a0ae28f59a166006b12fd77e383aa957ce3339e30a"}
Oct 31 12:45:06.949524 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:06.930746    5060 scope.go:122] "RemoveContainer" containerID="0aaaec949983d26e2264f6c453c770f101f7fa114b0e02f0a931db96a61f8ef1"
Oct 31 12:45:06.949524 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:06.931267    5060 scope.go:122] "RemoveContainer" containerID="ac287b2eaf2128ed22fa26a0ae28f59a166006b12fd77e383aa957ce3339e30a"
Oct 31 12:45:06.949524 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:45:06.931822    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-67d410-29890845-6m5qx_odh-ai-gateway-infra(f1b5757d-78b6-4614-b359-5398628f7913)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-67d410-29890845-6m5qx" podUID="f1b5757d-78b6-4614-b359-5398628f7913"
Oct 31 12:45:06.949524 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:06.939552    5060 generic.go:356] "Generic (PLEG): container finished" podID="352baef6-da71-44a4-b1cc-f0960db22901" containerID="5a9d979681590812140679756bf0833174fceb811e9e730755c1e11275e1335e" exitCode=60
Oct 31 12:45:06.949524 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:06.939718    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-ed25a690-29890845-pvxsp" event={"ID":"352baef6-da71-44a4-b1cc-f0960db22901","Type":"ContainerDied","Data":"5a9d979681590812140679756bf0833174fceb811e9e730755c1e11275e1335e"}
Oct 31 12:45:06.949524 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:06.940396    5060 scope.go:122] "RemoveContainer" containerID="5a9d979681590812140679756bf0833174fceb811e9e730755c1e11275e1335e"
Oct 31 12:45:06.949524 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:45:06.941973    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-ed25a690-29890845-pvxsp_odh-ai-gateway-infra(352baef6-da71-44a4-b1cc-f0960db22901)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-ed25a690-29890845-pvxsp" podUID="352baef6-da71-44a4-b1cc-f0960db22901"
Oct 31 12:45:06.949524 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:06.943860    5060 scope.go:122] "RemoveContainer" containerID="2a1cf6ca4dd02f62f5689c3dbe9da74ddc83c40744557cdd05a3201ad5f357e5"
Oct 31 12:45:06.949524 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:45:06.944181    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-fb0343-29890845-cxhdt_odh-ai-gateway-infra(de4cc1ae-6e6f-4124-a0cd-464549508f9e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fb0343-29890845-cxhdt" podUID="de4cc1ae-6e6f-4124-a0cd-464549508f9e"
Oct 31 12:45:06.949524 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:06.947273    5060 generic.go:356] "Generic (PLEG): container finished" podID="09c33526-ddf7-41df-8b1a-bf098b046b84" containerID="07d3a57bd3fe730ec0d7a60dbe432c87f73c6466dd95ec6ab817d6f875b7264b" exitCode=60
Oct 31 12:45:06.949524 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:06.947392    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e6c79d-29890845-9lxh7" event={"ID":"09c33526-ddf7-41df-8b1a-bf098b046b84","Type":"ContainerDied","Data":"07d3a57bd3fe730ec0d7a60dbe432c87f73c6466dd95ec6ab817d6f875b7264b"}
Oct 31 12:45:06.949524 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:06.947716    5060 scope.go:122] "RemoveContainer" containerID="07d3a57bd3fe730ec0d7a60dbe432c87f73c6466dd95ec6ab817d6f875b7264b"
Oct 31 12:45:06.949524 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:45:06.948021    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-e6c79d-29890845-9lxh7_odh-ai-gateway-infra(09c33526-ddf7-41df-8b1a-bf098b046b84)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e6c79d-29890845-9lxh7" podUID="09c33526-ddf7-41df-8b1a-bf098b046b84"
Oct 31 12:45:06.950913 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:06.950566    5060 generic.go:356] "Generic (PLEG): container finished" podID="5361e3ad-5107-41cc-8afa-f0c64e8236b1" containerID="805b2810854b16c730d546248fc36bc3d28a3c3ba63214343f7abf818910d9af" exitCode=60
Oct 31 12:45:06.950913 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:06.950810    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-62734c-29890845-tbvmk" event={"ID":"5361e3ad-5107-41cc-8afa-f0c64e8236b1","Type":"ContainerDied","Data":"805b2810854b16c730d546248fc36bc3d28a3c3ba63214343f7abf818910d9af"}
Oct 31 12:45:06.959609 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:06.951150    5060 scope.go:122] "RemoveContainer" containerID="805b2810854b16c730d546248fc36bc3d28a3c3ba63214343f7abf818910d9af"
Oct 31 12:45:06.959609 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:45:06.951521    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-62734c-29890845-tbvmk_odh-ai-gateway-infra(5361e3ad-5107-41cc-8afa-f0c64e8236b1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-62734c-29890845-tbvmk" podUID="5361e3ad-5107-41cc-8afa-f0c64e8236b1"
Oct 31 12:45:06.978763 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:06.977686    5060 generic.go:356] "Generic (PLEG): container finished" podID="d87b9890-61ac-4018-a312-c02787e3c371" containerID="3ab544abb92df8ed94885c493ea81134ae67c9b04bb9712e34d1f006e4c6031c" exitCode=60
Oct 31 12:45:06.978763 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:06.977847    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890845-92l2w" event={"ID":"d87b9890-61ac-4018-a312-c02787e3c371","Type":"ContainerDied","Data":"3ab544abb92df8ed94885c493ea81134ae67c9b04bb9712e34d1f006e4c6031c"}
Oct 31 12:45:06.978763 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:06.978319    5060 scope.go:122] "RemoveContainer" containerID="3ab544abb92df8ed94885c493ea81134ae67c9b04bb9712e34d1f006e4c6031c"
Oct 31 12:45:06.985786 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:45:06.979966    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890845-92l2w_odh-ai-gateway-infra(d87b9890-61ac-4018-a312-c02787e3c371)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890845-92l2w" podUID="d87b9890-61ac-4018-a312-c02787e3c371"
Oct 31 12:45:06.985786 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:06.984473    5060 scope.go:122] "RemoveContainer" containerID="6dcf2ea6b61540a89880d45c677894e4986b913bb7dd0e6e64bfffe0a9215600"
Oct 31 12:45:06.985786 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:45:06.984835    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-2d6201-29890845-tm2bd_odh-ai-gateway-infra(dd0d4341-08cb-40e3-9062-c24256031504)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-2d6201-29890845-tm2bd" podUID="dd0d4341-08cb-40e3-9062-c24256031504"
Oct 31 12:45:07.097996 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:07.097955    5060 scope.go:122] "RemoveContainer" containerID="6c5b11607f61d3ae3126eb929c458acb9f1b1a241f27c32b84840080c9abea12"
Oct 31 12:45:07.115405 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:07.115339    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3eb4bba9-52x2t"]
Oct 31 12:45:07.120902 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:07.120875    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3eb4bba9-52x2t"]
Oct 31 12:45:07.124866 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:07.124843    5060 scope.go:122] "RemoveContainer" containerID="b093363e94f05433bfd08698e236d68a50886bc4afe3aa9f6320dd72a6dccba6"
Oct 31 12:45:07.145839 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:07.145718    5060 scope.go:122] "RemoveContainer" containerID="3d444cf5cdf7dc9e294458adead651fc185b727476ff09e28ddf244f30354ffd"
Oct 31 12:45:07.169474 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:07.169445    5060 scope.go:122] "RemoveContainer" containerID="e0092fe7d928e0cd52241dace262309fccf9a6523a05cd6e78cefd72dee391f9"
Oct 31 12:45:08.162393 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:08.162363    5060 scope.go:122] "RemoveContainer" containerID="ac287b2eaf2128ed22fa26a0ae28f59a166006b12fd77e383aa957ce3339e30a"
Oct 31 12:45:08.172923 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:45:08.172799    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-67d410-29890845-6m5qx_odh-ai-gateway-infra(f1b5757d-78b6-4614-b359-5398628f7913)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-67d410-29890845-6m5qx" podUID="f1b5757d-78b6-4614-b359-5398628f7913"
Oct 31 12:45:08.194851 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:08.194819    5060 scope.go:122] "RemoveContainer" containerID="5a9d979681590812140679756bf0833174fceb811e9e730755c1e11275e1335e"
Oct 31 12:45:08.195623 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:45:08.195587    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-ed25a690-29890845-pvxsp_odh-ai-gateway-infra(352baef6-da71-44a4-b1cc-f0960db22901)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-ed25a690-29890845-pvxsp" podUID="352baef6-da71-44a4-b1cc-f0960db22901"
Oct 31 12:45:08.331804 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:08.329701    5060 scope.go:122] "RemoveContainer" containerID="07d3a57bd3fe730ec0d7a60dbe432c87f73c6466dd95ec6ab817d6f875b7264b"
Oct 31 12:45:08.331804 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:45:08.330209    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-e6c79d-29890845-9lxh7_odh-ai-gateway-infra(09c33526-ddf7-41df-8b1a-bf098b046b84)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e6c79d-29890845-9lxh7" podUID="09c33526-ddf7-41df-8b1a-bf098b046b84"
Oct 31 12:45:08.331804 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:08.330748    5060 scope.go:122] "RemoveContainer" containerID="805b2810854b16c730d546248fc36bc3d28a3c3ba63214343f7abf818910d9af"
Oct 31 12:45:08.331804 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:45:08.331116    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-62734c-29890845-tbvmk_odh-ai-gateway-infra(5361e3ad-5107-41cc-8afa-f0c64e8236b1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-62734c-29890845-tbvmk" podUID="5361e3ad-5107-41cc-8afa-f0c64e8236b1"
Oct 31 12:45:08.391754 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:08.391723    5060 scope.go:122] "RemoveContainer" containerID="3ab544abb92df8ed94885c493ea81134ae67c9b04bb9712e34d1f006e4c6031c"
Oct 31 12:45:08.392462 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:45:08.392431    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890845-92l2w_odh-ai-gateway-infra(d87b9890-61ac-4018-a312-c02787e3c371)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890845-92l2w" podUID="d87b9890-61ac-4018-a312-c02787e3c371"
Oct 31 12:45:09.073041 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:09.073003    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="627354b5-bbf0-46b0-80f6-5a0ce90b377c" path="/var/lib/kubelet/pods/627354b5-bbf0-46b0-80f6-5a0ce90b377c/volumes"
Oct 31 12:45:11.909515 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:11.908805    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b8ceeeb-9sbgk"]
Oct 31 12:45:11.932819 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:11.921959    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b8ceeeb-9sbgk"
Oct 31 12:45:11.932819 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:11.927607    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 12:45:11.932819 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:11.929544    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b8ceeeb-9sbgk"]
Oct 31 12:45:12.023533 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:12.021902    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd073675-8da6-4d49-bf2d-bf6fd1340b51-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4b8ceeeb-9sbgk\" (UID: \"cd073675-8da6-4d49-bf2d-bf6fd1340b51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b8ceeeb-9sbgk"
Oct 31 12:45:12.023533 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:12.021998    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgbck\" (UniqueName: \"kubernetes.io/projected/cd073675-8da6-4d49-bf2d-bf6fd1340b51-kube-api-access-dgbck\") pod \"maas-api-revoke-sub-4b8ceeeb-9sbgk\" (UID: \"cd073675-8da6-4d49-bf2d-bf6fd1340b51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b8ceeeb-9sbgk"
Oct 31 12:45:12.023533 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:12.022067    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd073675-8da6-4d49-bf2d-bf6fd1340b51-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4b8ceeeb-9sbgk\" (UID: \"cd073675-8da6-4d49-bf2d-bf6fd1340b51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b8ceeeb-9sbgk"
Oct 31 12:45:12.123639 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:12.123601    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd073675-8da6-4d49-bf2d-bf6fd1340b51-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4b8ceeeb-9sbgk\" (UID: \"cd073675-8da6-4d49-bf2d-bf6fd1340b51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b8ceeeb-9sbgk"
Oct 31 12:45:12.123824 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:12.123677    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgbck\" (UniqueName: \"kubernetes.io/projected/cd073675-8da6-4d49-bf2d-bf6fd1340b51-kube-api-access-dgbck\") pod \"maas-api-revoke-sub-4b8ceeeb-9sbgk\" (UID: \"cd073675-8da6-4d49-bf2d-bf6fd1340b51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b8ceeeb-9sbgk"
Oct 31 12:45:12.123824 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:12.123738    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd073675-8da6-4d49-bf2d-bf6fd1340b51-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4b8ceeeb-9sbgk\" (UID: \"cd073675-8da6-4d49-bf2d-bf6fd1340b51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b8ceeeb-9sbgk"
Oct 31 12:45:12.124768 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:12.124747    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd073675-8da6-4d49-bf2d-bf6fd1340b51-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4b8ceeeb-9sbgk\" (UID: \"cd073675-8da6-4d49-bf2d-bf6fd1340b51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b8ceeeb-9sbgk"
Oct 31 12:45:12.127354 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:12.127330    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd073675-8da6-4d49-bf2d-bf6fd1340b51-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4b8ceeeb-9sbgk\" (UID: \"cd073675-8da6-4d49-bf2d-bf6fd1340b51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b8ceeeb-9sbgk"
Oct 31 12:45:12.140671 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:12.140633    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgbck\" (UniqueName: \"kubernetes.io/projected/cd073675-8da6-4d49-bf2d-bf6fd1340b51-kube-api-access-dgbck\") pod \"maas-api-revoke-sub-4b8ceeeb-9sbgk\" (UID: \"cd073675-8da6-4d49-bf2d-bf6fd1340b51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b8ceeeb-9sbgk"
Oct 31 12:45:12.245564 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:12.243410    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b8ceeeb-9sbgk"
Oct 31 12:45:12.587265 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:45:12.587216    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd073675_8da6_4d49_bf2d_bf6fd1340b51.slice/crio-1cccc6b59baba09bb0f0dba8314d0bf2adc01e74c548d54c97f2feecdf674b7e WatchSource:0}: Error finding container 1cccc6b59baba09bb0f0dba8314d0bf2adc01e74c548d54c97f2feecdf674b7e: Status 404 returned error can't find the container with id 1cccc6b59baba09bb0f0dba8314d0bf2adc01e74c548d54c97f2feecdf674b7e
Oct 31 12:45:12.588424 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:12.588399    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b8ceeeb-9sbgk"]
Oct 31 12:45:13.575854 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:13.575811    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b8ceeeb-9sbgk" event={"ID":"cd073675-8da6-4d49-bf2d-bf6fd1340b51","Type":"ContainerStarted","Data":"d96e44f8637e694f466bd30c2363a2268c34547106ca907b518782800d362ca0"}
Oct 31 12:45:13.576268 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:13.575872    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b8ceeeb-9sbgk" event={"ID":"cd073675-8da6-4d49-bf2d-bf6fd1340b51","Type":"ContainerStarted","Data":"1cccc6b59baba09bb0f0dba8314d0bf2adc01e74c548d54c97f2feecdf674b7e"}
Oct 31 12:45:13.599135 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:13.599079    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b8ceeeb-9sbgk" podStartSLOduration=2.599064416 podStartE2EDuration="2.599064416s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:45:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:45:13.594083568 +0000 UTC m=+1076.697151281" watchObservedRunningTime="2026-10-31 12:45:13.599064416 +0000 UTC m=+1076.702132127"
Oct 31 12:45:14.595652 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:14.595610    5060 generic.go:356] "Generic (PLEG): container finished" podID="cd073675-8da6-4d49-bf2d-bf6fd1340b51" containerID="d96e44f8637e694f466bd30c2363a2268c34547106ca907b518782800d362ca0" exitCode=0
Oct 31 12:45:14.596067 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:14.595692    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b8ceeeb-9sbgk" event={"ID":"cd073675-8da6-4d49-bf2d-bf6fd1340b51","Type":"ContainerDied","Data":"d96e44f8637e694f466bd30c2363a2268c34547106ca907b518782800d362ca0"}
Oct 31 12:45:15.805666 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:15.805643    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b8ceeeb-9sbgk"
Oct 31 12:45:15.912823 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:15.912740    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd073675-8da6-4d49-bf2d-bf6fd1340b51-kube-api-access-dgbck\" (UniqueName: \"kubernetes.io/projected/cd073675-8da6-4d49-bf2d-bf6fd1340b51-kube-api-access-dgbck\") pod \"cd073675-8da6-4d49-bf2d-bf6fd1340b51\" (UID: \"cd073675-8da6-4d49-bf2d-bf6fd1340b51\") "
Oct 31 12:45:15.912970 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:15.912860    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd073675-8da6-4d49-bf2d-bf6fd1340b51-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd073675-8da6-4d49-bf2d-bf6fd1340b51-maas-api-xks-ca\") pod \"cd073675-8da6-4d49-bf2d-bf6fd1340b51\" (UID: \"cd073675-8da6-4d49-bf2d-bf6fd1340b51\") "
Oct 31 12:45:15.913013 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:15.912980    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd073675-8da6-4d49-bf2d-bf6fd1340b51-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd073675-8da6-4d49-bf2d-bf6fd1340b51-maas-api-openshift-ca\") pod \"cd073675-8da6-4d49-bf2d-bf6fd1340b51\" (UID: \"cd073675-8da6-4d49-bf2d-bf6fd1340b51\") "
Oct 31 12:45:15.913396 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:15.913364    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd073675-8da6-4d49-bf2d-bf6fd1340b51-maas-api-openshift-ca" pod "cd073675-8da6-4d49-bf2d-bf6fd1340b51" (UID: "cd073675-8da6-4d49-bf2d-bf6fd1340b51"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:45:15.913908 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:15.913882    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd073675-8da6-4d49-bf2d-bf6fd1340b51-maas-api-openshift-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:45:15.915446 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:15.915419    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd073675-8da6-4d49-bf2d-bf6fd1340b51-kube-api-access-dgbck" pod "cd073675-8da6-4d49-bf2d-bf6fd1340b51" (UID: "cd073675-8da6-4d49-bf2d-bf6fd1340b51"). InnerVolumeSpecName "kube-api-access-dgbck". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:45:15.915585 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:15.915454    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd073675-8da6-4d49-bf2d-bf6fd1340b51-maas-api-xks-ca" pod "cd073675-8da6-4d49-bf2d-bf6fd1340b51" (UID: "cd073675-8da6-4d49-bf2d-bf6fd1340b51"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:45:16.015339 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:16.015302    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd073675-8da6-4d49-bf2d-bf6fd1340b51-maas-api-xks-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:45:16.015339 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:16.015334    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgbck\" (UniqueName: \"kubernetes.io/projected/cd073675-8da6-4d49-bf2d-bf6fd1340b51-kube-api-access-dgbck\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:45:16.041081 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:16.041051    5060 scope.go:122] "RemoveContainer" containerID="6dcf2ea6b61540a89880d45c677894e4986b913bb7dd0e6e64bfffe0a9215600"
Oct 31 12:45:16.622868 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:16.622795    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b8ceeeb-9sbgk"
Oct 31 12:45:16.623040 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:16.622968    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b8ceeeb-9sbgk" event={"ID":"cd073675-8da6-4d49-bf2d-bf6fd1340b51","Type":"ContainerDied","Data":"1cccc6b59baba09bb0f0dba8314d0bf2adc01e74c548d54c97f2feecdf674b7e"}
Oct 31 12:45:16.623040 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:16.623021    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1cccc6b59baba09bb0f0dba8314d0bf2adc01e74c548d54c97f2feecdf674b7e"
Oct 31 12:45:16.625470 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:16.625443    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-2d6201-29890845-tm2bd" event={"ID":"dd0d4341-08cb-40e3-9062-c24256031504","Type":"ContainerStarted","Data":"01044ebd2e0473c015313fded108a171f0044a682c14dc566f9c09550ee5ce9d"}
Oct 31 12:45:17.047211 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:17.047176    5060 scope.go:122] "RemoveContainer" containerID="3ab544abb92df8ed94885c493ea81134ae67c9b04bb9712e34d1f006e4c6031c"
Oct 31 12:45:17.047678 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:17.047542    5060 scope.go:122] "RemoveContainer" containerID="2a1cf6ca4dd02f62f5689c3dbe9da74ddc83c40744557cdd05a3201ad5f357e5"
Oct 31 12:45:17.642816 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:17.642719    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fb0343-29890845-cxhdt" event={"ID":"de4cc1ae-6e6f-4124-a0cd-464549508f9e","Type":"ContainerStarted","Data":"c3aca860574b24886183f22f55169ab7535bee88e409de4b6e60c000690f3b4e"}
Oct 31 12:45:17.645802 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:17.645725    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890845-92l2w" event={"ID":"d87b9890-61ac-4018-a312-c02787e3c371","Type":"ContainerStarted","Data":"aed67f27709746d41d0c7c188c46ab8840174cb16d90c6394b7cf6a778dc5fcc"}
Oct 31 12:45:17.648650 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:17.648619    5060 generic.go:356] "Generic (PLEG): container finished" podID="dd0d4341-08cb-40e3-9062-c24256031504" containerID="01044ebd2e0473c015313fded108a171f0044a682c14dc566f9c09550ee5ce9d" exitCode=60
Oct 31 12:45:17.648771 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:17.648672    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-2d6201-29890845-tm2bd" event={"ID":"dd0d4341-08cb-40e3-9062-c24256031504","Type":"ContainerDied","Data":"01044ebd2e0473c015313fded108a171f0044a682c14dc566f9c09550ee5ce9d"}
Oct 31 12:45:17.648771 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:17.648745    5060 scope.go:122] "RemoveContainer" containerID="6dcf2ea6b61540a89880d45c677894e4986b913bb7dd0e6e64bfffe0a9215600"
Oct 31 12:45:17.649160 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:17.649138    5060 scope.go:122] "RemoveContainer" containerID="01044ebd2e0473c015313fded108a171f0044a682c14dc566f9c09550ee5ce9d"
Oct 31 12:45:17.651389 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:45:17.649538    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-2d6201-29890845-tm2bd_odh-ai-gateway-infra(dd0d4341-08cb-40e3-9062-c24256031504)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-2d6201-29890845-tm2bd" podUID="dd0d4341-08cb-40e3-9062-c24256031504"
Oct 31 12:45:17.672520 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:17.672095    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-2d6201-29890845-tm2bd"]
Oct 31 12:45:18.041099 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:18.041063    5060 scope.go:122] "RemoveContainer" containerID="ac287b2eaf2128ed22fa26a0ae28f59a166006b12fd77e383aa957ce3339e30a"
Oct 31 12:45:18.680473 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:18.680395    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-67d410-29890845-6m5qx" event={"ID":"f1b5757d-78b6-4614-b359-5398628f7913","Type":"ContainerStarted","Data":"829c9122fa7c9e8a9f895f4ebbd546277a7e0efcf9ac6c6728bbf49a729bb179"}
Oct 31 12:45:18.688226 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:18.687753    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fb0343-29890845-cxhdt"]
Oct 31 12:45:18.689136 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:18.689099    5060 generic.go:356] "Generic (PLEG): container finished" podID="de4cc1ae-6e6f-4124-a0cd-464549508f9e" containerID="c3aca860574b24886183f22f55169ab7535bee88e409de4b6e60c000690f3b4e" exitCode=60
Oct 31 12:45:18.689532 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:18.689503    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fb0343-29890845-cxhdt" event={"ID":"de4cc1ae-6e6f-4124-a0cd-464549508f9e","Type":"ContainerDied","Data":"c3aca860574b24886183f22f55169ab7535bee88e409de4b6e60c000690f3b4e"}
Oct 31 12:45:18.689663 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:18.689588    5060 scope.go:122] "RemoveContainer" containerID="2a1cf6ca4dd02f62f5689c3dbe9da74ddc83c40744557cdd05a3201ad5f357e5"
Oct 31 12:45:18.695685 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:18.692706    5060 generic.go:356] "Generic (PLEG): container finished" podID="d87b9890-61ac-4018-a312-c02787e3c371" containerID="aed67f27709746d41d0c7c188c46ab8840174cb16d90c6394b7cf6a778dc5fcc" exitCode=60
Oct 31 12:45:18.695685 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:18.692747    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890845-92l2w" event={"ID":"d87b9890-61ac-4018-a312-c02787e3c371","Type":"ContainerDied","Data":"aed67f27709746d41d0c7c188c46ab8840174cb16d90c6394b7cf6a778dc5fcc"}
Oct 31 12:45:18.695685 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:18.693295    5060 scope.go:122] "RemoveContainer" containerID="aed67f27709746d41d0c7c188c46ab8840174cb16d90c6394b7cf6a778dc5fcc"
Oct 31 12:45:18.695685 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:45:18.693763    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29890845-92l2w_odh-ai-gateway-infra(d87b9890-61ac-4018-a312-c02787e3c371)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890845-92l2w" podUID="d87b9890-61ac-4018-a312-c02787e3c371"
Oct 31 12:45:18.725801 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:18.725777    5060 scope.go:122] "RemoveContainer" containerID="3ab544abb92df8ed94885c493ea81134ae67c9b04bb9712e34d1f006e4c6031c"
Oct 31 12:45:18.728349 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:18.728325    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890845-92l2w"]
Oct 31 12:45:19.072878 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:19.072848    5060 scope.go:122] "RemoveContainer" containerID="805b2810854b16c730d546248fc36bc3d28a3c3ba63214343f7abf818910d9af"
Oct 31 12:45:19.080106 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:19.080083    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fb0343-29890845-cxhdt"
Oct 31 12:45:19.103071 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:19.103045    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-2d6201-29890845-tm2bd"
Oct 31 12:45:19.177287 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:19.177202    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de4cc1ae-6e6f-4124-a0cd-464549508f9e-kube-api-access-ghnqm\" (UniqueName: \"kubernetes.io/projected/de4cc1ae-6e6f-4124-a0cd-464549508f9e-kube-api-access-ghnqm\") pod \"de4cc1ae-6e6f-4124-a0cd-464549508f9e\" (UID: \"de4cc1ae-6e6f-4124-a0cd-464549508f9e\") "
Oct 31 12:45:19.177465 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:19.177287    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd0d4341-08cb-40e3-9062-c24256031504-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd0d4341-08cb-40e3-9062-c24256031504-maas-api-service-ca\") pod \"dd0d4341-08cb-40e3-9062-c24256031504\" (UID: \"dd0d4341-08cb-40e3-9062-c24256031504\") "
Oct 31 12:45:19.177465 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:19.177379    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de4cc1ae-6e6f-4124-a0cd-464549508f9e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/de4cc1ae-6e6f-4124-a0cd-464549508f9e-maas-api-service-ca\") pod \"de4cc1ae-6e6f-4124-a0cd-464549508f9e\" (UID: \"de4cc1ae-6e6f-4124-a0cd-464549508f9e\") "
Oct 31 12:45:19.177946 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:19.177903    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd0d4341-08cb-40e3-9062-c24256031504-maas-api-service-ca" pod "dd0d4341-08cb-40e3-9062-c24256031504" (UID: "dd0d4341-08cb-40e3-9062-c24256031504"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:45:19.178422 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:19.178389    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de4cc1ae-6e6f-4124-a0cd-464549508f9e-maas-api-service-ca" pod "de4cc1ae-6e6f-4124-a0cd-464549508f9e" (UID: "de4cc1ae-6e6f-4124-a0cd-464549508f9e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:45:19.178721 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:19.178672    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd0d4341-08cb-40e3-9062-c24256031504-kube-api-access-x577d\" (UniqueName: \"kubernetes.io/projected/dd0d4341-08cb-40e3-9062-c24256031504-kube-api-access-x577d\") pod \"dd0d4341-08cb-40e3-9062-c24256031504\" (UID: \"dd0d4341-08cb-40e3-9062-c24256031504\") "
Oct 31 12:45:19.179990 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:19.179540    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd0d4341-08cb-40e3-9062-c24256031504-maas-api-service-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:45:19.179990 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:19.179568    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/de4cc1ae-6e6f-4124-a0cd-464549508f9e-maas-api-service-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:45:19.257343 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:19.257310    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd0d4341-08cb-40e3-9062-c24256031504-kube-api-access-x577d" pod "dd0d4341-08cb-40e3-9062-c24256031504" (UID: "dd0d4341-08cb-40e3-9062-c24256031504"). InnerVolumeSpecName "kube-api-access-x577d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:45:19.258476 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:19.258435    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de4cc1ae-6e6f-4124-a0cd-464549508f9e-kube-api-access-ghnqm" pod "de4cc1ae-6e6f-4124-a0cd-464549508f9e" (UID: "de4cc1ae-6e6f-4124-a0cd-464549508f9e"). InnerVolumeSpecName "kube-api-access-ghnqm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:45:19.281906 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:19.281870    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x577d\" (UniqueName: \"kubernetes.io/projected/dd0d4341-08cb-40e3-9062-c24256031504-kube-api-access-x577d\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:45:19.282030 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:19.281912    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ghnqm\" (UniqueName: \"kubernetes.io/projected/de4cc1ae-6e6f-4124-a0cd-464549508f9e-kube-api-access-ghnqm\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:45:19.305332 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:19.305300    5060 scope.go:122] "RemoveContainer" containerID="af85ee4d4436585ba9405c88b237ab12bf0900688f701c39b74fa1c1f57e3ae5"
Oct 31 12:45:19.336757 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:19.336729    5060 scope.go:122] "RemoveContainer" containerID="17ac645399249d4e7e7283ce79c8b50c48e6c3dca2982f14aaf643c8a90030e4"
Oct 31 12:45:19.722086 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:19.722049    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fb0343-29890845-cxhdt"
Oct 31 12:45:19.725613 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:19.725565    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fb0343-29890845-cxhdt" event={"ID":"de4cc1ae-6e6f-4124-a0cd-464549508f9e","Type":"ContainerDied","Data":"9c1918e697137e873f372e51bd8dfb89191acc77747f3980028d94a30e053544"}
Oct 31 12:45:19.725794 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:19.725658    5060 scope.go:122] "RemoveContainer" containerID="c3aca860574b24886183f22f55169ab7535bee88e409de4b6e60c000690f3b4e"
Oct 31 12:45:19.762613 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:19.757517    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-62734c-29890845-tbvmk" event={"ID":"5361e3ad-5107-41cc-8afa-f0c64e8236b1","Type":"ContainerStarted","Data":"e9529c6a4b74acbe67713641944452028089afbeee7d7871e2ee8912317d3e19"}
Oct 31 12:45:19.782963 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:19.777440    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-2d6201-29890845-tm2bd" event={"ID":"dd0d4341-08cb-40e3-9062-c24256031504","Type":"ContainerDied","Data":"200620095c53c5e095fd0a54c496b9cb86e860be744ec6d875f4b28ebf6f784a"}
Oct 31 12:45:19.782963 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:19.777567    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-2d6201-29890845-tm2bd"
Oct 31 12:45:19.793112 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:19.793083    5060 generic.go:356] "Generic (PLEG): container finished" podID="f1b5757d-78b6-4614-b359-5398628f7913" containerID="829c9122fa7c9e8a9f895f4ebbd546277a7e0efcf9ac6c6728bbf49a729bb179" exitCode=60
Oct 31 12:45:19.793259 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:19.793211    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-67d410-29890845-6m5qx" event={"ID":"f1b5757d-78b6-4614-b359-5398628f7913","Type":"ContainerDied","Data":"829c9122fa7c9e8a9f895f4ebbd546277a7e0efcf9ac6c6728bbf49a729bb179"}
Oct 31 12:45:19.793754 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:19.793709    5060 scope.go:122] "RemoveContainer" containerID="829c9122fa7c9e8a9f895f4ebbd546277a7e0efcf9ac6c6728bbf49a729bb179"
Oct 31 12:45:19.794094 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:45:19.794072    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-67d410-29890845-6m5qx_odh-ai-gateway-infra(f1b5757d-78b6-4614-b359-5398628f7913)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-67d410-29890845-6m5qx" podUID="f1b5757d-78b6-4614-b359-5398628f7913"
Oct 31 12:45:19.798280 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:19.798258    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-67d410-29890845-6m5qx"]
Oct 31 12:45:19.817737 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:19.817662    5060 scope.go:122] "RemoveContainer" containerID="01044ebd2e0473c015313fded108a171f0044a682c14dc566f9c09550ee5ce9d"
Oct 31 12:45:19.831195 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:19.831170    5060 scope.go:122] "RemoveContainer" containerID="ac287b2eaf2128ed22fa26a0ae28f59a166006b12fd77e383aa957ce3339e30a"
Oct 31 12:45:19.858560 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:19.858505    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fb0343-29890845-cxhdt"]
Oct 31 12:45:19.875282 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:19.874870    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fb0343-29890845-cxhdt"]
Oct 31 12:45:19.885705 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:19.885100    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-2d6201-29890845-tm2bd"]
Oct 31 12:45:19.897321 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:19.897290    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-2d6201-29890845-tm2bd"]
Oct 31 12:45:20.034543 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:20.034514    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890845-92l2w"
Oct 31 12:45:20.040544 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:20.040486    5060 scope.go:122] "RemoveContainer" containerID="07d3a57bd3fe730ec0d7a60dbe432c87f73c6466dd95ec6ab817d6f875b7264b"
Oct 31 12:45:20.040703 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:20.040608    5060 scope.go:122] "RemoveContainer" containerID="5a9d979681590812140679756bf0833174fceb811e9e730755c1e11275e1335e"
Oct 31 12:45:20.094977 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:20.094648    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d87b9890-61ac-4018-a312-c02787e3c371-kube-api-access-cq7ng\" (UniqueName: \"kubernetes.io/projected/d87b9890-61ac-4018-a312-c02787e3c371-kube-api-access-cq7ng\") pod \"d87b9890-61ac-4018-a312-c02787e3c371\" (UID: \"d87b9890-61ac-4018-a312-c02787e3c371\") "
Oct 31 12:45:20.095082 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:20.094974    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d87b9890-61ac-4018-a312-c02787e3c371-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d87b9890-61ac-4018-a312-c02787e3c371-maas-api-service-ca\") pod \"d87b9890-61ac-4018-a312-c02787e3c371\" (UID: \"d87b9890-61ac-4018-a312-c02787e3c371\") "
Oct 31 12:45:20.096991 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:20.096958    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d87b9890-61ac-4018-a312-c02787e3c371-maas-api-service-ca" pod "d87b9890-61ac-4018-a312-c02787e3c371" (UID: "d87b9890-61ac-4018-a312-c02787e3c371"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:45:20.097318 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:20.097297    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d87b9890-61ac-4018-a312-c02787e3c371-maas-api-service-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:45:20.102754 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:20.102714    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d87b9890-61ac-4018-a312-c02787e3c371-kube-api-access-cq7ng" pod "d87b9890-61ac-4018-a312-c02787e3c371" (UID: "d87b9890-61ac-4018-a312-c02787e3c371"). InnerVolumeSpecName "kube-api-access-cq7ng". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:45:20.198371 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:20.198338    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cq7ng\" (UniqueName: \"kubernetes.io/projected/d87b9890-61ac-4018-a312-c02787e3c371-kube-api-access-cq7ng\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:45:20.799594 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:20.799556    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-62734c-29890845-tbvmk"]
Oct 31 12:45:20.811634 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:20.811601    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890845-92l2w" event={"ID":"d87b9890-61ac-4018-a312-c02787e3c371","Type":"ContainerDied","Data":"1981eb813b2628a6cb36dd904b88c52d130f458f5848df57dfc7494f7ba6a509"}
Oct 31 12:45:20.811827 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:20.811638    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890845-92l2w"
Oct 31 12:45:20.811827 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:20.811670    5060 scope.go:122] "RemoveContainer" containerID="aed67f27709746d41d0c7c188c46ab8840174cb16d90c6394b7cf6a778dc5fcc"
Oct 31 12:45:20.820251 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:20.820221    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e6c79d-29890845-9lxh7" event={"ID":"09c33526-ddf7-41df-8b1a-bf098b046b84","Type":"ContainerStarted","Data":"168f2fbd92a17c7267826a8dac6b9807e0159cec87a3b6e7d1ec858fd4105a17"}
Oct 31 12:45:20.822998 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:20.822972    5060 generic.go:356] "Generic (PLEG): container finished" podID="5361e3ad-5107-41cc-8afa-f0c64e8236b1" containerID="e9529c6a4b74acbe67713641944452028089afbeee7d7871e2ee8912317d3e19" exitCode=60
Oct 31 12:45:20.823144 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:20.823062    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-62734c-29890845-tbvmk" event={"ID":"5361e3ad-5107-41cc-8afa-f0c64e8236b1","Type":"ContainerDied","Data":"e9529c6a4b74acbe67713641944452028089afbeee7d7871e2ee8912317d3e19"}
Oct 31 12:45:20.831403 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:20.831378    5060 scope.go:122] "RemoveContainer" containerID="805b2810854b16c730d546248fc36bc3d28a3c3ba63214343f7abf818910d9af"
Oct 31 12:45:20.898748 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:20.898713    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890845-92l2w"]
Oct 31 12:45:20.918605 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:20.909990    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890845-92l2w"]
Oct 31 12:45:21.058703 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:21.056482    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d87b9890-61ac-4018-a312-c02787e3c371" path="/var/lib/kubelet/pods/d87b9890-61ac-4018-a312-c02787e3c371/volumes"
Oct 31 12:45:21.058703 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:21.057343    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd0d4341-08cb-40e3-9062-c24256031504" path="/var/lib/kubelet/pods/dd0d4341-08cb-40e3-9062-c24256031504/volumes"
Oct 31 12:45:21.062124 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:21.062097    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de4cc1ae-6e6f-4124-a0cd-464549508f9e" path="/var/lib/kubelet/pods/de4cc1ae-6e6f-4124-a0cd-464549508f9e/volumes"
Oct 31 12:45:21.075543 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:21.073114    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-67d410-29890845-6m5qx"
Oct 31 12:45:21.102616 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:21.102581    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-62734c-29890845-tbvmk"
Oct 31 12:45:21.123288 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:21.123253    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1b5757d-78b6-4614-b359-5398628f7913-kube-api-access-8x8z9\" (UniqueName: \"kubernetes.io/projected/f1b5757d-78b6-4614-b359-5398628f7913-kube-api-access-8x8z9\") pod \"f1b5757d-78b6-4614-b359-5398628f7913\" (UID: \"f1b5757d-78b6-4614-b359-5398628f7913\") "
Oct 31 12:45:21.123460 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:21.123341    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1b5757d-78b6-4614-b359-5398628f7913-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1b5757d-78b6-4614-b359-5398628f7913-maas-api-service-ca\") pod \"f1b5757d-78b6-4614-b359-5398628f7913\" (UID: \"f1b5757d-78b6-4614-b359-5398628f7913\") "
Oct 31 12:45:21.123653 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:21.123614    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5361e3ad-5107-41cc-8afa-f0c64e8236b1-kube-api-access-ph6qp\" (UniqueName: \"kubernetes.io/projected/5361e3ad-5107-41cc-8afa-f0c64e8236b1-kube-api-access-ph6qp\") pod \"5361e3ad-5107-41cc-8afa-f0c64e8236b1\" (UID: \"5361e3ad-5107-41cc-8afa-f0c64e8236b1\") "
Oct 31 12:45:21.123761 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:21.123743    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5361e3ad-5107-41cc-8afa-f0c64e8236b1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5361e3ad-5107-41cc-8afa-f0c64e8236b1-maas-api-service-ca\") pod \"5361e3ad-5107-41cc-8afa-f0c64e8236b1\" (UID: \"5361e3ad-5107-41cc-8afa-f0c64e8236b1\") "
Oct 31 12:45:21.126193 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:21.126151    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5361e3ad-5107-41cc-8afa-f0c64e8236b1-maas-api-service-ca" pod "5361e3ad-5107-41cc-8afa-f0c64e8236b1" (UID: "5361e3ad-5107-41cc-8afa-f0c64e8236b1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:45:21.126796 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:21.126769    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1b5757d-78b6-4614-b359-5398628f7913-maas-api-service-ca" pod "f1b5757d-78b6-4614-b359-5398628f7913" (UID: "f1b5757d-78b6-4614-b359-5398628f7913"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:45:21.158065 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:21.158031    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5361e3ad-5107-41cc-8afa-f0c64e8236b1-kube-api-access-ph6qp" pod "5361e3ad-5107-41cc-8afa-f0c64e8236b1" (UID: "5361e3ad-5107-41cc-8afa-f0c64e8236b1"). InnerVolumeSpecName "kube-api-access-ph6qp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:45:21.164462 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:21.164429    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1b5757d-78b6-4614-b359-5398628f7913-kube-api-access-8x8z9" pod "f1b5757d-78b6-4614-b359-5398628f7913" (UID: "f1b5757d-78b6-4614-b359-5398628f7913"). InnerVolumeSpecName "kube-api-access-8x8z9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:45:21.227650 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:21.227614    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ph6qp\" (UniqueName: \"kubernetes.io/projected/5361e3ad-5107-41cc-8afa-f0c64e8236b1-kube-api-access-ph6qp\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:45:21.227650 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:21.227651    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5361e3ad-5107-41cc-8afa-f0c64e8236b1-maas-api-service-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:45:21.227825 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:21.227668    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8x8z9\" (UniqueName: \"kubernetes.io/projected/f1b5757d-78b6-4614-b359-5398628f7913-kube-api-access-8x8z9\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:45:21.227825 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:21.227683    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1b5757d-78b6-4614-b359-5398628f7913-maas-api-service-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:45:21.837098 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:21.837056    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-67d410-29890845-6m5qx"
Oct 31 12:45:21.837098 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:21.837080    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-67d410-29890845-6m5qx" event={"ID":"f1b5757d-78b6-4614-b359-5398628f7913","Type":"ContainerDied","Data":"8f7a93440ff80d46e51702bf118ea949090067121b359f98e6548c97ff55c48b"}
Oct 31 12:45:21.837660 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:21.837154    5060 scope.go:122] "RemoveContainer" containerID="829c9122fa7c9e8a9f895f4ebbd546277a7e0efcf9ac6c6728bbf49a729bb179"
Oct 31 12:45:21.839639 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:21.839609    5060 generic.go:356] "Generic (PLEG): container finished" podID="352baef6-da71-44a4-b1cc-f0960db22901" containerID="a160574fa49990389502e709b0df73d331ae4366aeb71619209de3837eae6890" exitCode=60
Oct 31 12:45:21.839639 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:21.839628    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-ed25a690-29890845-pvxsp" event={"ID":"352baef6-da71-44a4-b1cc-f0960db22901","Type":"ContainerDied","Data":"a160574fa49990389502e709b0df73d331ae4366aeb71619209de3837eae6890"}
Oct 31 12:45:21.840130 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:21.840101    5060 scope.go:122] "RemoveContainer" containerID="a160574fa49990389502e709b0df73d331ae4366aeb71619209de3837eae6890"
Oct 31 12:45:21.840459 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:45:21.840436    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-ed25a690-29890845-pvxsp_odh-ai-gateway-infra(352baef6-da71-44a4-b1cc-f0960db22901)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-ed25a690-29890845-pvxsp" podUID="352baef6-da71-44a4-b1cc-f0960db22901"
Oct 31 12:45:21.842360 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:21.842336    5060 generic.go:356] "Generic (PLEG): container finished" podID="09c33526-ddf7-41df-8b1a-bf098b046b84" containerID="168f2fbd92a17c7267826a8dac6b9807e0159cec87a3b6e7d1ec858fd4105a17" exitCode=60
Oct 31 12:45:21.842541 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:21.842504    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e6c79d-29890845-9lxh7" event={"ID":"09c33526-ddf7-41df-8b1a-bf098b046b84","Type":"ContainerDied","Data":"168f2fbd92a17c7267826a8dac6b9807e0159cec87a3b6e7d1ec858fd4105a17"}
Oct 31 12:45:21.842973 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:21.842946    5060 scope.go:122] "RemoveContainer" containerID="168f2fbd92a17c7267826a8dac6b9807e0159cec87a3b6e7d1ec858fd4105a17"
Oct 31 12:45:21.843353 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:45:21.843330    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-e6c79d-29890845-9lxh7_odh-ai-gateway-infra(09c33526-ddf7-41df-8b1a-bf098b046b84)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e6c79d-29890845-9lxh7" podUID="09c33526-ddf7-41df-8b1a-bf098b046b84"
Oct 31 12:45:21.846065 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:21.846035    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-62734c-29890845-tbvmk"
Oct 31 12:45:21.846169 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:21.846074    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-62734c-29890845-tbvmk" event={"ID":"5361e3ad-5107-41cc-8afa-f0c64e8236b1","Type":"ContainerDied","Data":"67b2aad281fc70873a3fc6a3e21b4024f4d4075eef105abbb7ca9f5a3a433429"}
Oct 31 12:45:21.853883 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:21.853860    5060 scope.go:122] "RemoveContainer" containerID="5a9d979681590812140679756bf0833174fceb811e9e730755c1e11275e1335e"
Oct 31 12:45:21.877268 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:21.877248    5060 scope.go:122] "RemoveContainer" containerID="07d3a57bd3fe730ec0d7a60dbe432c87f73c6466dd95ec6ab817d6f875b7264b"
Oct 31 12:45:21.881423 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:21.881396    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e6c79d-29890845-9lxh7"]
Oct 31 12:45:21.902483 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:21.902444    5060 scope.go:122] "RemoveContainer" containerID="e9529c6a4b74acbe67713641944452028089afbeee7d7871e2ee8912317d3e19"
Oct 31 12:45:21.928345 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:21.928312    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-67d410-29890845-6m5qx"]
Oct 31 12:45:21.936661 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:21.936629    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-67d410-29890845-6m5qx"]
Oct 31 12:45:21.945541 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:21.945460    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-62734c-29890845-tbvmk"]
Oct 31 12:45:21.952016 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:21.951986    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-62734c-29890845-tbvmk"]
Oct 31 12:45:22.029974 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:22.029936    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-466d864d-94qhv"]
Oct 31 12:45:22.037275 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:22.037248    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-466d864d-94qhv"]
Oct 31 12:45:22.877451 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:22.877414    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-ed25a690-29890845-pvxsp"]
Oct 31 12:45:23.092120 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:23.080658    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5361e3ad-5107-41cc-8afa-f0c64e8236b1" path="/var/lib/kubelet/pods/5361e3ad-5107-41cc-8afa-f0c64e8236b1/volumes"
Oct 31 12:45:23.092120 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:23.081379    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b50eace0-22ba-490b-9344-0722c9b6cfa0" path="/var/lib/kubelet/pods/b50eace0-22ba-490b-9344-0722c9b6cfa0/volumes"
Oct 31 12:45:23.092120 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:23.081965    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1b5757d-78b6-4614-b359-5398628f7913" path="/var/lib/kubelet/pods/f1b5757d-78b6-4614-b359-5398628f7913/volumes"
Oct 31 12:45:23.299072 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:23.299042    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e6c79d-29890845-9lxh7"
Oct 31 12:45:23.305013 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:23.304986    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-ed25a690-29890845-pvxsp"
Oct 31 12:45:23.434208 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:23.434112    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/352baef6-da71-44a4-b1cc-f0960db22901-kube-api-access-vkf85\" (UniqueName: \"kubernetes.io/projected/352baef6-da71-44a4-b1cc-f0960db22901-kube-api-access-vkf85\") pod \"352baef6-da71-44a4-b1cc-f0960db22901\" (UID: \"352baef6-da71-44a4-b1cc-f0960db22901\") "
Oct 31 12:45:23.434208 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:23.434189    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09c33526-ddf7-41df-8b1a-bf098b046b84-kube-api-access-v2t5g\" (UniqueName: \"kubernetes.io/projected/09c33526-ddf7-41df-8b1a-bf098b046b84-kube-api-access-v2t5g\") pod \"09c33526-ddf7-41df-8b1a-bf098b046b84\" (UID: \"09c33526-ddf7-41df-8b1a-bf098b046b84\") "
Oct 31 12:45:23.434443 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:23.434290    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/352baef6-da71-44a4-b1cc-f0960db22901-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/352baef6-da71-44a4-b1cc-f0960db22901-maas-api-service-ca\") pod \"352baef6-da71-44a4-b1cc-f0960db22901\" (UID: \"352baef6-da71-44a4-b1cc-f0960db22901\") "
Oct 31 12:45:23.434443 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:23.434353    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09c33526-ddf7-41df-8b1a-bf098b046b84-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09c33526-ddf7-41df-8b1a-bf098b046b84-maas-api-service-ca\") pod \"09c33526-ddf7-41df-8b1a-bf098b046b84\" (UID: \"09c33526-ddf7-41df-8b1a-bf098b046b84\") "
Oct 31 12:45:23.434761 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:23.434711    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/352baef6-da71-44a4-b1cc-f0960db22901-maas-api-service-ca" pod "352baef6-da71-44a4-b1cc-f0960db22901" (UID: "352baef6-da71-44a4-b1cc-f0960db22901"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:45:23.434761 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:23.434737    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09c33526-ddf7-41df-8b1a-bf098b046b84-maas-api-service-ca" pod "09c33526-ddf7-41df-8b1a-bf098b046b84" (UID: "09c33526-ddf7-41df-8b1a-bf098b046b84"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:45:23.435265 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:23.435244    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/352baef6-da71-44a4-b1cc-f0960db22901-maas-api-service-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:45:23.435349 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:23.435267    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09c33526-ddf7-41df-8b1a-bf098b046b84-maas-api-service-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:45:23.436591 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:23.436563    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/352baef6-da71-44a4-b1cc-f0960db22901-kube-api-access-vkf85" pod "352baef6-da71-44a4-b1cc-f0960db22901" (UID: "352baef6-da71-44a4-b1cc-f0960db22901"). InnerVolumeSpecName "kube-api-access-vkf85". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:45:23.437346 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:23.437326    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09c33526-ddf7-41df-8b1a-bf098b046b84-kube-api-access-v2t5g" pod "09c33526-ddf7-41df-8b1a-bf098b046b84" (UID: "09c33526-ddf7-41df-8b1a-bf098b046b84"). InnerVolumeSpecName "kube-api-access-v2t5g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:45:23.537298 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:23.537264    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vkf85\" (UniqueName: \"kubernetes.io/projected/352baef6-da71-44a4-b1cc-f0960db22901-kube-api-access-vkf85\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:45:23.537298 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:23.537297    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2t5g\" (UniqueName: \"kubernetes.io/projected/09c33526-ddf7-41df-8b1a-bf098b046b84-kube-api-access-v2t5g\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:45:23.885686 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:23.885655    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-ed25a690-29890845-pvxsp"
Oct 31 12:45:23.886123 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:23.885701    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-ed25a690-29890845-pvxsp" event={"ID":"352baef6-da71-44a4-b1cc-f0960db22901","Type":"ContainerDied","Data":"38a21b8a4f2a23460d5e80338e9d0cbe0bb2958b79db0f2d2b8e327e5898f869"}
Oct 31 12:45:23.886123 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:23.885764    5060 scope.go:122] "RemoveContainer" containerID="a160574fa49990389502e709b0df73d331ae4366aeb71619209de3837eae6890"
Oct 31 12:45:23.887702 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:23.887680    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e6c79d-29890845-9lxh7"
Oct 31 12:45:23.887842 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:23.887725    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e6c79d-29890845-9lxh7" event={"ID":"09c33526-ddf7-41df-8b1a-bf098b046b84","Type":"ContainerDied","Data":"3baf4733f3a6679d07b12678e9b9dc51a5756fef7e17110a79a82c7b4332c27c"}
Oct 31 12:45:23.901656 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:23.901630    5060 scope.go:122] "RemoveContainer" containerID="168f2fbd92a17c7267826a8dac6b9807e0159cec87a3b6e7d1ec858fd4105a17"
Oct 31 12:45:23.931170 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:23.931141    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e6c79d-29890845-9lxh7"]
Oct 31 12:45:23.937837 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:23.937810    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e6c79d-29890845-9lxh7"]
Oct 31 12:45:23.947774 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:23.947742    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-ed25a690-29890845-pvxsp"]
Oct 31 12:45:23.956879 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:23.956844    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-ed25a690-29890845-pvxsp"]
Oct 31 12:45:25.100516 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:25.100203    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09c33526-ddf7-41df-8b1a-bf098b046b84" path="/var/lib/kubelet/pods/09c33526-ddf7-41df-8b1a-bf098b046b84/volumes"
Oct 31 12:45:25.100983 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:25.100819    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="352baef6-da71-44a4-b1cc-f0960db22901" path="/var/lib/kubelet/pods/352baef6-da71-44a4-b1cc-f0960db22901/volumes"
Oct 31 12:45:25.933952 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:25.933918    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ed25a690-z8cwl"]
Oct 31 12:45:25.934863 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:25.934840    5060 cpu_manager.go:414] "containerMap: removing container" podUID="de4cc1ae-6e6f-4124-a0cd-464549508f9e" containerName="cleanup"
Oct 31 12:45:25.934959 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:25.934863    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de4cc1ae-6e6f-4124-a0cd-464549508f9e" containerName="cleanup"
Oct 31 12:45:25.934959 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:25.934873    5060 cpu_manager.go:414] "containerMap: removing container" podUID="5361e3ad-5107-41cc-8afa-f0c64e8236b1" containerName="cleanup"
Oct 31 12:45:25.934959 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:25.934879    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5361e3ad-5107-41cc-8afa-f0c64e8236b1" containerName="cleanup"
Oct 31 12:45:25.934959 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:25.934888    5060 cpu_manager.go:414] "containerMap: removing container" podUID="d87b9890-61ac-4018-a312-c02787e3c371" containerName="cleanup"
Oct 31 12:45:25.934959 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:25.934894    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d87b9890-61ac-4018-a312-c02787e3c371" containerName="cleanup"
Oct 31 12:45:25.934959 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:25.934919    5060 cpu_manager.go:414] "containerMap: removing container" podUID="de4cc1ae-6e6f-4124-a0cd-464549508f9e" containerName="cleanup"
Oct 31 12:45:25.934959 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:25.934926    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de4cc1ae-6e6f-4124-a0cd-464549508f9e" containerName="cleanup"
Oct 31 12:45:25.934959 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:25.934939    5060 cpu_manager.go:414] "containerMap: removing container" podUID="09c33526-ddf7-41df-8b1a-bf098b046b84" containerName="cleanup"
Oct 31 12:45:25.934959 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:25.934943    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09c33526-ddf7-41df-8b1a-bf098b046b84" containerName="cleanup"
Oct 31 12:45:25.934959 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:25.934953    5060 cpu_manager.go:414] "containerMap: removing container" podUID="dd0d4341-08cb-40e3-9062-c24256031504" containerName="cleanup"
Oct 31 12:45:25.934959 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:25.934958    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd0d4341-08cb-40e3-9062-c24256031504" containerName="cleanup"
Oct 31 12:45:25.935507 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:25.934968    5060 cpu_manager.go:414] "containerMap: removing container" podUID="352baef6-da71-44a4-b1cc-f0960db22901" containerName="cleanup"
Oct 31 12:45:25.935507 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:25.934972    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="352baef6-da71-44a4-b1cc-f0960db22901" containerName="cleanup"
Oct 31 12:45:25.935507 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:25.934978    5060 cpu_manager.go:414] "containerMap: removing container" podUID="f1b5757d-78b6-4614-b359-5398628f7913" containerName="cleanup"
Oct 31 12:45:25.935507 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:25.934982    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1b5757d-78b6-4614-b359-5398628f7913" containerName="cleanup"
Oct 31 12:45:25.935507 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:25.934990    5060 cpu_manager.go:414] "containerMap: removing container" podUID="f1b5757d-78b6-4614-b359-5398628f7913" containerName="cleanup"
Oct 31 12:45:25.935507 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:25.934994    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1b5757d-78b6-4614-b359-5398628f7913" containerName="cleanup"
Oct 31 12:45:25.935507 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:25.935005    5060 cpu_manager.go:414] "containerMap: removing container" podUID="d87b9890-61ac-4018-a312-c02787e3c371" containerName="cleanup"
Oct 31 12:45:25.935507 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:25.935009    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d87b9890-61ac-4018-a312-c02787e3c371" containerName="cleanup"
Oct 31 12:45:25.935507 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:25.935017    5060 cpu_manager.go:414] "containerMap: removing container" podUID="09c33526-ddf7-41df-8b1a-bf098b046b84" containerName="cleanup"
Oct 31 12:45:25.935507 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:25.935022    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09c33526-ddf7-41df-8b1a-bf098b046b84" containerName="cleanup"
Oct 31 12:45:25.935507 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:25.935028    5060 cpu_manager.go:414] "containerMap: removing container" podUID="f1b5757d-78b6-4614-b359-5398628f7913" containerName="cleanup"
Oct 31 12:45:25.935507 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:25.935032    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1b5757d-78b6-4614-b359-5398628f7913" containerName="cleanup"
Oct 31 12:45:25.935507 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:25.935040    5060 cpu_manager.go:414] "containerMap: removing container" podUID="09c33526-ddf7-41df-8b1a-bf098b046b84" containerName="cleanup"
Oct 31 12:45:25.935507 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:25.935046    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09c33526-ddf7-41df-8b1a-bf098b046b84" containerName="cleanup"
Oct 31 12:45:25.935507 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:25.935051    5060 cpu_manager.go:414] "containerMap: removing container" podUID="352baef6-da71-44a4-b1cc-f0960db22901" containerName="cleanup"
Oct 31 12:45:25.935507 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:25.935056    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="352baef6-da71-44a4-b1cc-f0960db22901" containerName="cleanup"
Oct 31 12:45:25.935507 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:25.935065    5060 cpu_manager.go:414] "containerMap: removing container" podUID="5361e3ad-5107-41cc-8afa-f0c64e8236b1" containerName="cleanup"
Oct 31 12:45:25.935507 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:25.935069    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5361e3ad-5107-41cc-8afa-f0c64e8236b1" containerName="cleanup"
Oct 31 12:45:25.935507 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:25.935091    5060 cpu_manager.go:414] "containerMap: removing container" podUID="cd073675-8da6-4d49-bf2d-bf6fd1340b51" containerName="revoke-keys"
Oct 31 12:45:25.935507 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:25.935095    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd073675-8da6-4d49-bf2d-bf6fd1340b51" containerName="revoke-keys"
Oct 31 12:45:25.935507 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:25.935103    5060 cpu_manager.go:414] "containerMap: removing container" podUID="dd0d4341-08cb-40e3-9062-c24256031504" containerName="cleanup"
Oct 31 12:45:25.935507 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:25.935107    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd0d4341-08cb-40e3-9062-c24256031504" containerName="cleanup"
Oct 31 12:45:25.935507 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:25.935221    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="de4cc1ae-6e6f-4124-a0cd-464549508f9e" containerName="cleanup"
Oct 31 12:45:25.935507 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:25.935232    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd073675-8da6-4d49-bf2d-bf6fd1340b51" containerName="revoke-keys"
Oct 31 12:45:25.935507 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:25.935240    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd0d4341-08cb-40e3-9062-c24256031504" containerName="cleanup"
Oct 31 12:45:25.935507 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:25.935246    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="352baef6-da71-44a4-b1cc-f0960db22901" containerName="cleanup"
Oct 31 12:45:25.935507 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:25.935252    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="de4cc1ae-6e6f-4124-a0cd-464549508f9e" containerName="cleanup"
Oct 31 12:45:25.935507 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:25.935257    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="5361e3ad-5107-41cc-8afa-f0c64e8236b1" containerName="cleanup"
Oct 31 12:45:25.935507 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:25.935262    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="d87b9890-61ac-4018-a312-c02787e3c371" containerName="cleanup"
Oct 31 12:45:25.935507 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:25.935270    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="09c33526-ddf7-41df-8b1a-bf098b046b84" containerName="cleanup"
Oct 31 12:45:25.935507 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:25.935276    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd0d4341-08cb-40e3-9062-c24256031504" containerName="cleanup"
Oct 31 12:45:25.935507 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:25.935282    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="352baef6-da71-44a4-b1cc-f0960db22901" containerName="cleanup"
Oct 31 12:45:25.935507 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:25.935287    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1b5757d-78b6-4614-b359-5398628f7913" containerName="cleanup"
Oct 31 12:45:25.935507 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:25.935292    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1b5757d-78b6-4614-b359-5398628f7913" containerName="cleanup"
Oct 31 12:45:25.935507 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:25.935299    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="5361e3ad-5107-41cc-8afa-f0c64e8236b1" containerName="cleanup"
Oct 31 12:45:25.935507 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:25.935305    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="09c33526-ddf7-41df-8b1a-bf098b046b84" containerName="cleanup"
Oct 31 12:45:25.935507 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:25.935312    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="5361e3ad-5107-41cc-8afa-f0c64e8236b1" containerName="cleanup"
Oct 31 12:45:25.935507 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:25.935317    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="d87b9890-61ac-4018-a312-c02787e3c371" containerName="cleanup"
Oct 31 12:45:25.941030 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:25.941007    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ed25a690-z8cwl"
Oct 31 12:45:25.945066 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:25.945037    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 12:45:25.945417 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:25.945391    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 12:45:25.947301 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:25.947279    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ed25a690-z8cwl"]
Oct 31 12:45:25.949660 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:25.949635    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 12:45:25.949803 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:25.949784    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7wsmb\""
Oct 31 12:45:26.027996 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:26.027957    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3e89004-fbpg7"]
Oct 31 12:45:26.035605 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:26.035577    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3e89004-fbpg7"]
Oct 31 12:45:26.090816 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:26.090777    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04c44c3d-e699-4f80-b1b9-2ac15a1544e1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-ed25a690-z8cwl\" (UID: \"04c44c3d-e699-4f80-b1b9-2ac15a1544e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ed25a690-z8cwl"
Oct 31 12:45:26.091046 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:26.091030    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bd4r\" (UniqueName: \"kubernetes.io/projected/04c44c3d-e699-4f80-b1b9-2ac15a1544e1-kube-api-access-2bd4r\") pod \"maas-api-revoke-keys-e2e-mt-ed25a690-z8cwl\" (UID: \"04c44c3d-e699-4f80-b1b9-2ac15a1544e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ed25a690-z8cwl"
Oct 31 12:45:26.091093 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:26.091077    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04c44c3d-e699-4f80-b1b9-2ac15a1544e1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-ed25a690-z8cwl\" (UID: \"04c44c3d-e699-4f80-b1b9-2ac15a1544e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ed25a690-z8cwl"
Oct 31 12:45:26.193015 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:26.192925    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bd4r\" (UniqueName: \"kubernetes.io/projected/04c44c3d-e699-4f80-b1b9-2ac15a1544e1-kube-api-access-2bd4r\") pod \"maas-api-revoke-keys-e2e-mt-ed25a690-z8cwl\" (UID: \"04c44c3d-e699-4f80-b1b9-2ac15a1544e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ed25a690-z8cwl"
Oct 31 12:45:26.193015 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:26.192985    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04c44c3d-e699-4f80-b1b9-2ac15a1544e1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-ed25a690-z8cwl\" (UID: \"04c44c3d-e699-4f80-b1b9-2ac15a1544e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ed25a690-z8cwl"
Oct 31 12:45:26.193592 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:26.193125    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04c44c3d-e699-4f80-b1b9-2ac15a1544e1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-ed25a690-z8cwl\" (UID: \"04c44c3d-e699-4f80-b1b9-2ac15a1544e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ed25a690-z8cwl"
Oct 31 12:45:26.194227 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:26.194201    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04c44c3d-e699-4f80-b1b9-2ac15a1544e1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-ed25a690-z8cwl\" (UID: \"04c44c3d-e699-4f80-b1b9-2ac15a1544e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ed25a690-z8cwl"
Oct 31 12:45:26.195770 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:26.195741    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04c44c3d-e699-4f80-b1b9-2ac15a1544e1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-ed25a690-z8cwl\" (UID: \"04c44c3d-e699-4f80-b1b9-2ac15a1544e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ed25a690-z8cwl"
Oct 31 12:45:26.205844 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:26.205820    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bd4r\" (UniqueName: \"kubernetes.io/projected/04c44c3d-e699-4f80-b1b9-2ac15a1544e1-kube-api-access-2bd4r\") pod \"maas-api-revoke-keys-e2e-mt-ed25a690-z8cwl\" (UID: \"04c44c3d-e699-4f80-b1b9-2ac15a1544e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ed25a690-z8cwl"
Oct 31 12:45:26.265153 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:26.265115    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ed25a690-z8cwl"
Oct 31 12:45:26.969536 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:45:26.967408    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04c44c3d_e699_4f80_b1b9_2ac15a1544e1.slice/crio-6cffdcb7630d22c0002cc50961948285a72010a4dd855cf7cbe5265fbf6ad545 WatchSource:0}: Error finding container 6cffdcb7630d22c0002cc50961948285a72010a4dd855cf7cbe5265fbf6ad545: Status 404 returned error can't find the container with id 6cffdcb7630d22c0002cc50961948285a72010a4dd855cf7cbe5265fbf6ad545
Oct 31 12:45:26.969536 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:26.969285    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ed25a690-z8cwl"]
Oct 31 12:45:27.052635 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:27.052600    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19589aa0-238f-4078-9052-b27af7131832" path="/var/lib/kubelet/pods/19589aa0-238f-4078-9052-b27af7131832/volumes"
Oct 31 12:45:28.002041 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:28.001587    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ed25a690-z8cwl" event={"ID":"04c44c3d-e699-4f80-b1b9-2ac15a1544e1","Type":"ContainerStarted","Data":"20a88c3241dd7e4aee428d848660c36f9ebe7d091db2450674a045d0c27d7eab"}
Oct 31 12:45:28.002041 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:28.001673    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ed25a690-z8cwl" event={"ID":"04c44c3d-e699-4f80-b1b9-2ac15a1544e1","Type":"ContainerStarted","Data":"6cffdcb7630d22c0002cc50961948285a72010a4dd855cf7cbe5265fbf6ad545"}
Oct 31 12:45:29.017314 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:29.017276    5060 generic.go:356] "Generic (PLEG): container finished" podID="04c44c3d-e699-4f80-b1b9-2ac15a1544e1" containerID="20a88c3241dd7e4aee428d848660c36f9ebe7d091db2450674a045d0c27d7eab" exitCode=0
Oct 31 12:45:29.017807 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:29.017350    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ed25a690-z8cwl" event={"ID":"04c44c3d-e699-4f80-b1b9-2ac15a1544e1","Type":"ContainerDied","Data":"20a88c3241dd7e4aee428d848660c36f9ebe7d091db2450674a045d0c27d7eab"}
Oct 31 12:45:29.242174 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:29.242142    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ed25a690-z8cwl"
Oct 31 12:45:29.284448 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:29.284409    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04c44c3d-e699-4f80-b1b9-2ac15a1544e1-kube-api-access-2bd4r\" (UniqueName: \"kubernetes.io/projected/04c44c3d-e699-4f80-b1b9-2ac15a1544e1-kube-api-access-2bd4r\") pod \"04c44c3d-e699-4f80-b1b9-2ac15a1544e1\" (UID: \"04c44c3d-e699-4f80-b1b9-2ac15a1544e1\") "
Oct 31 12:45:29.284685 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:29.284657    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04c44c3d-e699-4f80-b1b9-2ac15a1544e1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04c44c3d-e699-4f80-b1b9-2ac15a1544e1-maas-api-openshift-ca\") pod \"04c44c3d-e699-4f80-b1b9-2ac15a1544e1\" (UID: \"04c44c3d-e699-4f80-b1b9-2ac15a1544e1\") "
Oct 31 12:45:29.284816 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:29.284799    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/04c44c3d-e699-4f80-b1b9-2ac15a1544e1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04c44c3d-e699-4f80-b1b9-2ac15a1544e1-maas-api-xks-ca\") pod \"04c44c3d-e699-4f80-b1b9-2ac15a1544e1\" (UID: \"04c44c3d-e699-4f80-b1b9-2ac15a1544e1\") "
Oct 31 12:45:29.285014 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:29.284987    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04c44c3d-e699-4f80-b1b9-2ac15a1544e1-maas-api-openshift-ca" pod "04c44c3d-e699-4f80-b1b9-2ac15a1544e1" (UID: "04c44c3d-e699-4f80-b1b9-2ac15a1544e1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:45:29.285472 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:29.285446    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04c44c3d-e699-4f80-b1b9-2ac15a1544e1-maas-api-openshift-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:45:29.287239 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:29.287216    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04c44c3d-e699-4f80-b1b9-2ac15a1544e1-maas-api-xks-ca" pod "04c44c3d-e699-4f80-b1b9-2ac15a1544e1" (UID: "04c44c3d-e699-4f80-b1b9-2ac15a1544e1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:45:29.296453 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:29.296421    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04c44c3d-e699-4f80-b1b9-2ac15a1544e1-kube-api-access-2bd4r" pod "04c44c3d-e699-4f80-b1b9-2ac15a1544e1" (UID: "04c44c3d-e699-4f80-b1b9-2ac15a1544e1"). InnerVolumeSpecName "kube-api-access-2bd4r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:45:29.386577 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:29.386547    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04c44c3d-e699-4f80-b1b9-2ac15a1544e1-maas-api-xks-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:45:29.386577 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:29.386575    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2bd4r\" (UniqueName: \"kubernetes.io/projected/04c44c3d-e699-4f80-b1b9-2ac15a1544e1-kube-api-access-2bd4r\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:45:29.926639 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:45:29.926602    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7phb" containerName="extractor"
Oct 31 12:45:30.030413 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:30.030133    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98581cd6-p78l7"]
Oct 31 12:45:30.043956 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:30.043925    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ed25a690-z8cwl"
Oct 31 12:45:30.044239 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:30.044213    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ed25a690-z8cwl" event={"ID":"04c44c3d-e699-4f80-b1b9-2ac15a1544e1","Type":"ContainerDied","Data":"6cffdcb7630d22c0002cc50961948285a72010a4dd855cf7cbe5265fbf6ad545"}
Oct 31 12:45:30.044313 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:30.044263    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6cffdcb7630d22c0002cc50961948285a72010a4dd855cf7cbe5265fbf6ad545"
Oct 31 12:45:30.048602 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:30.048572    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98581cd6-p78l7"]
Oct 31 12:45:31.054018 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:31.053985    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d547045f-f0f5-4244-a709-dc84a1fd5de9" path="/var/lib/kubelet/pods/d547045f-f0f5-4244-a709-dc84a1fd5de9/volumes"
Oct 31 12:45:32.765431 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:32.765399    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4a6b547-bnbgj"]
Oct 31 12:45:32.766254 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:32.766235    5060 cpu_manager.go:414] "containerMap: removing container" podUID="5361e3ad-5107-41cc-8afa-f0c64e8236b1" containerName="cleanup"
Oct 31 12:45:32.766299 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:32.766255    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5361e3ad-5107-41cc-8afa-f0c64e8236b1" containerName="cleanup"
Oct 31 12:45:32.766299 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:32.766274    5060 cpu_manager.go:414] "containerMap: removing container" podUID="de4cc1ae-6e6f-4124-a0cd-464549508f9e" containerName="cleanup"
Oct 31 12:45:32.766299 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:32.766279    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de4cc1ae-6e6f-4124-a0cd-464549508f9e" containerName="cleanup"
Oct 31 12:45:32.766397 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:32.766301    5060 cpu_manager.go:414] "containerMap: removing container" podUID="d87b9890-61ac-4018-a312-c02787e3c371" containerName="cleanup"
Oct 31 12:45:32.766397 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:32.766307    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d87b9890-61ac-4018-a312-c02787e3c371" containerName="cleanup"
Oct 31 12:45:32.766397 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:32.766320    5060 cpu_manager.go:414] "containerMap: removing container" podUID="352baef6-da71-44a4-b1cc-f0960db22901" containerName="cleanup"
Oct 31 12:45:32.766397 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:32.766325    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="352baef6-da71-44a4-b1cc-f0960db22901" containerName="cleanup"
Oct 31 12:45:32.766397 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:32.766333    5060 cpu_manager.go:414] "containerMap: removing container" podUID="04c44c3d-e699-4f80-b1b9-2ac15a1544e1" containerName="revoke-keys"
Oct 31 12:45:32.766397 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:32.766338    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04c44c3d-e699-4f80-b1b9-2ac15a1544e1" containerName="revoke-keys"
Oct 31 12:45:32.766397 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:32.766358    5060 cpu_manager.go:414] "containerMap: removing container" podUID="dd0d4341-08cb-40e3-9062-c24256031504" containerName="cleanup"
Oct 31 12:45:32.766397 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:32.766362    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd0d4341-08cb-40e3-9062-c24256031504" containerName="cleanup"
Oct 31 12:45:32.766656 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:32.766475    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="09c33526-ddf7-41df-8b1a-bf098b046b84" containerName="cleanup"
Oct 31 12:45:32.766656 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:32.766485    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="352baef6-da71-44a4-b1cc-f0960db22901" containerName="cleanup"
Oct 31 12:45:32.766656 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:32.766508    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1b5757d-78b6-4614-b359-5398628f7913" containerName="cleanup"
Oct 31 12:45:32.766656 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:32.766519    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="04c44c3d-e699-4f80-b1b9-2ac15a1544e1" containerName="revoke-keys"
Oct 31 12:45:32.766656 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:32.766532    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd0d4341-08cb-40e3-9062-c24256031504" containerName="cleanup"
Oct 31 12:45:32.766656 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:32.766543    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="de4cc1ae-6e6f-4124-a0cd-464549508f9e" containerName="cleanup"
Oct 31 12:45:32.766656 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:32.766551    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="d87b9890-61ac-4018-a312-c02787e3c371" containerName="cleanup"
Oct 31 12:45:32.772246 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:32.772225    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4a6b547-bnbgj"
Oct 31 12:45:32.776638 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:32.776608    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 12:45:32.776823 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:32.776791    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7wsmb\""
Oct 31 12:45:32.778520 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:32.778397    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 12:45:32.778520 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:32.778453    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 12:45:32.781552 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:32.781526    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4a6b547-bnbgj"]
Oct 31 12:45:32.920286 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:32.920249    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c26a0a34-fc47-4c98-9eaa-2e36700c645d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4a6b547-bnbgj\" (UID: \"c26a0a34-fc47-4c98-9eaa-2e36700c645d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4a6b547-bnbgj"
Oct 31 12:45:32.920552 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:32.920395    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c969z\" (UniqueName: \"kubernetes.io/projected/c26a0a34-fc47-4c98-9eaa-2e36700c645d-kube-api-access-c969z\") pod \"maas-api-revoke-sub-a4a6b547-bnbgj\" (UID: \"c26a0a34-fc47-4c98-9eaa-2e36700c645d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4a6b547-bnbgj"
Oct 31 12:45:32.920794 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:32.920773    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c26a0a34-fc47-4c98-9eaa-2e36700c645d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4a6b547-bnbgj\" (UID: \"c26a0a34-fc47-4c98-9eaa-2e36700c645d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4a6b547-bnbgj"
Oct 31 12:45:33.030558 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:33.026273    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c26a0a34-fc47-4c98-9eaa-2e36700c645d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4a6b547-bnbgj\" (UID: \"c26a0a34-fc47-4c98-9eaa-2e36700c645d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4a6b547-bnbgj"
Oct 31 12:45:33.030558 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:33.026425    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c26a0a34-fc47-4c98-9eaa-2e36700c645d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4a6b547-bnbgj\" (UID: \"c26a0a34-fc47-4c98-9eaa-2e36700c645d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4a6b547-bnbgj"
Oct 31 12:45:33.030558 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:33.026540    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c969z\" (UniqueName: \"kubernetes.io/projected/c26a0a34-fc47-4c98-9eaa-2e36700c645d-kube-api-access-c969z\") pod \"maas-api-revoke-sub-a4a6b547-bnbgj\" (UID: \"c26a0a34-fc47-4c98-9eaa-2e36700c645d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4a6b547-bnbgj"
Oct 31 12:45:33.030558 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:33.028738    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c26a0a34-fc47-4c98-9eaa-2e36700c645d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4a6b547-bnbgj\" (UID: \"c26a0a34-fc47-4c98-9eaa-2e36700c645d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4a6b547-bnbgj"
Oct 31 12:45:33.043878 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:33.032654    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c26a0a34-fc47-4c98-9eaa-2e36700c645d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4a6b547-bnbgj\" (UID: \"c26a0a34-fc47-4c98-9eaa-2e36700c645d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4a6b547-bnbgj"
Oct 31 12:45:33.067701 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:33.051141    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c969z\" (UniqueName: \"kubernetes.io/projected/c26a0a34-fc47-4c98-9eaa-2e36700c645d-kube-api-access-c969z\") pod \"maas-api-revoke-sub-a4a6b547-bnbgj\" (UID: \"c26a0a34-fc47-4c98-9eaa-2e36700c645d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4a6b547-bnbgj"
Oct 31 12:45:33.099970 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:33.096822    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4a6b547-bnbgj"
Oct 31 12:45:33.443743 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:45:33.443711    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc26a0a34_fc47_4c98_9eaa_2e36700c645d.slice/crio-96287d0115e0f9d594f7c6658a40f19ec15a109bbe3fe14d2c19cce3630e186f WatchSource:0}: Error finding container 96287d0115e0f9d594f7c6658a40f19ec15a109bbe3fe14d2c19cce3630e186f: Status 404 returned error can't find the container with id 96287d0115e0f9d594f7c6658a40f19ec15a109bbe3fe14d2c19cce3630e186f
Oct 31 12:45:33.444269 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:33.444245    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4a6b547-bnbgj"]
Oct 31 12:45:34.149014 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:34.148922    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4a6b547-bnbgj" event={"ID":"c26a0a34-fc47-4c98-9eaa-2e36700c645d","Type":"ContainerStarted","Data":"91656a6e403cac0b8e7054ee9ad5f7e3746f95bf700620e2d08f90191d058f40"}
Oct 31 12:45:34.149587 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:34.149560    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4a6b547-bnbgj" event={"ID":"c26a0a34-fc47-4c98-9eaa-2e36700c645d","Type":"ContainerStarted","Data":"96287d0115e0f9d594f7c6658a40f19ec15a109bbe3fe14d2c19cce3630e186f"}
Oct 31 12:45:35.163259 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:35.163226    5060 generic.go:356] "Generic (PLEG): container finished" podID="c26a0a34-fc47-4c98-9eaa-2e36700c645d" containerID="91656a6e403cac0b8e7054ee9ad5f7e3746f95bf700620e2d08f90191d058f40" exitCode=0
Oct 31 12:45:35.163734 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:35.163316    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4a6b547-bnbgj" event={"ID":"c26a0a34-fc47-4c98-9eaa-2e36700c645d","Type":"ContainerDied","Data":"91656a6e403cac0b8e7054ee9ad5f7e3746f95bf700620e2d08f90191d058f40"}
Oct 31 12:45:36.601567 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:36.586803    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4a6b547-bnbgj"
Oct 31 12:45:36.717538 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:36.701042    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c26a0a34-fc47-4c98-9eaa-2e36700c645d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c26a0a34-fc47-4c98-9eaa-2e36700c645d-maas-api-openshift-ca\") pod \"c26a0a34-fc47-4c98-9eaa-2e36700c645d\" (UID: \"c26a0a34-fc47-4c98-9eaa-2e36700c645d\") "
Oct 31 12:45:36.717538 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:36.701174    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c26a0a34-fc47-4c98-9eaa-2e36700c645d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c26a0a34-fc47-4c98-9eaa-2e36700c645d-maas-api-xks-ca\") pod \"c26a0a34-fc47-4c98-9eaa-2e36700c645d\" (UID: \"c26a0a34-fc47-4c98-9eaa-2e36700c645d\") "
Oct 31 12:45:36.717538 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:36.701283    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c26a0a34-fc47-4c98-9eaa-2e36700c645d-kube-api-access-c969z\" (UniqueName: \"kubernetes.io/projected/c26a0a34-fc47-4c98-9eaa-2e36700c645d-kube-api-access-c969z\") pod \"c26a0a34-fc47-4c98-9eaa-2e36700c645d\" (UID: \"c26a0a34-fc47-4c98-9eaa-2e36700c645d\") "
Oct 31 12:45:36.739601 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:36.731722    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c26a0a34-fc47-4c98-9eaa-2e36700c645d-maas-api-openshift-ca" pod "c26a0a34-fc47-4c98-9eaa-2e36700c645d" (UID: "c26a0a34-fc47-4c98-9eaa-2e36700c645d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:45:36.779566 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:36.778166    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c26a0a34-fc47-4c98-9eaa-2e36700c645d-maas-api-xks-ca" pod "c26a0a34-fc47-4c98-9eaa-2e36700c645d" (UID: "c26a0a34-fc47-4c98-9eaa-2e36700c645d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:45:36.783530 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:36.780639    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c26a0a34-fc47-4c98-9eaa-2e36700c645d-kube-api-access-c969z" pod "c26a0a34-fc47-4c98-9eaa-2e36700c645d" (UID: "c26a0a34-fc47-4c98-9eaa-2e36700c645d"). InnerVolumeSpecName "kube-api-access-c969z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:45:36.811511 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:36.811132    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c26a0a34-fc47-4c98-9eaa-2e36700c645d-maas-api-openshift-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:45:36.811511 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:36.811181    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c26a0a34-fc47-4c98-9eaa-2e36700c645d-maas-api-xks-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:45:36.811511 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:36.811195    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c969z\" (UniqueName: \"kubernetes.io/projected/c26a0a34-fc47-4c98-9eaa-2e36700c645d-kube-api-access-c969z\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:45:37.215838 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:37.215809    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4a6b547-bnbgj"
Oct 31 12:45:37.216020 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:37.215844    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4a6b547-bnbgj" event={"ID":"c26a0a34-fc47-4c98-9eaa-2e36700c645d","Type":"ContainerDied","Data":"96287d0115e0f9d594f7c6658a40f19ec15a109bbe3fe14d2c19cce3630e186f"}
Oct 31 12:45:37.216020 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:37.215901    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="96287d0115e0f9d594f7c6658a40f19ec15a109bbe3fe14d2c19cce3630e186f"
Oct 31 12:45:42.033231 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:42.033193    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f518e29f-nt5kd"]
Oct 31 12:45:42.040074 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:42.040044    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f518e29f-nt5kd"]
Oct 31 12:45:43.055432 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:43.055395    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef3b98e5-482d-4f09-8c3d-ee886685f11f" path="/var/lib/kubelet/pods/ef3b98e5-482d-4f09-8c3d-ee886685f11f/volumes"
Oct 31 12:45:43.550029 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:43.549995    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ed25a690-z8cwl"]
Oct 31 12:45:43.557912 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:43.557868    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ed25a690-z8cwl"]
Oct 31 12:45:45.050238 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:45.050204    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04c44c3d-e699-4f80-b1b9-2ac15a1544e1" path="/var/lib/kubelet/pods/04c44c3d-e699-4f80-b1b9-2ac15a1544e1/volumes"
Oct 31 12:45:46.960231 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:46.960192    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d9c635e-6g2s2"]
Oct 31 12:45:46.961514 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:46.961472    5060 cpu_manager.go:414] "containerMap: removing container" podUID="c26a0a34-fc47-4c98-9eaa-2e36700c645d" containerName="revoke-keys"
Oct 31 12:45:46.961514 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:46.961512    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c26a0a34-fc47-4c98-9eaa-2e36700c645d" containerName="revoke-keys"
Oct 31 12:45:46.961819 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:46.961692    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="c26a0a34-fc47-4c98-9eaa-2e36700c645d" containerName="revoke-keys"
Oct 31 12:45:46.972822 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:46.972795    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d9c635e-6g2s2"
Oct 31 12:45:46.974561 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:46.974485    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d9c635e-6g2s2"]
Oct 31 12:45:46.981150 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:46.981129    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 12:45:46.984201 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:46.981442    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 12:45:46.984473 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:46.983675    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 12:45:46.984642 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:46.983738    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7wsmb\""
Oct 31 12:45:47.052139 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:47.052096    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8bz2\" (UniqueName: \"kubernetes.io/projected/6d3acdbc-4d7e-4381-9d28-58426f9b3ab1-kube-api-access-j8bz2\") pod \"maas-api-revoke-sub-3d9c635e-6g2s2\" (UID: \"6d3acdbc-4d7e-4381-9d28-58426f9b3ab1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d9c635e-6g2s2"
Oct 31 12:45:47.052377 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:47.052298    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d3acdbc-4d7e-4381-9d28-58426f9b3ab1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d9c635e-6g2s2\" (UID: \"6d3acdbc-4d7e-4381-9d28-58426f9b3ab1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d9c635e-6g2s2"
Oct 31 12:45:47.052595 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:47.052572    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d3acdbc-4d7e-4381-9d28-58426f9b3ab1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d9c635e-6g2s2\" (UID: \"6d3acdbc-4d7e-4381-9d28-58426f9b3ab1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d9c635e-6g2s2"
Oct 31 12:45:47.155035 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:47.154995    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8bz2\" (UniqueName: \"kubernetes.io/projected/6d3acdbc-4d7e-4381-9d28-58426f9b3ab1-kube-api-access-j8bz2\") pod \"maas-api-revoke-sub-3d9c635e-6g2s2\" (UID: \"6d3acdbc-4d7e-4381-9d28-58426f9b3ab1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d9c635e-6g2s2"
Oct 31 12:45:47.155231 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:47.155064    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d3acdbc-4d7e-4381-9d28-58426f9b3ab1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d9c635e-6g2s2\" (UID: \"6d3acdbc-4d7e-4381-9d28-58426f9b3ab1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d9c635e-6g2s2"
Oct 31 12:45:47.155354 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:47.155330    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d3acdbc-4d7e-4381-9d28-58426f9b3ab1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d9c635e-6g2s2\" (UID: \"6d3acdbc-4d7e-4381-9d28-58426f9b3ab1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d9c635e-6g2s2"
Oct 31 12:45:47.156103 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:47.156076    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d3acdbc-4d7e-4381-9d28-58426f9b3ab1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d9c635e-6g2s2\" (UID: \"6d3acdbc-4d7e-4381-9d28-58426f9b3ab1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d9c635e-6g2s2"
Oct 31 12:45:47.157699 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:47.157670    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d3acdbc-4d7e-4381-9d28-58426f9b3ab1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d9c635e-6g2s2\" (UID: \"6d3acdbc-4d7e-4381-9d28-58426f9b3ab1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d9c635e-6g2s2"
Oct 31 12:45:47.168554 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:47.168528    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8bz2\" (UniqueName: \"kubernetes.io/projected/6d3acdbc-4d7e-4381-9d28-58426f9b3ab1-kube-api-access-j8bz2\") pod \"maas-api-revoke-sub-3d9c635e-6g2s2\" (UID: \"6d3acdbc-4d7e-4381-9d28-58426f9b3ab1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d9c635e-6g2s2"
Oct 31 12:45:47.316756 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:47.316709    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d9c635e-6g2s2"
Oct 31 12:45:47.437234 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:47.436565    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-5d310d-openshift-default-7c8bb48c95-n2tkf"]
Oct 31 12:45:47.473680 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:47.473645    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-5d310d-openshift-default-7c8bb48c95-n2tkf"]
Oct 31 12:45:47.473876 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:47.473766    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-5d310d-openshift-default-7c8bb48c95-n2tkf"
Oct 31 12:45:47.480409 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:47.480378    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-5d310d-openshift-default-dockercfg-5wkkt\""
Oct 31 12:45:47.596565 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:47.592622    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e880a6e-3d1c-42e4-a6f3-a7a22c2cc35d-istiod-ca-cert\") pod \"e2e-api-a-5d310d-openshift-default-7c8bb48c95-n2tkf\" (UID: \"3e880a6e-3d1c-42e4-a6f3-a7a22c2cc35d\") " pod="openshift-ingress/e2e-api-a-5d310d-openshift-default-7c8bb48c95-n2tkf"
Oct 31 12:45:47.596565 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:47.592724    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e880a6e-3d1c-42e4-a6f3-a7a22c2cc35d-istio-token\") pod \"e2e-api-a-5d310d-openshift-default-7c8bb48c95-n2tkf\" (UID: \"3e880a6e-3d1c-42e4-a6f3-a7a22c2cc35d\") " pod="openshift-ingress/e2e-api-a-5d310d-openshift-default-7c8bb48c95-n2tkf"
Oct 31 12:45:47.596565 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:47.592764    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hgrh\" (UniqueName: \"kubernetes.io/projected/3e880a6e-3d1c-42e4-a6f3-a7a22c2cc35d-kube-api-access-6hgrh\") pod \"e2e-api-a-5d310d-openshift-default-7c8bb48c95-n2tkf\" (UID: \"3e880a6e-3d1c-42e4-a6f3-a7a22c2cc35d\") " pod="openshift-ingress/e2e-api-a-5d310d-openshift-default-7c8bb48c95-n2tkf"
Oct 31 12:45:47.596565 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:47.592836    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e880a6e-3d1c-42e4-a6f3-a7a22c2cc35d-workload-certs\") pod \"e2e-api-a-5d310d-openshift-default-7c8bb48c95-n2tkf\" (UID: \"3e880a6e-3d1c-42e4-a6f3-a7a22c2cc35d\") " pod="openshift-ingress/e2e-api-a-5d310d-openshift-default-7c8bb48c95-n2tkf"
Oct 31 12:45:47.596565 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:47.593082    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e880a6e-3d1c-42e4-a6f3-a7a22c2cc35d-credential-socket\") pod \"e2e-api-a-5d310d-openshift-default-7c8bb48c95-n2tkf\" (UID: \"3e880a6e-3d1c-42e4-a6f3-a7a22c2cc35d\") " pod="openshift-ingress/e2e-api-a-5d310d-openshift-default-7c8bb48c95-n2tkf"
Oct 31 12:45:47.596565 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:47.593120    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e880a6e-3d1c-42e4-a6f3-a7a22c2cc35d-istio-podinfo\") pod \"e2e-api-a-5d310d-openshift-default-7c8bb48c95-n2tkf\" (UID: \"3e880a6e-3d1c-42e4-a6f3-a7a22c2cc35d\") " pod="openshift-ingress/e2e-api-a-5d310d-openshift-default-7c8bb48c95-n2tkf"
Oct 31 12:45:47.596565 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:47.593286    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e880a6e-3d1c-42e4-a6f3-a7a22c2cc35d-workload-socket\") pod \"e2e-api-a-5d310d-openshift-default-7c8bb48c95-n2tkf\" (UID: \"3e880a6e-3d1c-42e4-a6f3-a7a22c2cc35d\") " pod="openshift-ingress/e2e-api-a-5d310d-openshift-default-7c8bb48c95-n2tkf"
Oct 31 12:45:47.596565 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:47.593314    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e880a6e-3d1c-42e4-a6f3-a7a22c2cc35d-istio-envoy\") pod \"e2e-api-a-5d310d-openshift-default-7c8bb48c95-n2tkf\" (UID: \"3e880a6e-3d1c-42e4-a6f3-a7a22c2cc35d\") " pod="openshift-ingress/e2e-api-a-5d310d-openshift-default-7c8bb48c95-n2tkf"
Oct 31 12:45:47.596565 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:47.593414    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e880a6e-3d1c-42e4-a6f3-a7a22c2cc35d-istio-data\") pod \"e2e-api-a-5d310d-openshift-default-7c8bb48c95-n2tkf\" (UID: \"3e880a6e-3d1c-42e4-a6f3-a7a22c2cc35d\") " pod="openshift-ingress/e2e-api-a-5d310d-openshift-default-7c8bb48c95-n2tkf"
Oct 31 12:45:47.820528 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:47.768746    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e880a6e-3d1c-42e4-a6f3-a7a22c2cc35d-credential-socket\") pod \"e2e-api-a-5d310d-openshift-default-7c8bb48c95-n2tkf\" (UID: \"3e880a6e-3d1c-42e4-a6f3-a7a22c2cc35d\") " pod="openshift-ingress/e2e-api-a-5d310d-openshift-default-7c8bb48c95-n2tkf"
Oct 31 12:45:47.820528 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:47.768791    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e880a6e-3d1c-42e4-a6f3-a7a22c2cc35d-istio-podinfo\") pod \"e2e-api-a-5d310d-openshift-default-7c8bb48c95-n2tkf\" (UID: \"3e880a6e-3d1c-42e4-a6f3-a7a22c2cc35d\") " pod="openshift-ingress/e2e-api-a-5d310d-openshift-default-7c8bb48c95-n2tkf"
Oct 31 12:45:47.820528 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:47.768891    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e880a6e-3d1c-42e4-a6f3-a7a22c2cc35d-workload-socket\") pod \"e2e-api-a-5d310d-openshift-default-7c8bb48c95-n2tkf\" (UID: \"3e880a6e-3d1c-42e4-a6f3-a7a22c2cc35d\") " pod="openshift-ingress/e2e-api-a-5d310d-openshift-default-7c8bb48c95-n2tkf"
Oct 31 12:45:47.820528 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:47.768912    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e880a6e-3d1c-42e4-a6f3-a7a22c2cc35d-istio-envoy\") pod \"e2e-api-a-5d310d-openshift-default-7c8bb48c95-n2tkf\" (UID: \"3e880a6e-3d1c-42e4-a6f3-a7a22c2cc35d\") " pod="openshift-ingress/e2e-api-a-5d310d-openshift-default-7c8bb48c95-n2tkf"
Oct 31 12:45:47.820528 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:47.768985    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e880a6e-3d1c-42e4-a6f3-a7a22c2cc35d-istio-data\") pod \"e2e-api-a-5d310d-openshift-default-7c8bb48c95-n2tkf\" (UID: \"3e880a6e-3d1c-42e4-a6f3-a7a22c2cc35d\") " pod="openshift-ingress/e2e-api-a-5d310d-openshift-default-7c8bb48c95-n2tkf"
Oct 31 12:45:47.820528 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:47.769056    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e880a6e-3d1c-42e4-a6f3-a7a22c2cc35d-istiod-ca-cert\") pod \"e2e-api-a-5d310d-openshift-default-7c8bb48c95-n2tkf\" (UID: \"3e880a6e-3d1c-42e4-a6f3-a7a22c2cc35d\") " pod="openshift-ingress/e2e-api-a-5d310d-openshift-default-7c8bb48c95-n2tkf"
Oct 31 12:45:47.820528 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:47.769109    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e880a6e-3d1c-42e4-a6f3-a7a22c2cc35d-istio-token\") pod \"e2e-api-a-5d310d-openshift-default-7c8bb48c95-n2tkf\" (UID: \"3e880a6e-3d1c-42e4-a6f3-a7a22c2cc35d\") " pod="openshift-ingress/e2e-api-a-5d310d-openshift-default-7c8bb48c95-n2tkf"
Oct 31 12:45:47.820528 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:47.769139    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hgrh\" (UniqueName: \"kubernetes.io/projected/3e880a6e-3d1c-42e4-a6f3-a7a22c2cc35d-kube-api-access-6hgrh\") pod \"e2e-api-a-5d310d-openshift-default-7c8bb48c95-n2tkf\" (UID: \"3e880a6e-3d1c-42e4-a6f3-a7a22c2cc35d\") " pod="openshift-ingress/e2e-api-a-5d310d-openshift-default-7c8bb48c95-n2tkf"
Oct 31 12:45:47.820528 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:47.769192    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e880a6e-3d1c-42e4-a6f3-a7a22c2cc35d-workload-certs\") pod \"e2e-api-a-5d310d-openshift-default-7c8bb48c95-n2tkf\" (UID: \"3e880a6e-3d1c-42e4-a6f3-a7a22c2cc35d\") " pod="openshift-ingress/e2e-api-a-5d310d-openshift-default-7c8bb48c95-n2tkf"
Oct 31 12:45:47.820528 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:47.769754    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e880a6e-3d1c-42e4-a6f3-a7a22c2cc35d-workload-certs\") pod \"e2e-api-a-5d310d-openshift-default-7c8bb48c95-n2tkf\" (UID: \"3e880a6e-3d1c-42e4-a6f3-a7a22c2cc35d\") " pod="openshift-ingress/e2e-api-a-5d310d-openshift-default-7c8bb48c95-n2tkf"
Oct 31 12:45:47.820528 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:47.770347    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e880a6e-3d1c-42e4-a6f3-a7a22c2cc35d-credential-socket\") pod \"e2e-api-a-5d310d-openshift-default-7c8bb48c95-n2tkf\" (UID: \"3e880a6e-3d1c-42e4-a6f3-a7a22c2cc35d\") " pod="openshift-ingress/e2e-api-a-5d310d-openshift-default-7c8bb48c95-n2tkf"
Oct 31 12:45:47.820528 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:47.779542    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e880a6e-3d1c-42e4-a6f3-a7a22c2cc35d-workload-socket\") pod \"e2e-api-a-5d310d-openshift-default-7c8bb48c95-n2tkf\" (UID: \"3e880a6e-3d1c-42e4-a6f3-a7a22c2cc35d\") " pod="openshift-ingress/e2e-api-a-5d310d-openshift-default-7c8bb48c95-n2tkf"
Oct 31 12:45:47.820528 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:47.779862    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e880a6e-3d1c-42e4-a6f3-a7a22c2cc35d-istio-data\") pod \"e2e-api-a-5d310d-openshift-default-7c8bb48c95-n2tkf\" (UID: \"3e880a6e-3d1c-42e4-a6f3-a7a22c2cc35d\") " pod="openshift-ingress/e2e-api-a-5d310d-openshift-default-7c8bb48c95-n2tkf"
Oct 31 12:45:47.945776 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:47.943406    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e880a6e-3d1c-42e4-a6f3-a7a22c2cc35d-istio-envoy\") pod \"e2e-api-a-5d310d-openshift-default-7c8bb48c95-n2tkf\" (UID: \"3e880a6e-3d1c-42e4-a6f3-a7a22c2cc35d\") " pod="openshift-ingress/e2e-api-a-5d310d-openshift-default-7c8bb48c95-n2tkf"
Oct 31 12:45:47.948468 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:45:47.948310    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d3acdbc_4d7e_4381_9d28_58426f9b3ab1.slice/crio-c8b99f0ceafb58f8c4be69a5fd1abcebdc769c599db3ddf39159be10914244aa WatchSource:0}: Error finding container c8b99f0ceafb58f8c4be69a5fd1abcebdc769c599db3ddf39159be10914244aa: Status 404 returned error can't find the container with id c8b99f0ceafb58f8c4be69a5fd1abcebdc769c599db3ddf39159be10914244aa
Oct 31 12:45:48.029079 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:47.980086    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e880a6e-3d1c-42e4-a6f3-a7a22c2cc35d-istiod-ca-cert\") pod \"e2e-api-a-5d310d-openshift-default-7c8bb48c95-n2tkf\" (UID: \"3e880a6e-3d1c-42e4-a6f3-a7a22c2cc35d\") " pod="openshift-ingress/e2e-api-a-5d310d-openshift-default-7c8bb48c95-n2tkf"
Oct 31 12:45:48.029079 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:48.015280    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e880a6e-3d1c-42e4-a6f3-a7a22c2cc35d-istio-token\") pod \"e2e-api-a-5d310d-openshift-default-7c8bb48c95-n2tkf\" (UID: \"3e880a6e-3d1c-42e4-a6f3-a7a22c2cc35d\") " pod="openshift-ingress/e2e-api-a-5d310d-openshift-default-7c8bb48c95-n2tkf"
Oct 31 12:45:48.029079 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:48.016008    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hgrh\" (UniqueName: \"kubernetes.io/projected/3e880a6e-3d1c-42e4-a6f3-a7a22c2cc35d-kube-api-access-6hgrh\") pod \"e2e-api-a-5d310d-openshift-default-7c8bb48c95-n2tkf\" (UID: \"3e880a6e-3d1c-42e4-a6f3-a7a22c2cc35d\") " pod="openshift-ingress/e2e-api-a-5d310d-openshift-default-7c8bb48c95-n2tkf"
Oct 31 12:45:48.029079 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:48.021111    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e880a6e-3d1c-42e4-a6f3-a7a22c2cc35d-istio-podinfo\") pod \"e2e-api-a-5d310d-openshift-default-7c8bb48c95-n2tkf\" (UID: \"3e880a6e-3d1c-42e4-a6f3-a7a22c2cc35d\") " pod="openshift-ingress/e2e-api-a-5d310d-openshift-default-7c8bb48c95-n2tkf"
Oct 31 12:45:48.029079 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:48.024179    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d9c635e-6g2s2"]
Oct 31 12:45:48.204792 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:48.193829    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-5d310d-openshift-default-7c8bb48c95-n2tkf"
Oct 31 12:45:49.009260 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:49.009164    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d9c635e-6g2s2" event={"ID":"6d3acdbc-4d7e-4381-9d28-58426f9b3ab1","Type":"ContainerStarted","Data":"c8b99f0ceafb58f8c4be69a5fd1abcebdc769c599db3ddf39159be10914244aa"}
Oct 31 12:45:49.236515 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:49.226247    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-5d310d-openshift-default-7c8bb48c95-n2tkf"]
Oct 31 12:45:49.297855 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:45:49.297807    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e880a6e_3d1c_42e4_a6f3_a7a22c2cc35d.slice/crio-52d1909554c75a71bc00ec9ae2bc7752793b197eaf51c99f87aadd164f432fe6 WatchSource:0}: Error finding container 52d1909554c75a71bc00ec9ae2bc7752793b197eaf51c99f87aadd164f432fe6: Status 404 returned error can't find the container with id 52d1909554c75a71bc00ec9ae2bc7752793b197eaf51c99f87aadd164f432fe6
Oct 31 12:45:49.360840 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:49.355101    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:45:49.360840 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:49.355228    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:45:49.368764 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:49.365578    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:45:50.052709 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:50.052674    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5d310d-openshift-default-7c8bb48c95-n2tkf" event={"ID":"3e880a6e-3d1c-42e4-a6f3-a7a22c2cc35d","Type":"ContainerStarted","Data":"a7f4a5da16afcef056b5a5fda046ab0306d428991771005d472a0b191d3ec202"}
Oct 31 12:45:50.052978 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:50.052956    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5d310d-openshift-default-7c8bb48c95-n2tkf" event={"ID":"3e880a6e-3d1c-42e4-a6f3-a7a22c2cc35d","Type":"ContainerStarted","Data":"52d1909554c75a71bc00ec9ae2bc7752793b197eaf51c99f87aadd164f432fe6"}
Oct 31 12:45:50.082482 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:50.082446    5060 generic.go:356] "Generic (PLEG): container finished" podID="6d3acdbc-4d7e-4381-9d28-58426f9b3ab1" containerID="c447daaf1be3e2ecf74dbc99fd58d71d16f56325999e75a7bbbf6fb6300625dc" exitCode=0
Oct 31 12:45:50.110981 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:50.083225    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d9c635e-6g2s2" event={"ID":"6d3acdbc-4d7e-4381-9d28-58426f9b3ab1","Type":"ContainerDied","Data":"c447daaf1be3e2ecf74dbc99fd58d71d16f56325999e75a7bbbf6fb6300625dc"}
Oct 31 12:45:50.136732 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:50.136673    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-5d310d-openshift-default-7c8bb48c95-n2tkf" podStartSLOduration=3.136653851 podStartE2EDuration="3.136653851s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:45:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:45:50.115217673 +0000 UTC m=+1113.218285386" watchObservedRunningTime="2026-10-31 12:45:50.136653851 +0000 UTC m=+1113.239721565"
Oct 31 12:45:50.212051 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:50.212019    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-5d310d-openshift-default-7c8bb48c95-n2tkf"
Oct 31 12:45:50.212383 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:50.212360    5060 patch_prober.go:29] interesting pod/e2e-api-a-5d310d-openshift-default-7c8bb48c95-n2tkf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body=
Oct 31 12:45:50.212559 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:50.212533    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-5d310d-openshift-default-7c8bb48c95-n2tkf" podUID="3e880a6e-3d1c-42e4-a6f3-a7a22c2cc35d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused"
Oct 31 12:45:51.229514 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:51.229068    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-5d310d-openshift-default-7c8bb48c95-n2tkf"
Oct 31 12:45:51.256517 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:51.255724    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-5d310d-openshift-default-7c8bb48c95-n2tkf"
Oct 31 12:45:51.280051 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:51.278828    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-5d310d-openshift-default-7c8bb48c95-n2tkf"
Oct 31 12:45:52.041986 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:52.041926    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/premium-w2-e6c79d-kserve-6468bf46d4-h66f8" podUID="cd6a266f-8d12-4089-83ee-c75c716bf26c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Oct 31 12:45:52.050532 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:52.050396    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-cr49v"]
Oct 31 12:45:52.050751 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:52.050668    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-cr49v" podUID="f957e25b-5bf0-4259-afbd-fcd50a9364d5" containerName="istio-proxy" containerID="cri-o://5626dc1a2d0125d74428281ca91d1a94e7eddb74d5d5b713dd45ea1ed1aa7227" gracePeriod=30
Oct 31 12:45:52.151758 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:52.151698    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x2qzv" podUID="c247ed2a-aa90-424b-9b7e-4f2936d67e3f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:45:52.205665 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:52.205600    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/simulator-w4-c67de2-kserve-868679cf9b-gqjbx" podUID="e104d7da-1fb2-4c99-a480-b4ae84c4f273" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded"
Oct 31 12:45:52.363571 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:52.360723    5060 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x2qzv" podUID="c247ed2a-aa90-424b-9b7e-4f2936d67e3f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:45:53.030550 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:52.985616    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-2-w5-b25d3b-kserve-76f798d59f-vvzhg" podUID="394ed76c-84ac-4612-9acb-a7e36cb68d89" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:45:53.062534 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:53.054164    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/premium-w2-e6c79d-kserve-6468bf46d4-h66f8" podUID="cd6a266f-8d12-4089-83ee-c75c716bf26c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Oct 31 12:45:53.237950 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:53.175565    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x2qzv" podUID="c247ed2a-aa90-424b-9b7e-4f2936d67e3f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:45:53.237950 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:53.211728    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/simulator-w4-c67de2-kserve-868679cf9b-gqjbx" podUID="e104d7da-1fb2-4c99-a480-b4ae84c4f273" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:45:53.684152 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:53.679322    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e6c79d/premium-w2-e6c79d-kserve-6468bf46d4-h66f8"
Oct 31 12:45:53.684152 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:53.679396    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x2qzv"
Oct 31 12:45:53.684152 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:53.679437    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c67de2/simulator-w4-c67de2-kserve-868679cf9b-gqjbx"
Oct 31 12:45:53.785747 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:53.785710    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a4565dd-gcdht"]
Oct 31 12:45:53.962219 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:53.930934    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a4565dd-gcdht"]
Oct 31 12:45:53.999636 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:53.999586    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-2-w5-b25d3b-kserve-76f798d59f-vvzhg" podUID="394ed76c-84ac-4612-9acb-a7e36cb68d89" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:45:54.000440 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:54.000410    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-2-w5-b25d3b-kserve-76f798d59f-vvzhg"
Oct 31 12:45:54.079516 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:54.054394    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/premium-w2-e6c79d-kserve-6468bf46d4-h66f8" podUID="cd6a266f-8d12-4089-83ee-c75c716bf26c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:45:54.108023 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:54.107638    5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-645cf464d7-v2sv5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 12:45:54.108023 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:54.107713    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-v2sv5" podUID="1e33b790-15e5-4754-929f-c0a7330b9361" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:45:54.199539 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:54.188605    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x2qzv" podUID="c247ed2a-aa90-424b-9b7e-4f2936d67e3f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:45:54.219572 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:54.201533    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d9c635e-6g2s2"
Oct 31 12:45:54.230159 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:54.227048    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d9c635e-6g2s2" event={"ID":"6d3acdbc-4d7e-4381-9d28-58426f9b3ab1","Type":"ContainerDied","Data":"c8b99f0ceafb58f8c4be69a5fd1abcebdc769c599db3ddf39159be10914244aa"}
Oct 31 12:45:54.230159 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:54.227107    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c8b99f0ceafb58f8c4be69a5fd1abcebdc769c599db3ddf39159be10914244aa"
Oct 31 12:45:54.265711 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:54.265479    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c67de2/simulator-w4-c67de2-kserve-868679cf9b-gqjbx"
Oct 31 12:45:54.450525 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:54.411138    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6d3acdbc-4d7e-4381-9d28-58426f9b3ab1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d3acdbc-4d7e-4381-9d28-58426f9b3ab1-maas-api-xks-ca\") pod \"6d3acdbc-4d7e-4381-9d28-58426f9b3ab1\" (UID: \"6d3acdbc-4d7e-4381-9d28-58426f9b3ab1\") "
Oct 31 12:45:54.450525 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:54.411365    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d3acdbc-4d7e-4381-9d28-58426f9b3ab1-kube-api-access-j8bz2\" (UniqueName: \"kubernetes.io/projected/6d3acdbc-4d7e-4381-9d28-58426f9b3ab1-kube-api-access-j8bz2\") pod \"6d3acdbc-4d7e-4381-9d28-58426f9b3ab1\" (UID: \"6d3acdbc-4d7e-4381-9d28-58426f9b3ab1\") "
Oct 31 12:45:54.450525 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:54.411557    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d3acdbc-4d7e-4381-9d28-58426f9b3ab1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d3acdbc-4d7e-4381-9d28-58426f9b3ab1-maas-api-openshift-ca\") pod \"6d3acdbc-4d7e-4381-9d28-58426f9b3ab1\" (UID: \"6d3acdbc-4d7e-4381-9d28-58426f9b3ab1\") "
Oct 31 12:45:54.450525 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:54.448989    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-2-w5-b25d3b-kserve-76f798d59f-vvzhg"
Oct 31 12:45:54.450892 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:54.450829    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d3acdbc-4d7e-4381-9d28-58426f9b3ab1-maas-api-openshift-ca" pod "6d3acdbc-4d7e-4381-9d28-58426f9b3ab1" (UID: "6d3acdbc-4d7e-4381-9d28-58426f9b3ab1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:45:54.555805 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:54.555755    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d3acdbc-4d7e-4381-9d28-58426f9b3ab1-maas-api-openshift-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:45:54.886536 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:54.874907    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d3acdbc-4d7e-4381-9d28-58426f9b3ab1-maas-api-xks-ca" pod "6d3acdbc-4d7e-4381-9d28-58426f9b3ab1" (UID: "6d3acdbc-4d7e-4381-9d28-58426f9b3ab1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:45:55.326676 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:55.326630    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x2qzv" podUID="c247ed2a-aa90-424b-9b7e-4f2936d67e3f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:45:55.423722 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:55.423684    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d3acdbc-4d7e-4381-9d28-58426f9b3ab1-maas-api-xks-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:45:55.453550 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:55.451255    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d3acdbc-4d7e-4381-9d28-58426f9b3ab1-kube-api-access-j8bz2" pod "6d3acdbc-4d7e-4381-9d28-58426f9b3ab1" (UID: "6d3acdbc-4d7e-4381-9d28-58426f9b3ab1"). InnerVolumeSpecName "kube-api-access-j8bz2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:45:55.453550 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:55.347988    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e6c79d/premium-w2-e6c79d-kserve-6468bf46d4-h66f8"
Oct 31 12:45:55.527202 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:55.527170    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8bz2\" (UniqueName: \"kubernetes.io/projected/6d3acdbc-4d7e-4381-9d28-58426f9b3ab1-kube-api-access-j8bz2\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:45:55.540747 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:55.540715    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d9c635e-6g2s2"
Oct 31 12:45:55.551434 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:55.551401    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f2bbf07-a93d-4238-a69a-3530a1cbe1e2" path="/var/lib/kubelet/pods/0f2bbf07-a93d-4238-a69a-3530a1cbe1e2/volumes"
Oct 31 12:45:55.554402 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:55.554377    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x2qzv"
Oct 31 12:45:57.621566 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:57.621483    5060 generic.go:356] "Generic (PLEG): container finished" podID="f957e25b-5bf0-4259-afbd-fcd50a9364d5" containerID="5626dc1a2d0125d74428281ca91d1a94e7eddb74d5d5b713dd45ea1ed1aa7227" exitCode=0
Oct 31 12:45:57.622055 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:57.621584    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-cr49v" event={"ID":"f957e25b-5bf0-4259-afbd-fcd50a9364d5","Type":"ContainerDied","Data":"5626dc1a2d0125d74428281ca91d1a94e7eddb74d5d5b713dd45ea1ed1aa7227"}
Oct 31 12:45:57.622055 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:57.621665    5060 scope.go:122] "RemoveContainer" containerID="3fdffae8b53e3c69a2867c4800bf0be6233f9ec417c3386499390379583398f6"
Oct 31 12:45:57.914534 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:57.914473    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-cr49v"
Oct 31 12:45:57.943041 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:57.943004    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f957e25b-5bf0-4259-afbd-fcd50a9364d5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f957e25b-5bf0-4259-afbd-fcd50a9364d5-istio-podinfo\") pod \"f957e25b-5bf0-4259-afbd-fcd50a9364d5\" (UID: \"f957e25b-5bf0-4259-afbd-fcd50a9364d5\") "
Oct 31 12:45:57.944336 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:57.943223    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f957e25b-5bf0-4259-afbd-fcd50a9364d5-istio-token\" (UniqueName: \"kubernetes.io/projected/f957e25b-5bf0-4259-afbd-fcd50a9364d5-istio-token\") pod \"f957e25b-5bf0-4259-afbd-fcd50a9364d5\" (UID: \"f957e25b-5bf0-4259-afbd-fcd50a9364d5\") "
Oct 31 12:45:57.944336 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:57.943267    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f957e25b-5bf0-4259-afbd-fcd50a9364d5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f957e25b-5bf0-4259-afbd-fcd50a9364d5-istiod-ca-cert\") pod \"f957e25b-5bf0-4259-afbd-fcd50a9364d5\" (UID: \"f957e25b-5bf0-4259-afbd-fcd50a9364d5\") "
Oct 31 12:45:57.945643 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:57.945600    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f957e25b-5bf0-4259-afbd-fcd50a9364d5-istiod-ca-cert" pod "f957e25b-5bf0-4259-afbd-fcd50a9364d5" (UID: "f957e25b-5bf0-4259-afbd-fcd50a9364d5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:45:57.949253 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:57.949204    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f957e25b-5bf0-4259-afbd-fcd50a9364d5-istio-podinfo" pod "f957e25b-5bf0-4259-afbd-fcd50a9364d5" (UID: "f957e25b-5bf0-4259-afbd-fcd50a9364d5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 12:45:57.951371 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:57.951333    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f957e25b-5bf0-4259-afbd-fcd50a9364d5-istio-token" pod "f957e25b-5bf0-4259-afbd-fcd50a9364d5" (UID: "f957e25b-5bf0-4259-afbd-fcd50a9364d5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:45:58.074566 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:58.063040    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f957e25b-5bf0-4259-afbd-fcd50a9364d5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f957e25b-5bf0-4259-afbd-fcd50a9364d5-istio-data\") pod \"f957e25b-5bf0-4259-afbd-fcd50a9364d5\" (UID: \"f957e25b-5bf0-4259-afbd-fcd50a9364d5\") "
Oct 31 12:45:58.074566 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:58.063108    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f957e25b-5bf0-4259-afbd-fcd50a9364d5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f957e25b-5bf0-4259-afbd-fcd50a9364d5-workload-certs\") pod \"f957e25b-5bf0-4259-afbd-fcd50a9364d5\" (UID: \"f957e25b-5bf0-4259-afbd-fcd50a9364d5\") "
Oct 31 12:45:58.074566 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:58.063171    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f957e25b-5bf0-4259-afbd-fcd50a9364d5-kube-api-access-pbrbd\" (UniqueName: \"kubernetes.io/projected/f957e25b-5bf0-4259-afbd-fcd50a9364d5-kube-api-access-pbrbd\") pod \"f957e25b-5bf0-4259-afbd-fcd50a9364d5\" (UID: \"f957e25b-5bf0-4259-afbd-fcd50a9364d5\") "
Oct 31 12:45:58.074566 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:58.063289    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f957e25b-5bf0-4259-afbd-fcd50a9364d5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f957e25b-5bf0-4259-afbd-fcd50a9364d5-workload-socket\") pod \"f957e25b-5bf0-4259-afbd-fcd50a9364d5\" (UID: \"f957e25b-5bf0-4259-afbd-fcd50a9364d5\") "
Oct 31 12:45:58.074566 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:58.063564    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f957e25b-5bf0-4259-afbd-fcd50a9364d5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f957e25b-5bf0-4259-afbd-fcd50a9364d5-credential-socket\") pod \"f957e25b-5bf0-4259-afbd-fcd50a9364d5\" (UID: \"f957e25b-5bf0-4259-afbd-fcd50a9364d5\") "
Oct 31 12:45:58.074566 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:58.063622    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f957e25b-5bf0-4259-afbd-fcd50a9364d5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f957e25b-5bf0-4259-afbd-fcd50a9364d5-istio-envoy\") pod \"f957e25b-5bf0-4259-afbd-fcd50a9364d5\" (UID: \"f957e25b-5bf0-4259-afbd-fcd50a9364d5\") "
Oct 31 12:45:58.074566 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:58.064560    5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f957e25b-5bf0-4259-afbd-fcd50a9364d5-istio-token\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:45:58.074566 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:58.064589    5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f957e25b-5bf0-4259-afbd-fcd50a9364d5-istiod-ca-cert\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:45:58.074566 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:58.064607    5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f957e25b-5bf0-4259-afbd-fcd50a9364d5-istio-podinfo\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:45:58.074566 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:58.071046    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f957e25b-5bf0-4259-afbd-fcd50a9364d5-istio-data" pod "f957e25b-5bf0-4259-afbd-fcd50a9364d5" (UID: "f957e25b-5bf0-4259-afbd-fcd50a9364d5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:45:58.074566 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:58.071240    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f957e25b-5bf0-4259-afbd-fcd50a9364d5-workload-certs" pod "f957e25b-5bf0-4259-afbd-fcd50a9364d5" (UID: "f957e25b-5bf0-4259-afbd-fcd50a9364d5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:45:58.074566 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:58.071922    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f957e25b-5bf0-4259-afbd-fcd50a9364d5-workload-socket" pod "f957e25b-5bf0-4259-afbd-fcd50a9364d5" (UID: "f957e25b-5bf0-4259-afbd-fcd50a9364d5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:45:58.074566 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:58.072111    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f957e25b-5bf0-4259-afbd-fcd50a9364d5-credential-socket" pod "f957e25b-5bf0-4259-afbd-fcd50a9364d5" (UID: "f957e25b-5bf0-4259-afbd-fcd50a9364d5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:45:58.081452 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:58.081402    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f957e25b-5bf0-4259-afbd-fcd50a9364d5-istio-envoy" pod "f957e25b-5bf0-4259-afbd-fcd50a9364d5" (UID: "f957e25b-5bf0-4259-afbd-fcd50a9364d5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:45:58.098152 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:58.098090    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f957e25b-5bf0-4259-afbd-fcd50a9364d5-kube-api-access-pbrbd" pod "f957e25b-5bf0-4259-afbd-fcd50a9364d5" (UID: "f957e25b-5bf0-4259-afbd-fcd50a9364d5"). InnerVolumeSpecName "kube-api-access-pbrbd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:45:58.168123 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:58.167014    5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f957e25b-5bf0-4259-afbd-fcd50a9364d5-credential-socket\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:45:58.168123 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:58.167049    5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f957e25b-5bf0-4259-afbd-fcd50a9364d5-istio-envoy\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:45:58.168123 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:58.167066    5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f957e25b-5bf0-4259-afbd-fcd50a9364d5-istio-data\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:45:58.168123 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:58.167081    5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f957e25b-5bf0-4259-afbd-fcd50a9364d5-workload-certs\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:45:58.168123 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:58.167097    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbrbd\" (UniqueName: \"kubernetes.io/projected/f957e25b-5bf0-4259-afbd-fcd50a9364d5-kube-api-access-pbrbd\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:45:58.168123 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:58.167111    5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f957e25b-5bf0-4259-afbd-fcd50a9364d5-workload-socket\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:45:58.638718 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:58.638680    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-cr49v"
Oct 31 12:45:58.639177 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:58.638671    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-cr49v" event={"ID":"f957e25b-5bf0-4259-afbd-fcd50a9364d5","Type":"ContainerDied","Data":"fa5bf1df3de5e9e3c6dbb7fd8d753b72bf72c33ada2ac1bec2284ed36f4709f0"}
Oct 31 12:45:58.639177 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:58.638833    5060 scope.go:122] "RemoveContainer" containerID="5626dc1a2d0125d74428281ca91d1a94e7eddb74d5d5b713dd45ea1ed1aa7227"
Oct 31 12:45:58.676743 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:58.676672    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-cr49v"]
Oct 31 12:45:58.684275 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:58.684242    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-cr49v"]
Oct 31 12:45:59.053313 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:45:59.053277    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f957e25b-5bf0-4259-afbd-fcd50a9364d5" path="/var/lib/kubelet/pods/f957e25b-5bf0-4259-afbd-fcd50a9364d5/volumes"
Oct 31 12:45:59.930976 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:45:59.930931    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7phb" containerName="extractor"
Oct 31 12:46:00.546517 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:00.514550    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-f22e7f-openshift-default-84f9758845-24j6m"]
Oct 31 12:46:00.546517 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:00.515852    5060 cpu_manager.go:414] "containerMap: removing container" podUID="f957e25b-5bf0-4259-afbd-fcd50a9364d5" containerName="istio-proxy"
Oct 31 12:46:00.546517 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:00.515877    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f957e25b-5bf0-4259-afbd-fcd50a9364d5" containerName="istio-proxy"
Oct 31 12:46:00.546517 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:00.515929    5060 cpu_manager.go:414] "containerMap: removing container" podUID="6d3acdbc-4d7e-4381-9d28-58426f9b3ab1" containerName="revoke-keys"
Oct 31 12:46:00.546517 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:00.515938    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d3acdbc-4d7e-4381-9d28-58426f9b3ab1" containerName="revoke-keys"
Oct 31 12:46:00.546517 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:00.515996    5060 cpu_manager.go:414] "containerMap: removing container" podUID="f957e25b-5bf0-4259-afbd-fcd50a9364d5" containerName="istio-proxy"
Oct 31 12:46:00.546517 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:00.516005    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f957e25b-5bf0-4259-afbd-fcd50a9364d5" containerName="istio-proxy"
Oct 31 12:46:00.546517 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:00.516203    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="f957e25b-5bf0-4259-afbd-fcd50a9364d5" containerName="istio-proxy"
Oct 31 12:46:00.546517 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:00.516222    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d3acdbc-4d7e-4381-9d28-58426f9b3ab1" containerName="revoke-keys"
Oct 31 12:46:00.725066 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:00.695587    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-f22e7f-openshift-default-84f9758845-24j6m"]
Oct 31 12:46:00.725066 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:00.705737    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f22e7f-openshift-default-84f9758845-24j6m"
Oct 31 12:46:00.725066 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:00.712839    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1af894f4-5dbb-495a-b896-670ce365742e-workload-socket\") pod \"e2e-api-a-f22e7f-openshift-default-84f9758845-24j6m\" (UID: \"1af894f4-5dbb-495a-b896-670ce365742e\") " pod="openshift-ingress/e2e-api-a-f22e7f-openshift-default-84f9758845-24j6m"
Oct 31 12:46:00.725066 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:00.712879    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1af894f4-5dbb-495a-b896-670ce365742e-credential-socket\") pod \"e2e-api-a-f22e7f-openshift-default-84f9758845-24j6m\" (UID: \"1af894f4-5dbb-495a-b896-670ce365742e\") " pod="openshift-ingress/e2e-api-a-f22e7f-openshift-default-84f9758845-24j6m"
Oct 31 12:46:00.725066 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:00.712975    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1af894f4-5dbb-495a-b896-670ce365742e-istiod-ca-cert\") pod \"e2e-api-a-f22e7f-openshift-default-84f9758845-24j6m\" (UID: \"1af894f4-5dbb-495a-b896-670ce365742e\") " pod="openshift-ingress/e2e-api-a-f22e7f-openshift-default-84f9758845-24j6m"
Oct 31 12:46:00.725066 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:00.713008    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzxtw\" (UniqueName: \"kubernetes.io/projected/1af894f4-5dbb-495a-b896-670ce365742e-kube-api-access-pzxtw\") pod \"e2e-api-a-f22e7f-openshift-default-84f9758845-24j6m\" (UID: \"1af894f4-5dbb-495a-b896-670ce365742e\") " pod="openshift-ingress/e2e-api-a-f22e7f-openshift-default-84f9758845-24j6m"
Oct 31 12:46:00.725066 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:00.713119    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1af894f4-5dbb-495a-b896-670ce365742e-workload-certs\") pod \"e2e-api-a-f22e7f-openshift-default-84f9758845-24j6m\" (UID: \"1af894f4-5dbb-495a-b896-670ce365742e\") " pod="openshift-ingress/e2e-api-a-f22e7f-openshift-default-84f9758845-24j6m"
Oct 31 12:46:00.725066 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:00.713238    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1af894f4-5dbb-495a-b896-670ce365742e-istio-data\") pod \"e2e-api-a-f22e7f-openshift-default-84f9758845-24j6m\" (UID: \"1af894f4-5dbb-495a-b896-670ce365742e\") " pod="openshift-ingress/e2e-api-a-f22e7f-openshift-default-84f9758845-24j6m"
Oct 31 12:46:00.725066 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:00.713275    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1af894f4-5dbb-495a-b896-670ce365742e-istio-podinfo\") pod \"e2e-api-a-f22e7f-openshift-default-84f9758845-24j6m\" (UID: \"1af894f4-5dbb-495a-b896-670ce365742e\") " pod="openshift-ingress/e2e-api-a-f22e7f-openshift-default-84f9758845-24j6m"
Oct 31 12:46:00.725066 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:00.713324    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1af894f4-5dbb-495a-b896-670ce365742e-istio-token\") pod \"e2e-api-a-f22e7f-openshift-default-84f9758845-24j6m\" (UID: \"1af894f4-5dbb-495a-b896-670ce365742e\") " pod="openshift-ingress/e2e-api-a-f22e7f-openshift-default-84f9758845-24j6m"
Oct 31 12:46:00.725066 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:00.713589    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1af894f4-5dbb-495a-b896-670ce365742e-istio-envoy\") pod \"e2e-api-a-f22e7f-openshift-default-84f9758845-24j6m\" (UID: \"1af894f4-5dbb-495a-b896-670ce365742e\") " pod="openshift-ingress/e2e-api-a-f22e7f-openshift-default-84f9758845-24j6m"
Oct 31 12:46:00.755972 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:00.755054    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-f22e7f-openshift-default-dockercfg-f9nkp\""
Oct 31 12:46:00.822484 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:00.818660    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1af894f4-5dbb-495a-b896-670ce365742e-workload-socket\") pod \"e2e-api-a-f22e7f-openshift-default-84f9758845-24j6m\" (UID: \"1af894f4-5dbb-495a-b896-670ce365742e\") " pod="openshift-ingress/e2e-api-a-f22e7f-openshift-default-84f9758845-24j6m"
Oct 31 12:46:00.822484 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:00.818717    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1af894f4-5dbb-495a-b896-670ce365742e-credential-socket\") pod \"e2e-api-a-f22e7f-openshift-default-84f9758845-24j6m\" (UID: \"1af894f4-5dbb-495a-b896-670ce365742e\") " pod="openshift-ingress/e2e-api-a-f22e7f-openshift-default-84f9758845-24j6m"
Oct 31 12:46:00.822484 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:00.818808    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1af894f4-5dbb-495a-b896-670ce365742e-istiod-ca-cert\") pod \"e2e-api-a-f22e7f-openshift-default-84f9758845-24j6m\" (UID: \"1af894f4-5dbb-495a-b896-670ce365742e\") " pod="openshift-ingress/e2e-api-a-f22e7f-openshift-default-84f9758845-24j6m"
Oct 31 12:46:00.822484 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:00.818837    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzxtw\" (UniqueName: \"kubernetes.io/projected/1af894f4-5dbb-495a-b896-670ce365742e-kube-api-access-pzxtw\") pod \"e2e-api-a-f22e7f-openshift-default-84f9758845-24j6m\" (UID: \"1af894f4-5dbb-495a-b896-670ce365742e\") " pod="openshift-ingress/e2e-api-a-f22e7f-openshift-default-84f9758845-24j6m"
Oct 31 12:46:00.822484 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:00.818942    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1af894f4-5dbb-495a-b896-670ce365742e-workload-certs\") pod \"e2e-api-a-f22e7f-openshift-default-84f9758845-24j6m\" (UID: \"1af894f4-5dbb-495a-b896-670ce365742e\") " pod="openshift-ingress/e2e-api-a-f22e7f-openshift-default-84f9758845-24j6m"
Oct 31 12:46:00.822484 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:00.819042    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1af894f4-5dbb-495a-b896-670ce365742e-istio-data\") pod \"e2e-api-a-f22e7f-openshift-default-84f9758845-24j6m\" (UID: \"1af894f4-5dbb-495a-b896-670ce365742e\") " pod="openshift-ingress/e2e-api-a-f22e7f-openshift-default-84f9758845-24j6m"
Oct 31 12:46:00.822484 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:00.819081    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1af894f4-5dbb-495a-b896-670ce365742e-istio-podinfo\") pod \"e2e-api-a-f22e7f-openshift-default-84f9758845-24j6m\" (UID: \"1af894f4-5dbb-495a-b896-670ce365742e\") " pod="openshift-ingress/e2e-api-a-f22e7f-openshift-default-84f9758845-24j6m"
Oct 31 12:46:00.822484 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:00.819126    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1af894f4-5dbb-495a-b896-670ce365742e-istio-token\") pod \"e2e-api-a-f22e7f-openshift-default-84f9758845-24j6m\" (UID: \"1af894f4-5dbb-495a-b896-670ce365742e\") " pod="openshift-ingress/e2e-api-a-f22e7f-openshift-default-84f9758845-24j6m"
Oct 31 12:46:00.822484 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:00.819283    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1af894f4-5dbb-495a-b896-670ce365742e-istio-envoy\") pod \"e2e-api-a-f22e7f-openshift-default-84f9758845-24j6m\" (UID: \"1af894f4-5dbb-495a-b896-670ce365742e\") " pod="openshift-ingress/e2e-api-a-f22e7f-openshift-default-84f9758845-24j6m"
Oct 31 12:46:00.822484 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:00.822075    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1af894f4-5dbb-495a-b896-670ce365742e-workload-socket\") pod \"e2e-api-a-f22e7f-openshift-default-84f9758845-24j6m\" (UID: \"1af894f4-5dbb-495a-b896-670ce365742e\") " pod="openshift-ingress/e2e-api-a-f22e7f-openshift-default-84f9758845-24j6m"
Oct 31 12:46:00.852394 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:00.851601    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1af894f4-5dbb-495a-b896-670ce365742e-credential-socket\") pod \"e2e-api-a-f22e7f-openshift-default-84f9758845-24j6m\" (UID: \"1af894f4-5dbb-495a-b896-670ce365742e\") " pod="openshift-ingress/e2e-api-a-f22e7f-openshift-default-84f9758845-24j6m"
Oct 31 12:46:00.852830 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:00.852779    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1af894f4-5dbb-495a-b896-670ce365742e-istio-data\") pod \"e2e-api-a-f22e7f-openshift-default-84f9758845-24j6m\" (UID: \"1af894f4-5dbb-495a-b896-670ce365742e\") " pod="openshift-ingress/e2e-api-a-f22e7f-openshift-default-84f9758845-24j6m"
Oct 31 12:46:00.853217 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:00.853188    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1af894f4-5dbb-495a-b896-670ce365742e-istiod-ca-cert\") pod \"e2e-api-a-f22e7f-openshift-default-84f9758845-24j6m\" (UID: \"1af894f4-5dbb-495a-b896-670ce365742e\") " pod="openshift-ingress/e2e-api-a-f22e7f-openshift-default-84f9758845-24j6m"
Oct 31 12:46:00.853357 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:00.853205    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1af894f4-5dbb-495a-b896-670ce365742e-workload-certs\") pod \"e2e-api-a-f22e7f-openshift-default-84f9758845-24j6m\" (UID: \"1af894f4-5dbb-495a-b896-670ce365742e\") " pod="openshift-ingress/e2e-api-a-f22e7f-openshift-default-84f9758845-24j6m"
Oct 31 12:46:00.854520 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:00.854435    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1af894f4-5dbb-495a-b896-670ce365742e-istio-token\") pod \"e2e-api-a-f22e7f-openshift-default-84f9758845-24j6m\" (UID: \"1af894f4-5dbb-495a-b896-670ce365742e\") " pod="openshift-ingress/e2e-api-a-f22e7f-openshift-default-84f9758845-24j6m"
Oct 31 12:46:00.854800 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:00.854779    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1af894f4-5dbb-495a-b896-670ce365742e-istio-podinfo\") pod \"e2e-api-a-f22e7f-openshift-default-84f9758845-24j6m\" (UID: \"1af894f4-5dbb-495a-b896-670ce365742e\") " pod="openshift-ingress/e2e-api-a-f22e7f-openshift-default-84f9758845-24j6m"
Oct 31 12:46:00.856526 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:00.856472    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1af894f4-5dbb-495a-b896-670ce365742e-istio-envoy\") pod \"e2e-api-a-f22e7f-openshift-default-84f9758845-24j6m\" (UID: \"1af894f4-5dbb-495a-b896-670ce365742e\") " pod="openshift-ingress/e2e-api-a-f22e7f-openshift-default-84f9758845-24j6m"
Oct 31 12:46:00.882312 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:00.882266    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzxtw\" (UniqueName: \"kubernetes.io/projected/1af894f4-5dbb-495a-b896-670ce365742e-kube-api-access-pzxtw\") pod \"e2e-api-a-f22e7f-openshift-default-84f9758845-24j6m\" (UID: \"1af894f4-5dbb-495a-b896-670ce365742e\") " pod="openshift-ingress/e2e-api-a-f22e7f-openshift-default-84f9758845-24j6m"
Oct 31 12:46:01.096811 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:01.096725    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f22e7f-openshift-default-84f9758845-24j6m"
Oct 31 12:46:01.638733 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:46:01.638705    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1af894f4_5dbb_495a_b896_670ce365742e.slice/crio-d3c8dc1ad394e45c85cc49c0d32de83a2fcccd406b5f9d415126504ca9ecd1f8 WatchSource:0}: Error finding container d3c8dc1ad394e45c85cc49c0d32de83a2fcccd406b5f9d415126504ca9ecd1f8: Status 404 returned error can't find the container with id d3c8dc1ad394e45c85cc49c0d32de83a2fcccd406b5f9d415126504ca9ecd1f8
Oct 31 12:46:01.641296 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:01.641267    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-f22e7f-openshift-default-84f9758845-24j6m"]
Oct 31 12:46:01.642099 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:01.642064    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:46:01.642201 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:01.642145    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:46:01.642201 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:01.642189    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:46:01.819164 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:01.819119    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f22e7f-openshift-default-84f9758845-24j6m" event={"ID":"1af894f4-5dbb-495a-b896-670ce365742e","Type":"ContainerStarted","Data":"2f2f93ba9ba80135ad1f844874a72f1bfa14b5aeb5af28cf429848555803afdb"}
Oct 31 12:46:01.819390 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:01.819231    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f22e7f-openshift-default-84f9758845-24j6m" event={"ID":"1af894f4-5dbb-495a-b896-670ce365742e","Type":"ContainerStarted","Data":"d3c8dc1ad394e45c85cc49c0d32de83a2fcccd406b5f9d415126504ca9ecd1f8"}
Oct 31 12:46:01.853798 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:01.853729    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-f22e7f-openshift-default-84f9758845-24j6m" podStartSLOduration=1.8537077260000001 podStartE2EDuration="1.853707726s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:46:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:46:01.84573554 +0000 UTC m=+1124.948803264" watchObservedRunningTime="2026-10-31 12:46:01.853707726 +0000 UTC m=+1124.956775443"
Oct 31 12:46:02.102070 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:02.099840    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-f22e7f-openshift-default-84f9758845-24j6m"
Oct 31 12:46:02.102792 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:02.102767    5060 patch_prober.go:29] interesting pod/e2e-api-a-f22e7f-openshift-default-84f9758845-24j6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body=
Oct 31 12:46:02.102950 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:02.102924    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f22e7f-openshift-default-84f9758845-24j6m" podUID="1af894f4-5dbb-495a-b896-670ce365742e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused"
Oct 31 12:46:02.370047 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:02.369960    5060 patch_prober.go:29] interesting pod/e2e-shared-b-w4-8faad6-openshift-default-9ff4fdc49-fc7fk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body=
Oct 31 12:46:02.370047 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:02.370029    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w4-8faad6-openshift-default-9ff4fdc49-fc7fk" podUID="4d1ddd1b-8dc4-4706-911f-18376b4b2b68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused"
Oct 31 12:46:02.837547 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:02.837509    5060 generic.go:356] "Generic (PLEG): container finished" podID="4d1ddd1b-8dc4-4706-911f-18376b4b2b68" containerID="d7d9f76e56891d8d11d5191df4ed997977d680b6bdc8f71963300232ed4a9340" exitCode=0
Oct 31 12:46:02.837777 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:02.837548    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-8faad6-openshift-default-9ff4fdc49-fc7fk" event={"ID":"4d1ddd1b-8dc4-4706-911f-18376b4b2b68","Type":"ContainerDied","Data":"d7d9f76e56891d8d11d5191df4ed997977d680b6bdc8f71963300232ed4a9340"}
Oct 31 12:46:02.838207 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:02.838180    5060 scope.go:122] "RemoveContainer" containerID="d7d9f76e56891d8d11d5191df4ed997977d680b6bdc8f71963300232ed4a9340"
Oct 31 12:46:02.841108 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:02.841080    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:46:02.841203 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:02.841154    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:46:02.841203 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:02.841189    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:46:03.105012 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:03.104926    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-f22e7f-openshift-default-84f9758845-24j6m"
Oct 31 12:46:03.978605 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:03.973844    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-8faad6-openshift-default-9ff4fdc49-fc7fk" event={"ID":"4d1ddd1b-8dc4-4706-911f-18376b4b2b68","Type":"ContainerStarted","Data":"e1fd78abfe67144fdad2fa06e6ca38520bc79562bf27dcd46d0c0565a00f842c"}
Oct 31 12:46:03.978605 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:03.973911    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-f22e7f-openshift-default-84f9758845-24j6m"
Oct 31 12:46:04.024521 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:04.022878    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-f22e7f-openshift-default-84f9758845-24j6m"
Oct 31 12:46:04.723511 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:04.723108    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-8faad6-openshift-default-9ff4fdc49-fc7fk"
Oct 31 12:46:04.723511 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:04.723277    5060 patch_prober.go:29] interesting pod/e2e-shared-b-w4-8faad6-openshift-default-9ff4fdc49-fc7fk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body=
Oct 31 12:46:04.723511 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:04.723330    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-8faad6-openshift-default-9ff4fdc49-fc7fk" podUID="4d1ddd1b-8dc4-4706-911f-18376b4b2b68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused"
Oct 31 12:46:05.703097 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:05.703060    5060 patch_prober.go:29] interesting pod/e2e-shared-b-w4-8faad6-openshift-default-9ff4fdc49-fc7fk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body=
Oct 31 12:46:05.703266 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:05.703112    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-8faad6-openshift-default-9ff4fdc49-fc7fk" podUID="4d1ddd1b-8dc4-4706-911f-18376b4b2b68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused"
Oct 31 12:46:06.708531 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:06.708087    5060 patch_prober.go:29] interesting pod/e2e-shared-b-w4-8faad6-openshift-default-9ff4fdc49-fc7fk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body=
Oct 31 12:46:06.708531 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:06.708151    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-8faad6-openshift-default-9ff4fdc49-fc7fk" podUID="4d1ddd1b-8dc4-4706-911f-18376b4b2b68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused"
Oct 31 12:46:07.703203 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:07.703136    5060 patch_prober.go:29] interesting pod/e2e-shared-b-w4-8faad6-openshift-default-9ff4fdc49-fc7fk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body=
Oct 31 12:46:07.703459 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:07.703219    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-8faad6-openshift-default-9ff4fdc49-fc7fk" podUID="4d1ddd1b-8dc4-4706-911f-18376b4b2b68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused"
Oct 31 12:46:08.702619 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:08.702580    5060 patch_prober.go:29] interesting pod/e2e-shared-b-w4-8faad6-openshift-default-9ff4fdc49-fc7fk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body=
Oct 31 12:46:08.703125 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:08.702646    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-8faad6-openshift-default-9ff4fdc49-fc7fk" podUID="4d1ddd1b-8dc4-4706-911f-18376b4b2b68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused"
Oct 31 12:46:09.705516 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:09.703255    5060 patch_prober.go:29] interesting pod/e2e-shared-b-w4-8faad6-openshift-default-9ff4fdc49-fc7fk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body=
Oct 31 12:46:09.705516 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:09.703331    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-8faad6-openshift-default-9ff4fdc49-fc7fk" podUID="4d1ddd1b-8dc4-4706-911f-18376b4b2b68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused"
Oct 31 12:46:10.708845 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:10.708814    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-8faad6-openshift-default-9ff4fdc49-fc7fk"
Oct 31 12:46:10.709301 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:10.709281    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-8faad6-openshift-default-9ff4fdc49-fc7fk"
Oct 31 12:46:10.709995 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:10.709973    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-8faad6-openshift-default-9ff4fdc49-fc7fk"
Oct 31 12:46:15.032113 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:15.032069    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19b7740c-lj5z9"]
Oct 31 12:46:15.060939 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:15.060897    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19b7740c-lj5z9"]
Oct 31 12:46:17.059222 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:17.059183    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88044eb8-3340-459c-8b36-f71cc2d3e79a" path="/var/lib/kubelet/pods/88044eb8-3340-459c-8b36-f71cc2d3e79a/volumes"
Oct 31 12:46:18.753755 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:18.753715    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-711af2-openshift-default-6549b5d6f9-zdqxc"]
Oct 31 12:46:18.759943 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:18.759912    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="f957e25b-5bf0-4259-afbd-fcd50a9364d5" containerName="istio-proxy"
Oct 31 12:46:18.765463 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:18.765440    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-711af2-openshift-default-6549b5d6f9-zdqxc"
Oct 31 12:46:18.779409 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:18.779381    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-711af2-openshift-default-dockercfg-5hf8h\""
Oct 31 12:46:18.785643 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:18.785617    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-711af2-openshift-default-6549b5d6f9-zdqxc"]
Oct 31 12:46:18.903555 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:18.886959    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/440e8c3e-006e-4a6b-82d1-5fb80f5e26d1-istiod-ca-cert\") pod \"e2e-api-b-711af2-openshift-default-6549b5d6f9-zdqxc\" (UID: \"440e8c3e-006e-4a6b-82d1-5fb80f5e26d1\") " pod="openshift-ingress/e2e-api-b-711af2-openshift-default-6549b5d6f9-zdqxc"
Oct 31 12:46:18.903555 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:18.887014    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/440e8c3e-006e-4a6b-82d1-5fb80f5e26d1-workload-socket\") pod \"e2e-api-b-711af2-openshift-default-6549b5d6f9-zdqxc\" (UID: \"440e8c3e-006e-4a6b-82d1-5fb80f5e26d1\") " pod="openshift-ingress/e2e-api-b-711af2-openshift-default-6549b5d6f9-zdqxc"
Oct 31 12:46:18.903555 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:18.887077    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnpqj\" (UniqueName: \"kubernetes.io/projected/440e8c3e-006e-4a6b-82d1-5fb80f5e26d1-kube-api-access-mnpqj\") pod \"e2e-api-b-711af2-openshift-default-6549b5d6f9-zdqxc\" (UID: \"440e8c3e-006e-4a6b-82d1-5fb80f5e26d1\") " pod="openshift-ingress/e2e-api-b-711af2-openshift-default-6549b5d6f9-zdqxc"
Oct 31 12:46:18.903555 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:18.887116    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/440e8c3e-006e-4a6b-82d1-5fb80f5e26d1-workload-certs\") pod \"e2e-api-b-711af2-openshift-default-6549b5d6f9-zdqxc\" (UID: \"440e8c3e-006e-4a6b-82d1-5fb80f5e26d1\") " pod="openshift-ingress/e2e-api-b-711af2-openshift-default-6549b5d6f9-zdqxc"
Oct 31 12:46:18.903555 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:18.887141    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/440e8c3e-006e-4a6b-82d1-5fb80f5e26d1-istio-podinfo\") pod \"e2e-api-b-711af2-openshift-default-6549b5d6f9-zdqxc\" (UID: \"440e8c3e-006e-4a6b-82d1-5fb80f5e26d1\") " pod="openshift-ingress/e2e-api-b-711af2-openshift-default-6549b5d6f9-zdqxc"
Oct 31 12:46:18.903555 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:18.887222    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/440e8c3e-006e-4a6b-82d1-5fb80f5e26d1-istio-token\") pod \"e2e-api-b-711af2-openshift-default-6549b5d6f9-zdqxc\" (UID: \"440e8c3e-006e-4a6b-82d1-5fb80f5e26d1\") " pod="openshift-ingress/e2e-api-b-711af2-openshift-default-6549b5d6f9-zdqxc"
Oct 31 12:46:18.903555 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:18.887291    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/440e8c3e-006e-4a6b-82d1-5fb80f5e26d1-istio-data\") pod \"e2e-api-b-711af2-openshift-default-6549b5d6f9-zdqxc\" (UID: \"440e8c3e-006e-4a6b-82d1-5fb80f5e26d1\") " pod="openshift-ingress/e2e-api-b-711af2-openshift-default-6549b5d6f9-zdqxc"
Oct 31 12:46:18.903555 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:18.887347    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/440e8c3e-006e-4a6b-82d1-5fb80f5e26d1-credential-socket\") pod \"e2e-api-b-711af2-openshift-default-6549b5d6f9-zdqxc\" (UID: \"440e8c3e-006e-4a6b-82d1-5fb80f5e26d1\") " pod="openshift-ingress/e2e-api-b-711af2-openshift-default-6549b5d6f9-zdqxc"
Oct 31 12:46:18.903555 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:18.887450    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/440e8c3e-006e-4a6b-82d1-5fb80f5e26d1-istio-envoy\") pod \"e2e-api-b-711af2-openshift-default-6549b5d6f9-zdqxc\" (UID: \"440e8c3e-006e-4a6b-82d1-5fb80f5e26d1\") " pod="openshift-ingress/e2e-api-b-711af2-openshift-default-6549b5d6f9-zdqxc"
Oct 31 12:46:19.031572 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:19.005236    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/440e8c3e-006e-4a6b-82d1-5fb80f5e26d1-istio-envoy\") pod \"e2e-api-b-711af2-openshift-default-6549b5d6f9-zdqxc\" (UID: \"440e8c3e-006e-4a6b-82d1-5fb80f5e26d1\") " pod="openshift-ingress/e2e-api-b-711af2-openshift-default-6549b5d6f9-zdqxc"
Oct 31 12:46:19.034914 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:19.034875    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/440e8c3e-006e-4a6b-82d1-5fb80f5e26d1-istiod-ca-cert\") pod \"e2e-api-b-711af2-openshift-default-6549b5d6f9-zdqxc\" (UID: \"440e8c3e-006e-4a6b-82d1-5fb80f5e26d1\") " pod="openshift-ingress/e2e-api-b-711af2-openshift-default-6549b5d6f9-zdqxc"
Oct 31 12:46:19.038611 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:19.038574    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/440e8c3e-006e-4a6b-82d1-5fb80f5e26d1-workload-socket\") pod \"e2e-api-b-711af2-openshift-default-6549b5d6f9-zdqxc\" (UID: \"440e8c3e-006e-4a6b-82d1-5fb80f5e26d1\") " pod="openshift-ingress/e2e-api-b-711af2-openshift-default-6549b5d6f9-zdqxc"
Oct 31 12:46:19.094682 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:19.094637    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnpqj\" (UniqueName: \"kubernetes.io/projected/440e8c3e-006e-4a6b-82d1-5fb80f5e26d1-kube-api-access-mnpqj\") pod \"e2e-api-b-711af2-openshift-default-6549b5d6f9-zdqxc\" (UID: \"440e8c3e-006e-4a6b-82d1-5fb80f5e26d1\") " pod="openshift-ingress/e2e-api-b-711af2-openshift-default-6549b5d6f9-zdqxc"
Oct 31 12:46:19.116801 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:19.116771    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/440e8c3e-006e-4a6b-82d1-5fb80f5e26d1-workload-certs\") pod \"e2e-api-b-711af2-openshift-default-6549b5d6f9-zdqxc\" (UID: \"440e8c3e-006e-4a6b-82d1-5fb80f5e26d1\") " pod="openshift-ingress/e2e-api-b-711af2-openshift-default-6549b5d6f9-zdqxc"
Oct 31 12:46:19.116977 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:19.116963    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/440e8c3e-006e-4a6b-82d1-5fb80f5e26d1-istio-podinfo\") pod \"e2e-api-b-711af2-openshift-default-6549b5d6f9-zdqxc\" (UID: \"440e8c3e-006e-4a6b-82d1-5fb80f5e26d1\") " pod="openshift-ingress/e2e-api-b-711af2-openshift-default-6549b5d6f9-zdqxc"
Oct 31 12:46:19.141519 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:19.117146    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/440e8c3e-006e-4a6b-82d1-5fb80f5e26d1-istio-token\") pod \"e2e-api-b-711af2-openshift-default-6549b5d6f9-zdqxc\" (UID: \"440e8c3e-006e-4a6b-82d1-5fb80f5e26d1\") " pod="openshift-ingress/e2e-api-b-711af2-openshift-default-6549b5d6f9-zdqxc"
Oct 31 12:46:19.265377 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:19.265340    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/440e8c3e-006e-4a6b-82d1-5fb80f5e26d1-istio-data\") pod \"e2e-api-b-711af2-openshift-default-6549b5d6f9-zdqxc\" (UID: \"440e8c3e-006e-4a6b-82d1-5fb80f5e26d1\") " pod="openshift-ingress/e2e-api-b-711af2-openshift-default-6549b5d6f9-zdqxc"
Oct 31 12:46:19.266141 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:19.266112    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/440e8c3e-006e-4a6b-82d1-5fb80f5e26d1-credential-socket\") pod \"e2e-api-b-711af2-openshift-default-6549b5d6f9-zdqxc\" (UID: \"440e8c3e-006e-4a6b-82d1-5fb80f5e26d1\") " pod="openshift-ingress/e2e-api-b-711af2-openshift-default-6549b5d6f9-zdqxc"
Oct 31 12:46:19.266813 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:19.063362    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/440e8c3e-006e-4a6b-82d1-5fb80f5e26d1-istiod-ca-cert\") pod \"e2e-api-b-711af2-openshift-default-6549b5d6f9-zdqxc\" (UID: \"440e8c3e-006e-4a6b-82d1-5fb80f5e26d1\") " pod="openshift-ingress/e2e-api-b-711af2-openshift-default-6549b5d6f9-zdqxc"
Oct 31 12:46:19.267105 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:19.068571    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/440e8c3e-006e-4a6b-82d1-5fb80f5e26d1-workload-socket\") pod \"e2e-api-b-711af2-openshift-default-6549b5d6f9-zdqxc\" (UID: \"440e8c3e-006e-4a6b-82d1-5fb80f5e26d1\") " pod="openshift-ingress/e2e-api-b-711af2-openshift-default-6549b5d6f9-zdqxc"
Oct 31 12:46:19.267628 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:19.267606    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/440e8c3e-006e-4a6b-82d1-5fb80f5e26d1-istio-data\") pod \"e2e-api-b-711af2-openshift-default-6549b5d6f9-zdqxc\" (UID: \"440e8c3e-006e-4a6b-82d1-5fb80f5e26d1\") " pod="openshift-ingress/e2e-api-b-711af2-openshift-default-6549b5d6f9-zdqxc"
Oct 31 12:46:19.269149 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:19.133216    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/440e8c3e-006e-4a6b-82d1-5fb80f5e26d1-workload-certs\") pod \"e2e-api-b-711af2-openshift-default-6549b5d6f9-zdqxc\" (UID: \"440e8c3e-006e-4a6b-82d1-5fb80f5e26d1\") " pod="openshift-ingress/e2e-api-b-711af2-openshift-default-6549b5d6f9-zdqxc"
Oct 31 12:46:19.269613 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:19.269592    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/440e8c3e-006e-4a6b-82d1-5fb80f5e26d1-credential-socket\") pod \"e2e-api-b-711af2-openshift-default-6549b5d6f9-zdqxc\" (UID: \"440e8c3e-006e-4a6b-82d1-5fb80f5e26d1\") " pod="openshift-ingress/e2e-api-b-711af2-openshift-default-6549b5d6f9-zdqxc"
Oct 31 12:46:19.272356 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:19.272333    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/440e8c3e-006e-4a6b-82d1-5fb80f5e26d1-istio-envoy\") pod \"e2e-api-b-711af2-openshift-default-6549b5d6f9-zdqxc\" (UID: \"440e8c3e-006e-4a6b-82d1-5fb80f5e26d1\") " pod="openshift-ingress/e2e-api-b-711af2-openshift-default-6549b5d6f9-zdqxc"
Oct 31 12:46:19.274940 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:19.274454    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/440e8c3e-006e-4a6b-82d1-5fb80f5e26d1-istio-podinfo\") pod \"e2e-api-b-711af2-openshift-default-6549b5d6f9-zdqxc\" (UID: \"440e8c3e-006e-4a6b-82d1-5fb80f5e26d1\") " pod="openshift-ingress/e2e-api-b-711af2-openshift-default-6549b5d6f9-zdqxc"
Oct 31 12:46:19.354586 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:19.353933    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/440e8c3e-006e-4a6b-82d1-5fb80f5e26d1-istio-token\") pod \"e2e-api-b-711af2-openshift-default-6549b5d6f9-zdqxc\" (UID: \"440e8c3e-006e-4a6b-82d1-5fb80f5e26d1\") " pod="openshift-ingress/e2e-api-b-711af2-openshift-default-6549b5d6f9-zdqxc"
Oct 31 12:46:19.355679 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:19.355649    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnpqj\" (UniqueName: \"kubernetes.io/projected/440e8c3e-006e-4a6b-82d1-5fb80f5e26d1-kube-api-access-mnpqj\") pod \"e2e-api-b-711af2-openshift-default-6549b5d6f9-zdqxc\" (UID: \"440e8c3e-006e-4a6b-82d1-5fb80f5e26d1\") " pod="openshift-ingress/e2e-api-b-711af2-openshift-default-6549b5d6f9-zdqxc"
Oct 31 12:46:19.389142 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:19.389109    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-711af2-openshift-default-6549b5d6f9-zdqxc"
Oct 31 12:46:19.402343 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:19.402313    5060 scope.go:122] "RemoveContainer" containerID="eb11c6394e5500487e31ba4aec826f0dabf80bc3cf568948907ae57effedb6a2"
Oct 31 12:46:19.420049 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:19.420025    5060 scope.go:122] "RemoveContainer" containerID="4433fc46a6142233b4f8c08806a0b6c4e9c3c05946e4b8537f34ef2128958efc"
Oct 31 12:46:19.436961 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:19.436934    5060 scope.go:122] "RemoveContainer" containerID="ecc196bdfebc5a6981bf5541dce205df3e0e9d385930dbada1d73df06c858ddb"
Oct 31 12:46:19.449461 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:19.449442    5060 scope.go:122] "RemoveContainer" containerID="c17f4a0a1e6158224d7c2f0f94b95f72dc790a1edf158227943173a0c456f449"
Oct 31 12:46:19.470544 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:19.469525    5060 scope.go:122] "RemoveContainer" containerID="6cd6703dd1c2cad435ce7e7555cabf53dc58af7c4777b656b6a9816458d77a52"
Oct 31 12:46:19.491201 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:19.491175    5060 scope.go:122] "RemoveContainer" containerID="efc110ed18fc5e62df112f661b077fcb7d25c830d7c26bb00fc648c666d99570"
Oct 31 12:46:19.533662 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:19.533583    5060 scope.go:122] "RemoveContainer" containerID="93ef9ce56f011215080b2b1c3b037c0dfc2d437e16e9f0b44496e0b65f614e5d"
Oct 31 12:46:19.547211 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:19.547124    5060 scope.go:122] "RemoveContainer" containerID="60adbef124b5b188954d00615a395b2f99d5f6c748acbd35b0dbd48d211d40b7"
Oct 31 12:46:19.572294 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:19.572060    5060 scope.go:122] "RemoveContainer" containerID="cf8416d48a62293528b93feed181db38360846c4012c673073664232cf39285d"
Oct 31 12:46:19.598820 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:19.598694    5060 scope.go:122] "RemoveContainer" containerID="11f365ff9233dad78c06d763b6d868b1fcc373c8d03beb179875edcaf8e8a23a"
Oct 31 12:46:19.613004 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:19.612947    5060 scope.go:122] "RemoveContainer" containerID="f9dd97d39421e34001bc73f6bedf1b26da6c1ada560389d20c81b52a07da4a6e"
Oct 31 12:46:19.624177 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:19.624155    5060 scope.go:122] "RemoveContainer" containerID="8311b665530e2a7cca9867d0458a473402c5caeec7a027460d7b0cf1a6d09422"
Oct 31 12:46:19.635273 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:19.635225    5060 scope.go:122] "RemoveContainer" containerID="c00c16c2d00eaf1343b63e6f816406d706d9049151d496c49ffb5f5aeed849e9"
Oct 31 12:46:19.649285 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:19.649230    5060 scope.go:122] "RemoveContainer" containerID="82822d00310366795621427ef47d398710cb2fbd3f89567ac4ae2abd0940060e"
Oct 31 12:46:19.689522 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:19.678707    5060 scope.go:122] "RemoveContainer" containerID="ba1fe25266ab8f7ab9ac332357a5de8f0049823d424b9d00dd2d34f896a52ddf"
Oct 31 12:46:19.786730 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:19.785272    5060 scope.go:122] "RemoveContainer" containerID="958a9da68ee1eaa86eb1aa54cd25322fe09534702a7b3935b0868ccb6d47fbe8"
Oct 31 12:46:19.820603 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:19.820270    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-711af2-openshift-default-6549b5d6f9-zdqxc"]
Oct 31 12:46:19.828084 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:19.827781    5060 scope.go:122] "RemoveContainer" containerID="f2ec50798bf391a0703c14078d553b13d6e5c5838f4fdd9f6b3faf8d75860f29"
Oct 31 12:46:19.829371 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:46:19.829239    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod440e8c3e_006e_4a6b_82d1_5fb80f5e26d1.slice/crio-4615eb5c1733c4e9bbf25ec01476fd43a6ecbc7d66a3c176acb4019cd39e7417 WatchSource:0}: Error finding container 4615eb5c1733c4e9bbf25ec01476fd43a6ecbc7d66a3c176acb4019cd39e7417: Status 404 returned error can't find the container with id 4615eb5c1733c4e9bbf25ec01476fd43a6ecbc7d66a3c176acb4019cd39e7417
Oct 31 12:46:19.834311 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:19.833227    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:46:19.834311 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:19.833314    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:46:19.834311 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:19.833361    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:46:19.850704 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:19.850679    5060 scope.go:122] "RemoveContainer" containerID="89ef3c27df05aa7946878decc263a3f4b9de33523c2259d80108c4266465fa79"
Oct 31 12:46:20.031117 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:20.031056    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff5915db-km2qc"]
Oct 31 12:46:20.041858 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:20.041824    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff5915db-km2qc"]
Oct 31 12:46:20.374262 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:20.374224    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-711af2-openshift-default-6549b5d6f9-zdqxc" event={"ID":"440e8c3e-006e-4a6b-82d1-5fb80f5e26d1","Type":"ContainerStarted","Data":"47e7f181c25f923c5479a64b7786d08ed6b5f4e7a0e117ea035685d1dd2d25eb"}
Oct 31 12:46:20.374463 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:20.374286    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-711af2-openshift-default-6549b5d6f9-zdqxc" event={"ID":"440e8c3e-006e-4a6b-82d1-5fb80f5e26d1","Type":"ContainerStarted","Data":"4615eb5c1733c4e9bbf25ec01476fd43a6ecbc7d66a3c176acb4019cd39e7417"}
Oct 31 12:46:20.389659 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:20.389578    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-711af2-openshift-default-6549b5d6f9-zdqxc"
Oct 31 12:46:20.395995 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:20.395963    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-711af2-openshift-default-6549b5d6f9-zdqxc"
Oct 31 12:46:20.405677 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:20.405627    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-711af2-openshift-default-6549b5d6f9-zdqxc" podStartSLOduration=2.405612804 podStartE2EDuration="2.405612804s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:46:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:46:20.400031895 +0000 UTC m=+1143.503099609" watchObservedRunningTime="2026-10-31 12:46:20.405612804 +0000 UTC m=+1143.508680515"
Oct 31 12:46:21.050944 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:21.050902    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36cecbf2-83f6-42d3-9846-ac361a13e427" path="/var/lib/kubelet/pods/36cecbf2-83f6-42d3-9846-ac361a13e427/volumes"
Oct 31 12:46:21.390777 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:21.390693    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-711af2-openshift-default-6549b5d6f9-zdqxc"
Oct 31 12:46:21.391958 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:21.391929    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-711af2-openshift-default-6549b5d6f9-zdqxc"
Oct 31 12:46:28.036860 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:28.035749    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75668bd6-vwl6q"]
Oct 31 12:46:28.048594 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:28.048529    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75668bd6-vwl6q"]
Oct 31 12:46:29.031563 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:29.031518    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96d6e803-8ckml"]
Oct 31 12:46:29.118510 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:29.100376    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1b6f29e-44fd-49de-a88e-ae6cf2efcfef" path="/var/lib/kubelet/pods/b1b6f29e-44fd-49de-a88e-ae6cf2efcfef/volumes"
Oct 31 12:46:29.118510 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:29.111269    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96d6e803-8ckml"]
Oct 31 12:46:30.277472 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:46:30.275937    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7phb" containerName="extractor"
Oct 31 12:46:30.637571 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:30.636107    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-bb3cf5-openshift-default-5fc978478f-mw7qm"]
Oct 31 12:46:30.795019 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:30.793689    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-bb3cf5-openshift-default-5fc978478f-mw7qm"
Oct 31 12:46:30.888673 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:30.835928    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-bb3cf5-openshift-default-dockercfg-p2ft8\""
Oct 31 12:46:31.093516 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:31.071773    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-bb3cf5-openshift-default-5fc978478f-mw7qm"]
Oct 31 12:46:31.239740 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:31.230801    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9cb64fe-d6b4-439d-bc3a-8598b7cd512e-workload-certs\") pod \"e2e-api-b-bb3cf5-openshift-default-5fc978478f-mw7qm\" (UID: \"d9cb64fe-d6b4-439d-bc3a-8598b7cd512e\") " pod="openshift-ingress/e2e-api-b-bb3cf5-openshift-default-5fc978478f-mw7qm"
Oct 31 12:46:31.239740 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:31.230900    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9cb64fe-d6b4-439d-bc3a-8598b7cd512e-istio-podinfo\") pod \"e2e-api-b-bb3cf5-openshift-default-5fc978478f-mw7qm\" (UID: \"d9cb64fe-d6b4-439d-bc3a-8598b7cd512e\") " pod="openshift-ingress/e2e-api-b-bb3cf5-openshift-default-5fc978478f-mw7qm"
Oct 31 12:46:31.239740 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:31.230947    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9cb64fe-d6b4-439d-bc3a-8598b7cd512e-workload-socket\") pod \"e2e-api-b-bb3cf5-openshift-default-5fc978478f-mw7qm\" (UID: \"d9cb64fe-d6b4-439d-bc3a-8598b7cd512e\") " pod="openshift-ingress/e2e-api-b-bb3cf5-openshift-default-5fc978478f-mw7qm"
Oct 31 12:46:31.239740 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:31.231029    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9cb64fe-d6b4-439d-bc3a-8598b7cd512e-istio-token\") pod \"e2e-api-b-bb3cf5-openshift-default-5fc978478f-mw7qm\" (UID: \"d9cb64fe-d6b4-439d-bc3a-8598b7cd512e\") " pod="openshift-ingress/e2e-api-b-bb3cf5-openshift-default-5fc978478f-mw7qm"
Oct 31 12:46:31.239740 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:31.231250    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bdk9\" (UniqueName: \"kubernetes.io/projected/d9cb64fe-d6b4-439d-bc3a-8598b7cd512e-kube-api-access-6bdk9\") pod \"e2e-api-b-bb3cf5-openshift-default-5fc978478f-mw7qm\" (UID: \"d9cb64fe-d6b4-439d-bc3a-8598b7cd512e\") " pod="openshift-ingress/e2e-api-b-bb3cf5-openshift-default-5fc978478f-mw7qm"
Oct 31 12:46:31.239740 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:31.231286    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9cb64fe-d6b4-439d-bc3a-8598b7cd512e-credential-socket\") pod \"e2e-api-b-bb3cf5-openshift-default-5fc978478f-mw7qm\" (UID: \"d9cb64fe-d6b4-439d-bc3a-8598b7cd512e\") " pod="openshift-ingress/e2e-api-b-bb3cf5-openshift-default-5fc978478f-mw7qm"
Oct 31 12:46:31.239740 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:31.231344    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9cb64fe-d6b4-439d-bc3a-8598b7cd512e-istio-data\") pod \"e2e-api-b-bb3cf5-openshift-default-5fc978478f-mw7qm\" (UID: \"d9cb64fe-d6b4-439d-bc3a-8598b7cd512e\") " pod="openshift-ingress/e2e-api-b-bb3cf5-openshift-default-5fc978478f-mw7qm"
Oct 31 12:46:31.239740 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:31.195825    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/simulator-w4-c67de2-kserve-868679cf9b-gqjbx" podUID="e104d7da-1fb2-4c99-a480-b4ae84c4f273" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:46:31.239740 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:31.231713    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9cb64fe-d6b4-439d-bc3a-8598b7cd512e-istio-envoy\") pod \"e2e-api-b-bb3cf5-openshift-default-5fc978478f-mw7qm\" (UID: \"d9cb64fe-d6b4-439d-bc3a-8598b7cd512e\") " pod="openshift-ingress/e2e-api-b-bb3cf5-openshift-default-5fc978478f-mw7qm"
Oct 31 12:46:31.523569 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:31.493564    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-2-w5-b25d3b-kserve-76f798d59f-vvzhg" podUID="394ed76c-84ac-4612-9acb-a7e36cb68d89" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": context deadline exceeded"
Oct 31 12:46:31.796784 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:31.796702    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48fd0a20-b1ec-47cd-9f68-4638965a5acc" path="/var/lib/kubelet/pods/48fd0a20-b1ec-47cd-9f68-4638965a5acc/volumes"
Oct 31 12:46:32.212581 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:32.187820    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9cb64fe-d6b4-439d-bc3a-8598b7cd512e-istiod-ca-cert\") pod \"e2e-api-b-bb3cf5-openshift-default-5fc978478f-mw7qm\" (UID: \"d9cb64fe-d6b4-439d-bc3a-8598b7cd512e\") " pod="openshift-ingress/e2e-api-b-bb3cf5-openshift-default-5fc978478f-mw7qm"
Oct 31 12:46:32.212581 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:32.197886    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x2qzv" podUID="c247ed2a-aa90-424b-9b7e-4f2936d67e3f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:46:32.290612 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:32.290571    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9cb64fe-d6b4-439d-bc3a-8598b7cd512e-istiod-ca-cert\") pod \"e2e-api-b-bb3cf5-openshift-default-5fc978478f-mw7qm\" (UID: \"d9cb64fe-d6b4-439d-bc3a-8598b7cd512e\") " pod="openshift-ingress/e2e-api-b-bb3cf5-openshift-default-5fc978478f-mw7qm"
Oct 31 12:46:32.290921 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:32.290900    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9cb64fe-d6b4-439d-bc3a-8598b7cd512e-workload-certs\") pod \"e2e-api-b-bb3cf5-openshift-default-5fc978478f-mw7qm\" (UID: \"d9cb64fe-d6b4-439d-bc3a-8598b7cd512e\") " pod="openshift-ingress/e2e-api-b-bb3cf5-openshift-default-5fc978478f-mw7qm"
Oct 31 12:46:32.291104 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:32.291084    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9cb64fe-d6b4-439d-bc3a-8598b7cd512e-istio-podinfo\") pod \"e2e-api-b-bb3cf5-openshift-default-5fc978478f-mw7qm\" (UID: \"d9cb64fe-d6b4-439d-bc3a-8598b7cd512e\") " pod="openshift-ingress/e2e-api-b-bb3cf5-openshift-default-5fc978478f-mw7qm"
Oct 31 12:46:32.291239 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:32.291223    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9cb64fe-d6b4-439d-bc3a-8598b7cd512e-workload-socket\") pod \"e2e-api-b-bb3cf5-openshift-default-5fc978478f-mw7qm\" (UID: \"d9cb64fe-d6b4-439d-bc3a-8598b7cd512e\") " pod="openshift-ingress/e2e-api-b-bb3cf5-openshift-default-5fc978478f-mw7qm"
Oct 31 12:46:32.291386 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:32.291367    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9cb64fe-d6b4-439d-bc3a-8598b7cd512e-istio-token\") pod \"e2e-api-b-bb3cf5-openshift-default-5fc978478f-mw7qm\" (UID: \"d9cb64fe-d6b4-439d-bc3a-8598b7cd512e\") " pod="openshift-ingress/e2e-api-b-bb3cf5-openshift-default-5fc978478f-mw7qm"
Oct 31 12:46:32.291634 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:32.291610    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bdk9\" (UniqueName: \"kubernetes.io/projected/d9cb64fe-d6b4-439d-bc3a-8598b7cd512e-kube-api-access-6bdk9\") pod \"e2e-api-b-bb3cf5-openshift-default-5fc978478f-mw7qm\" (UID: \"d9cb64fe-d6b4-439d-bc3a-8598b7cd512e\") " pod="openshift-ingress/e2e-api-b-bb3cf5-openshift-default-5fc978478f-mw7qm"
Oct 31 12:46:32.291795 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:32.291777    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9cb64fe-d6b4-439d-bc3a-8598b7cd512e-credential-socket\") pod \"e2e-api-b-bb3cf5-openshift-default-5fc978478f-mw7qm\" (UID: \"d9cb64fe-d6b4-439d-bc3a-8598b7cd512e\") " pod="openshift-ingress/e2e-api-b-bb3cf5-openshift-default-5fc978478f-mw7qm"
Oct 31 12:46:32.291951 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:32.291932    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9cb64fe-d6b4-439d-bc3a-8598b7cd512e-istio-data\") pod \"e2e-api-b-bb3cf5-openshift-default-5fc978478f-mw7qm\" (UID: \"d9cb64fe-d6b4-439d-bc3a-8598b7cd512e\") " pod="openshift-ingress/e2e-api-b-bb3cf5-openshift-default-5fc978478f-mw7qm"
Oct 31 12:46:32.318520 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:32.314475    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9cb64fe-d6b4-439d-bc3a-8598b7cd512e-istio-envoy\") pod \"e2e-api-b-bb3cf5-openshift-default-5fc978478f-mw7qm\" (UID: \"d9cb64fe-d6b4-439d-bc3a-8598b7cd512e\") " pod="openshift-ingress/e2e-api-b-bb3cf5-openshift-default-5fc978478f-mw7qm"
Oct 31 12:46:32.318520 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:32.295411    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9cb64fe-d6b4-439d-bc3a-8598b7cd512e-istiod-ca-cert\") pod \"e2e-api-b-bb3cf5-openshift-default-5fc978478f-mw7qm\" (UID: \"d9cb64fe-d6b4-439d-bc3a-8598b7cd512e\") " pod="openshift-ingress/e2e-api-b-bb3cf5-openshift-default-5fc978478f-mw7qm"
Oct 31 12:46:32.318520 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:32.296835    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9cb64fe-d6b4-439d-bc3a-8598b7cd512e-istio-data\") pod \"e2e-api-b-bb3cf5-openshift-default-5fc978478f-mw7qm\" (UID: \"d9cb64fe-d6b4-439d-bc3a-8598b7cd512e\") " pod="openshift-ingress/e2e-api-b-bb3cf5-openshift-default-5fc978478f-mw7qm"
Oct 31 12:46:32.318520 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:32.297720    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9cb64fe-d6b4-439d-bc3a-8598b7cd512e-credential-socket\") pod \"e2e-api-b-bb3cf5-openshift-default-5fc978478f-mw7qm\" (UID: \"d9cb64fe-d6b4-439d-bc3a-8598b7cd512e\") " pod="openshift-ingress/e2e-api-b-bb3cf5-openshift-default-5fc978478f-mw7qm"
Oct 31 12:46:32.318520 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:32.314286    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9cb64fe-d6b4-439d-bc3a-8598b7cd512e-workload-certs\") pod \"e2e-api-b-bb3cf5-openshift-default-5fc978478f-mw7qm\" (UID: \"d9cb64fe-d6b4-439d-bc3a-8598b7cd512e\") " pod="openshift-ingress/e2e-api-b-bb3cf5-openshift-default-5fc978478f-mw7qm"
Oct 31 12:46:32.318520 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:32.294448    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9cb64fe-d6b4-439d-bc3a-8598b7cd512e-workload-socket\") pod \"e2e-api-b-bb3cf5-openshift-default-5fc978478f-mw7qm\" (UID: \"d9cb64fe-d6b4-439d-bc3a-8598b7cd512e\") " pod="openshift-ingress/e2e-api-b-bb3cf5-openshift-default-5fc978478f-mw7qm"
Oct 31 12:46:32.367513 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:32.363615    5060 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x2qzv" podUID="c247ed2a-aa90-424b-9b7e-4f2936d67e3f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:46:32.375328 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:32.375188    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9cb64fe-d6b4-439d-bc3a-8598b7cd512e-istio-envoy\") pod \"e2e-api-b-bb3cf5-openshift-default-5fc978478f-mw7qm\" (UID: \"d9cb64fe-d6b4-439d-bc3a-8598b7cd512e\") " pod="openshift-ingress/e2e-api-b-bb3cf5-openshift-default-5fc978478f-mw7qm"
Oct 31 12:46:32.434515 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:32.387257    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9cb64fe-d6b4-439d-bc3a-8598b7cd512e-istio-token\") pod \"e2e-api-b-bb3cf5-openshift-default-5fc978478f-mw7qm\" (UID: \"d9cb64fe-d6b4-439d-bc3a-8598b7cd512e\") " pod="openshift-ingress/e2e-api-b-bb3cf5-openshift-default-5fc978478f-mw7qm"
Oct 31 12:46:32.529014 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:32.490602    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9cb64fe-d6b4-439d-bc3a-8598b7cd512e-istio-podinfo\") pod \"e2e-api-b-bb3cf5-openshift-default-5fc978478f-mw7qm\" (UID: \"d9cb64fe-d6b4-439d-bc3a-8598b7cd512e\") " pod="openshift-ingress/e2e-api-b-bb3cf5-openshift-default-5fc978478f-mw7qm"
Oct 31 12:46:32.529014 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:32.491258    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bdk9\" (UniqueName: \"kubernetes.io/projected/d9cb64fe-d6b4-439d-bc3a-8598b7cd512e-kube-api-access-6bdk9\") pod \"e2e-api-b-bb3cf5-openshift-default-5fc978478f-mw7qm\" (UID: \"d9cb64fe-d6b4-439d-bc3a-8598b7cd512e\") " pod="openshift-ingress/e2e-api-b-bb3cf5-openshift-default-5fc978478f-mw7qm"
Oct 31 12:46:32.638939 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:32.638880    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-bb3cf5-openshift-default-5fc978478f-mw7qm"
Oct 31 12:46:33.512809 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:46:33.512769    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9cb64fe_d6b4_439d_bc3a_8598b7cd512e.slice/crio-3dcf1e815720a92211f45f85fd88f04a245b38d399fa98bed0e0a5dcff3d1d5b WatchSource:0}: Error finding container 3dcf1e815720a92211f45f85fd88f04a245b38d399fa98bed0e0a5dcff3d1d5b: Status 404 returned error can't find the container with id 3dcf1e815720a92211f45f85fd88f04a245b38d399fa98bed0e0a5dcff3d1d5b
Oct 31 12:46:33.513314 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:33.513284    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-bb3cf5-openshift-default-5fc978478f-mw7qm"]
Oct 31 12:46:33.516066 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:33.516024    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:46:33.516184 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:33.516110    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:46:33.516184 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:33.516151    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:46:33.944583 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:33.942197    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-bb3cf5-openshift-default-5fc978478f-mw7qm" event={"ID":"d9cb64fe-d6b4-439d-bc3a-8598b7cd512e","Type":"ContainerStarted","Data":"15e608e4b881d6dc2804ab6cfc24e27e38e7e9798db2a22b07b5bdd67fffaf78"}
Oct 31 12:46:33.944583 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:33.942266    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-bb3cf5-openshift-default-5fc978478f-mw7qm" event={"ID":"d9cb64fe-d6b4-439d-bc3a-8598b7cd512e","Type":"ContainerStarted","Data":"3dcf1e815720a92211f45f85fd88f04a245b38d399fa98bed0e0a5dcff3d1d5b"}
Oct 31 12:46:33.995987 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:33.995744    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-bb3cf5-openshift-default-5fc978478f-mw7qm" podStartSLOduration=3.995723925 podStartE2EDuration="3.995723925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:46:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:46:33.972997807 +0000 UTC m=+1157.076065521" watchObservedRunningTime="2026-10-31 12:46:33.995723925 +0000 UTC m=+1157.098791638"
Oct 31 12:46:34.667522 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:34.663569    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-bb3cf5-openshift-default-5fc978478f-mw7qm"
Oct 31 12:46:34.730855 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:34.728863    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-bb3cf5-openshift-default-5fc978478f-mw7qm"
Oct 31 12:46:34.953370 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:34.953280    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-bb3cf5-openshift-default-5fc978478f-mw7qm"
Oct 31 12:46:34.954397 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:34.954372    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-bb3cf5-openshift-default-5fc978478f-mw7qm"
Oct 31 12:46:37.338511 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:37.327466    5060 patch_prober.go:29] interesting pod/e2e-api-a-5d310d-openshift-default-7c8bb48c95-n2tkf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 12:46:37.338511 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:37.327590    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-5d310d-openshift-default-7c8bb48c95-n2tkf" podUID="3e880a6e-3d1c-42e4-a6f3-a7a22c2cc35d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:46:38.139288 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:38.139251    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da149b49-bhdgd"]
Oct 31 12:46:38.230943 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:38.230847    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da149b49-bhdgd"]
Oct 31 12:46:39.052921 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:39.052884    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d8fa218-8592-4a2a-8efb-db09efab23ab" path="/var/lib/kubelet/pods/8d8fa218-8592-4a2a-8efb-db09efab23ab/volumes"
Oct 31 12:46:42.119586 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:42.119546    5060 generic.go:356] "Generic (PLEG): container finished" podID="3e880a6e-3d1c-42e4-a6f3-a7a22c2cc35d" containerID="a7f4a5da16afcef056b5a5fda046ab0306d428991771005d472a0b191d3ec202" exitCode=0
Oct 31 12:46:42.120057 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:42.119626    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5d310d-openshift-default-7c8bb48c95-n2tkf" event={"ID":"3e880a6e-3d1c-42e4-a6f3-a7a22c2cc35d","Type":"ContainerDied","Data":"a7f4a5da16afcef056b5a5fda046ab0306d428991771005d472a0b191d3ec202"}
Oct 31 12:46:42.120107 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:42.120066    5060 scope.go:122] "RemoveContainer" containerID="a7f4a5da16afcef056b5a5fda046ab0306d428991771005d472a0b191d3ec202"
Oct 31 12:46:42.121828 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:42.121796    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:46:42.121911 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:42.121870    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:46:42.121911 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:42.121900    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:46:43.133170 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:43.133132    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5d310d-openshift-default-7c8bb48c95-n2tkf" event={"ID":"3e880a6e-3d1c-42e4-a6f3-a7a22c2cc35d","Type":"ContainerStarted","Data":"4fac2d86dad1ebc23ea5d3f832ba1934f3485799c582624d659485c7ff99a2bd"}
Oct 31 12:46:43.194920 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:43.194885    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-5d310d-openshift-default-7c8bb48c95-n2tkf"
Oct 31 12:46:43.195118 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:43.195058    5060 patch_prober.go:29] interesting pod/e2e-api-a-5d310d-openshift-default-7c8bb48c95-n2tkf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body=
Oct 31 12:46:43.195201 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:43.195108    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-5d310d-openshift-default-7c8bb48c95-n2tkf" podUID="3e880a6e-3d1c-42e4-a6f3-a7a22c2cc35d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused"
Oct 31 12:46:44.195145 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:44.195103    5060 patch_prober.go:29] interesting pod/e2e-api-a-5d310d-openshift-default-7c8bb48c95-n2tkf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body=
Oct 31 12:46:44.195662 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:44.195174    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-5d310d-openshift-default-7c8bb48c95-n2tkf" podUID="3e880a6e-3d1c-42e4-a6f3-a7a22c2cc35d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused"
Oct 31 12:46:45.194357 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:45.194317    5060 patch_prober.go:29] interesting pod/e2e-api-a-5d310d-openshift-default-7c8bb48c95-n2tkf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body=
Oct 31 12:46:45.194661 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:45.194628    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-5d310d-openshift-default-7c8bb48c95-n2tkf" podUID="3e880a6e-3d1c-42e4-a6f3-a7a22c2cc35d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused"
Oct 31 12:46:46.041031 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:46.040972    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25afd7d2-4vdmm"]
Oct 31 12:46:46.210526 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:46.195460    5060 patch_prober.go:29] interesting pod/e2e-api-a-5d310d-openshift-default-7c8bb48c95-n2tkf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body=
Oct 31 12:46:46.210526 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:46.195549    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-5d310d-openshift-default-7c8bb48c95-n2tkf" podUID="3e880a6e-3d1c-42e4-a6f3-a7a22c2cc35d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused"
Oct 31 12:46:46.423913 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:46.423760    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25afd7d2-4vdmm"]
Oct 31 12:46:47.054298 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:47.054256    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3aa9227a-b604-4644-a5ee-97b5ccc1f33e" path="/var/lib/kubelet/pods/3aa9227a-b604-4644-a5ee-97b5ccc1f33e/volumes"
Oct 31 12:46:47.195074 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:47.195036    5060 patch_prober.go:29] interesting pod/e2e-api-a-5d310d-openshift-default-7c8bb48c95-n2tkf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body=
Oct 31 12:46:47.195358 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:47.195332    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-5d310d-openshift-default-7c8bb48c95-n2tkf" podUID="3e880a6e-3d1c-42e4-a6f3-a7a22c2cc35d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused"
Oct 31 12:46:48.238515 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:48.235629    5060 patch_prober.go:29] interesting pod/e2e-api-a-5d310d-openshift-default-7c8bb48c95-n2tkf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body=
Oct 31 12:46:48.238515 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:48.235766    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-5d310d-openshift-default-7c8bb48c95-n2tkf" podUID="3e880a6e-3d1c-42e4-a6f3-a7a22c2cc35d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused"
Oct 31 12:46:49.194187 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:49.194147    5060 patch_prober.go:29] interesting pod/e2e-api-a-5d310d-openshift-default-7c8bb48c95-n2tkf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body=
Oct 31 12:46:49.194405 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:49.194207    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-5d310d-openshift-default-7c8bb48c95-n2tkf" podUID="3e880a6e-3d1c-42e4-a6f3-a7a22c2cc35d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused"
Oct 31 12:46:50.257515 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:50.241884    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-5d310d-openshift-default-7c8bb48c95-n2tkf"
Oct 31 12:46:50.259343 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:50.259313    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-5d310d-openshift-default-7c8bb48c95-n2tkf"
Oct 31 12:46:50.261601 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:50.261578    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-5d310d-openshift-default-7c8bb48c95-n2tkf"
Oct 31 12:46:53.126005 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:53.125972    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-hkvl5"]
Oct 31 12:46:53.148179 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:53.148149    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-hkvl5"]
Oct 31 12:46:53.148550 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:53.148458    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-hkvl5"
Oct 31 12:46:53.182401 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:53.182370    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-fb2b1d-openshift-default-dockercfg-6b2h6\""
Oct 31 12:46:53.421574 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:53.418413    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e300ab1-6fe3-4548-90b3-0cbb8099e114-istio-envoy\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-hkvl5\" (UID: \"4e300ab1-6fe3-4548-90b3-0cbb8099e114\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-hkvl5"
Oct 31 12:46:53.421574 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:53.418477    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e300ab1-6fe3-4548-90b3-0cbb8099e114-credential-socket\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-hkvl5\" (UID: \"4e300ab1-6fe3-4548-90b3-0cbb8099e114\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-hkvl5"
Oct 31 12:46:53.421574 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:53.418559    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e300ab1-6fe3-4548-90b3-0cbb8099e114-istio-data\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-hkvl5\" (UID: \"4e300ab1-6fe3-4548-90b3-0cbb8099e114\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-hkvl5"
Oct 31 12:46:53.421574 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:53.418585    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e300ab1-6fe3-4548-90b3-0cbb8099e114-istio-podinfo\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-hkvl5\" (UID: \"4e300ab1-6fe3-4548-90b3-0cbb8099e114\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-hkvl5"
Oct 31 12:46:53.421574 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:53.418641    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e300ab1-6fe3-4548-90b3-0cbb8099e114-workload-certs\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-hkvl5\" (UID: \"4e300ab1-6fe3-4548-90b3-0cbb8099e114\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-hkvl5"
Oct 31 12:46:53.421574 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:53.418678    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e300ab1-6fe3-4548-90b3-0cbb8099e114-istio-token\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-hkvl5\" (UID: \"4e300ab1-6fe3-4548-90b3-0cbb8099e114\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-hkvl5"
Oct 31 12:46:53.421574 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:53.418740    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e300ab1-6fe3-4548-90b3-0cbb8099e114-workload-socket\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-hkvl5\" (UID: \"4e300ab1-6fe3-4548-90b3-0cbb8099e114\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-hkvl5"
Oct 31 12:46:53.421574 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:53.418922    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9sx4j\" (UniqueName: \"kubernetes.io/projected/4e300ab1-6fe3-4548-90b3-0cbb8099e114-kube-api-access-9sx4j\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-hkvl5\" (UID: \"4e300ab1-6fe3-4548-90b3-0cbb8099e114\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-hkvl5"
Oct 31 12:46:53.421574 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:53.419019    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e300ab1-6fe3-4548-90b3-0cbb8099e114-istiod-ca-cert\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-hkvl5\" (UID: \"4e300ab1-6fe3-4548-90b3-0cbb8099e114\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-hkvl5"
Oct 31 12:46:53.578231 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:53.548209    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9sx4j\" (UniqueName: \"kubernetes.io/projected/4e300ab1-6fe3-4548-90b3-0cbb8099e114-kube-api-access-9sx4j\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-hkvl5\" (UID: \"4e300ab1-6fe3-4548-90b3-0cbb8099e114\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-hkvl5"
Oct 31 12:46:53.578231 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:53.548338    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e300ab1-6fe3-4548-90b3-0cbb8099e114-istiod-ca-cert\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-hkvl5\" (UID: \"4e300ab1-6fe3-4548-90b3-0cbb8099e114\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-hkvl5"
Oct 31 12:46:53.578231 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:53.548418    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e300ab1-6fe3-4548-90b3-0cbb8099e114-istio-envoy\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-hkvl5\" (UID: \"4e300ab1-6fe3-4548-90b3-0cbb8099e114\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-hkvl5"
Oct 31 12:46:53.578231 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:53.548460    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e300ab1-6fe3-4548-90b3-0cbb8099e114-credential-socket\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-hkvl5\" (UID: \"4e300ab1-6fe3-4548-90b3-0cbb8099e114\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-hkvl5"
Oct 31 12:46:53.578231 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:53.548550    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e300ab1-6fe3-4548-90b3-0cbb8099e114-istio-data\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-hkvl5\" (UID: \"4e300ab1-6fe3-4548-90b3-0cbb8099e114\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-hkvl5"
Oct 31 12:46:53.578231 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:53.548578    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e300ab1-6fe3-4548-90b3-0cbb8099e114-istio-podinfo\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-hkvl5\" (UID: \"4e300ab1-6fe3-4548-90b3-0cbb8099e114\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-hkvl5"
Oct 31 12:46:53.578231 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:53.548631    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e300ab1-6fe3-4548-90b3-0cbb8099e114-workload-certs\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-hkvl5\" (UID: \"4e300ab1-6fe3-4548-90b3-0cbb8099e114\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-hkvl5"
Oct 31 12:46:53.578231 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:53.548664    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e300ab1-6fe3-4548-90b3-0cbb8099e114-istio-token\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-hkvl5\" (UID: \"4e300ab1-6fe3-4548-90b3-0cbb8099e114\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-hkvl5"
Oct 31 12:46:53.578231 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:53.548726    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e300ab1-6fe3-4548-90b3-0cbb8099e114-workload-socket\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-hkvl5\" (UID: \"4e300ab1-6fe3-4548-90b3-0cbb8099e114\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-hkvl5"
Oct 31 12:46:53.578231 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:53.549379    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e300ab1-6fe3-4548-90b3-0cbb8099e114-workload-socket\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-hkvl5\" (UID: \"4e300ab1-6fe3-4548-90b3-0cbb8099e114\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-hkvl5"
Oct 31 12:46:53.578231 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:53.550412    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e300ab1-6fe3-4548-90b3-0cbb8099e114-istiod-ca-cert\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-hkvl5\" (UID: \"4e300ab1-6fe3-4548-90b3-0cbb8099e114\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-hkvl5"
Oct 31 12:46:53.578231 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:53.552336    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e300ab1-6fe3-4548-90b3-0cbb8099e114-credential-socket\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-hkvl5\" (UID: \"4e300ab1-6fe3-4548-90b3-0cbb8099e114\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-hkvl5"
Oct 31 12:46:53.578231 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:53.552713    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e300ab1-6fe3-4548-90b3-0cbb8099e114-istio-data\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-hkvl5\" (UID: \"4e300ab1-6fe3-4548-90b3-0cbb8099e114\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-hkvl5"
Oct 31 12:46:53.578231 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:53.552956    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e300ab1-6fe3-4548-90b3-0cbb8099e114-workload-certs\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-hkvl5\" (UID: \"4e300ab1-6fe3-4548-90b3-0cbb8099e114\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-hkvl5"
Oct 31 12:46:53.578231 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:53.569739    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e300ab1-6fe3-4548-90b3-0cbb8099e114-istio-podinfo\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-hkvl5\" (UID: \"4e300ab1-6fe3-4548-90b3-0cbb8099e114\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-hkvl5"
Oct 31 12:46:53.579148 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:53.578767    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e300ab1-6fe3-4548-90b3-0cbb8099e114-istio-envoy\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-hkvl5\" (UID: \"4e300ab1-6fe3-4548-90b3-0cbb8099e114\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-hkvl5"
Oct 31 12:46:53.602512 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:53.593042    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e300ab1-6fe3-4548-90b3-0cbb8099e114-istio-token\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-hkvl5\" (UID: \"4e300ab1-6fe3-4548-90b3-0cbb8099e114\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-hkvl5"
Oct 31 12:46:53.602512 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:53.595244    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9sx4j\" (UniqueName: \"kubernetes.io/projected/4e300ab1-6fe3-4548-90b3-0cbb8099e114-kube-api-access-9sx4j\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-hkvl5\" (UID: \"4e300ab1-6fe3-4548-90b3-0cbb8099e114\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-hkvl5"
Oct 31 12:46:53.771890 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:53.771787    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-hkvl5"
Oct 31 12:46:54.336569 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:46:54.336483    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e300ab1_6fe3_4548_90b3_0cbb8099e114.slice/crio-3ec910f98a451578444bd506b9a47d60c26096043ad1f9d4b9b1da5a6186d9c1 WatchSource:0}: Error finding container 3ec910f98a451578444bd506b9a47d60c26096043ad1f9d4b9b1da5a6186d9c1: Status 404 returned error can't find the container with id 3ec910f98a451578444bd506b9a47d60c26096043ad1f9d4b9b1da5a6186d9c1
Oct 31 12:46:54.339535 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:54.339468    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-hkvl5"]
Oct 31 12:46:54.340225 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:54.340194    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:46:54.340268 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:54.340256    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:46:54.340308 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:54.340285    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:46:54.860515 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:54.850697    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-hkvl5" event={"ID":"4e300ab1-6fe3-4548-90b3-0cbb8099e114","Type":"ContainerStarted","Data":"1c42065af8b553bdc7cf3f32752a5147a10231f9339cf04598ee8ae8db27912e"}
Oct 31 12:46:54.860515 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:54.850821    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-hkvl5" event={"ID":"4e300ab1-6fe3-4548-90b3-0cbb8099e114","Type":"ContainerStarted","Data":"3ec910f98a451578444bd506b9a47d60c26096043ad1f9d4b9b1da5a6186d9c1"}
Oct 31 12:46:55.540320 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:55.540252    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-hkvl5" podStartSLOduration=2.54023075 podStartE2EDuration="2.54023075s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:46:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:46:54.908919367 +0000 UTC m=+1178.011987081" watchObservedRunningTime="2026-10-31 12:46:55.54023075 +0000 UTC m=+1178.643298464"
Oct 31 12:46:55.579341 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:55.557775    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5d310d-2h8b6"]
Oct 31 12:46:55.579341 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:55.568554    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5d310d-2h8b6"
Oct 31 12:46:55.579341 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:55.573969    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 12:46:55.587512 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:55.579905    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 12:46:55.587512 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:55.580018    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7wsmb\""
Oct 31 12:46:55.587512 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:55.580089    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 12:46:55.617819 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:55.617782    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5d310d-2h8b6"]
Oct 31 12:46:55.651951 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:55.651920    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ebdedf6-8a4c-46bf-b6b4-022472588c22-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-5d310d-2h8b6\" (UID: \"6ebdedf6-8a4c-46bf-b6b4-022472588c22\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5d310d-2h8b6"
Oct 31 12:46:55.652254 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:55.652224    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tjdg\" (UniqueName: \"kubernetes.io/projected/6ebdedf6-8a4c-46bf-b6b4-022472588c22-kube-api-access-8tjdg\") pod \"maas-api-revoke-keys-e2e-api-a-5d310d-2h8b6\" (UID: \"6ebdedf6-8a4c-46bf-b6b4-022472588c22\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5d310d-2h8b6"
Oct 31 12:46:55.652364 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:55.652348    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ebdedf6-8a4c-46bf-b6b4-022472588c22-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-5d310d-2h8b6\" (UID: \"6ebdedf6-8a4c-46bf-b6b4-022472588c22\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5d310d-2h8b6"
Oct 31 12:46:55.753745 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:55.753706    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ebdedf6-8a4c-46bf-b6b4-022472588c22-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-5d310d-2h8b6\" (UID: \"6ebdedf6-8a4c-46bf-b6b4-022472588c22\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5d310d-2h8b6"
Oct 31 12:46:55.753953 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:55.753904    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ebdedf6-8a4c-46bf-b6b4-022472588c22-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-5d310d-2h8b6\" (UID: \"6ebdedf6-8a4c-46bf-b6b4-022472588c22\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5d310d-2h8b6"
Oct 31 12:46:55.754116 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:55.754102    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tjdg\" (UniqueName: \"kubernetes.io/projected/6ebdedf6-8a4c-46bf-b6b4-022472588c22-kube-api-access-8tjdg\") pod \"maas-api-revoke-keys-e2e-api-a-5d310d-2h8b6\" (UID: \"6ebdedf6-8a4c-46bf-b6b4-022472588c22\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5d310d-2h8b6"
Oct 31 12:46:55.754404 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:55.754381    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ebdedf6-8a4c-46bf-b6b4-022472588c22-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-5d310d-2h8b6\" (UID: \"6ebdedf6-8a4c-46bf-b6b4-022472588c22\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5d310d-2h8b6"
Oct 31 12:46:55.794731 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:55.791066    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ebdedf6-8a4c-46bf-b6b4-022472588c22-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-5d310d-2h8b6\" (UID: \"6ebdedf6-8a4c-46bf-b6b4-022472588c22\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5d310d-2h8b6"
Oct 31 12:46:55.794731 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:55.791574    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tjdg\" (UniqueName: \"kubernetes.io/projected/6ebdedf6-8a4c-46bf-b6b4-022472588c22-kube-api-access-8tjdg\") pod \"maas-api-revoke-keys-e2e-api-a-5d310d-2h8b6\" (UID: \"6ebdedf6-8a4c-46bf-b6b4-022472588c22\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5d310d-2h8b6"
Oct 31 12:46:55.794731 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:55.793585    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-hkvl5"
Oct 31 12:46:55.800428 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:55.800401    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-hkvl5"
Oct 31 12:46:55.893274 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:55.893205    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-hkvl5"
Oct 31 12:46:55.894864 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:55.894838    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-hkvl5"
Oct 31 12:46:55.995767 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:55.995736    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5d310d-2h8b6"
Oct 31 12:46:56.434548 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:46:56.432331    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ebdedf6_8a4c_46bf_b6b4_022472588c22.slice/crio-2643d530ccd369a0c7c27cafd431b34a299058c8a94bffea6a9548a9003286a8 WatchSource:0}: Error finding container 2643d530ccd369a0c7c27cafd431b34a299058c8a94bffea6a9548a9003286a8: Status 404 returned error can't find the container with id 2643d530ccd369a0c7c27cafd431b34a299058c8a94bffea6a9548a9003286a8
Oct 31 12:46:56.457370 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:56.457337    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5d310d-2h8b6"]
Oct 31 12:46:56.906421 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:56.906382    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5d310d-2h8b6" event={"ID":"6ebdedf6-8a4c-46bf-b6b4-022472588c22","Type":"ContainerStarted","Data":"2643d530ccd369a0c7c27cafd431b34a299058c8a94bffea6a9548a9003286a8"}
Oct 31 12:46:57.921297 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:57.921215    5060 generic.go:356] "Generic (PLEG): container finished" podID="6ebdedf6-8a4c-46bf-b6b4-022472588c22" containerID="acb276220fae73031ba6385f851e7954286be7110291008ead66dfdf1cfcb663" exitCode=0
Oct 31 12:46:57.921751 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:57.921594    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5d310d-2h8b6" event={"ID":"6ebdedf6-8a4c-46bf-b6b4-022472588c22","Type":"ContainerDied","Data":"acb276220fae73031ba6385f851e7954286be7110291008ead66dfdf1cfcb663"}
Oct 31 12:46:59.821673 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:46:59.821641    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5d310d-2h8b6"
Oct 31 12:47:00.030584 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:00.024097    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ebdedf6-8a4c-46bf-b6b4-022472588c22-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ebdedf6-8a4c-46bf-b6b4-022472588c22-maas-api-openshift-ca\") pod \"6ebdedf6-8a4c-46bf-b6b4-022472588c22\" (UID: \"6ebdedf6-8a4c-46bf-b6b4-022472588c22\") "
Oct 31 12:47:00.030584 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:00.024221    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6ebdedf6-8a4c-46bf-b6b4-022472588c22-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ebdedf6-8a4c-46bf-b6b4-022472588c22-maas-api-xks-ca\") pod \"6ebdedf6-8a4c-46bf-b6b4-022472588c22\" (UID: \"6ebdedf6-8a4c-46bf-b6b4-022472588c22\") "
Oct 31 12:47:00.030584 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:00.024406    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ebdedf6-8a4c-46bf-b6b4-022472588c22-kube-api-access-8tjdg\" (UniqueName: \"kubernetes.io/projected/6ebdedf6-8a4c-46bf-b6b4-022472588c22-kube-api-access-8tjdg\") pod \"6ebdedf6-8a4c-46bf-b6b4-022472588c22\" (UID: \"6ebdedf6-8a4c-46bf-b6b4-022472588c22\") "
Oct 31 12:47:00.052564 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:00.052198    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ebdedf6-8a4c-46bf-b6b4-022472588c22-maas-api-openshift-ca" pod "6ebdedf6-8a4c-46bf-b6b4-022472588c22" (UID: "6ebdedf6-8a4c-46bf-b6b4-022472588c22"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:47:00.053905 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:00.053849    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ebdedf6-8a4c-46bf-b6b4-022472588c22-maas-api-openshift-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:47:00.080242 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:00.079837    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ebdedf6-8a4c-46bf-b6b4-022472588c22-maas-api-xks-ca" pod "6ebdedf6-8a4c-46bf-b6b4-022472588c22" (UID: "6ebdedf6-8a4c-46bf-b6b4-022472588c22"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:47:00.101812 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:00.097744    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ebdedf6-8a4c-46bf-b6b4-022472588c22-kube-api-access-8tjdg" pod "6ebdedf6-8a4c-46bf-b6b4-022472588c22" (UID: "6ebdedf6-8a4c-46bf-b6b4-022472588c22"). InnerVolumeSpecName "kube-api-access-8tjdg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:47:00.133899 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:47:00.133866    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7phb" containerName="extractor"
Oct 31 12:47:00.186527 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:00.155417    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ebdedf6-8a4c-46bf-b6b4-022472588c22-maas-api-xks-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:47:00.186527 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:00.155450    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8tjdg\" (UniqueName: \"kubernetes.io/projected/6ebdedf6-8a4c-46bf-b6b4-022472588c22-kube-api-access-8tjdg\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:47:00.294399 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:00.294306    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5d310d-2h8b6"
Oct 31 12:47:00.294399 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:00.294336    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5d310d-2h8b6" event={"ID":"6ebdedf6-8a4c-46bf-b6b4-022472588c22","Type":"ContainerDied","Data":"2643d530ccd369a0c7c27cafd431b34a299058c8a94bffea6a9548a9003286a8"}
Oct 31 12:47:00.294399 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:00.294388    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2643d530ccd369a0c7c27cafd431b34a299058c8a94bffea6a9548a9003286a8"
Oct 31 12:47:01.031022 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:01.030983    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f549b09-4lfmf"]
Oct 31 12:47:01.037808 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:01.037781    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-26c69f97-npzvq"]
Oct 31 12:47:01.098514 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:01.094024    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4505ecc2-xbmxd"]
Oct 31 12:47:01.098795 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:01.098603    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f549b09-4lfmf"]
Oct 31 12:47:01.105349 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:01.105321    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-26c69f97-npzvq"]
Oct 31 12:47:01.111979 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:01.111935    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4505ecc2-xbmxd"]
Oct 31 12:47:03.049888 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:03.049847    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a40f13f-5cbe-4cfe-bbf2-d1875c6537c4" path="/var/lib/kubelet/pods/0a40f13f-5cbe-4cfe-bbf2-d1875c6537c4/volumes"
Oct 31 12:47:03.050609 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:03.050586    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="798f63df-6d7f-442a-b8fc-9611c67040f8" path="/var/lib/kubelet/pods/798f63df-6d7f-442a-b8fc-9611c67040f8/volumes"
Oct 31 12:47:03.051112 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:03.051091    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96617f2d-e89b-43b1-8b6c-a580a2d372be" path="/var/lib/kubelet/pods/96617f2d-e89b-43b1-8b6c-a580a2d372be/volumes"
Oct 31 12:47:05.029576 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:05.021099    5060 patch_prober.go:29] interesting pod/e2e-api-a-f22e7f-openshift-default-84f9758845-24j6m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 12:47:05.029576 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:05.021189    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-f22e7f-openshift-default-84f9758845-24j6m" podUID="1af894f4-5dbb-495a-b896-670ce365742e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:47:07.032136 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:07.032079    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e6b5b0c1-v9xxw"]
Oct 31 12:47:07.045833 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:07.039023    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e6b5b0c1-v9xxw"]
Oct 31 12:47:07.060091 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:07.060054    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3747a92f-de3a-439a-8e3e-a4b7d2f3b048" path="/var/lib/kubelet/pods/3747a92f-de3a-439a-8e3e-a4b7d2f3b048/volumes"
Oct 31 12:47:07.512431 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:07.512352    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e1fc0ffe-xvmjr"]
Oct 31 12:47:07.513325 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:07.513300    5060 cpu_manager.go:414] "containerMap: removing container" podUID="6ebdedf6-8a4c-46bf-b6b4-022472588c22" containerName="revoke-keys"
Oct 31 12:47:07.513325 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:07.513322    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ebdedf6-8a4c-46bf-b6b4-022472588c22" containerName="revoke-keys"
Oct 31 12:47:07.513534 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:07.513522    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ebdedf6-8a4c-46bf-b6b4-022472588c22" containerName="revoke-keys"
Oct 31 12:47:07.540575 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:07.539359    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e1fc0ffe-xvmjr"
Oct 31 12:47:07.546031 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:07.545572    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 12:47:07.546589 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:07.546563    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 12:47:07.602854 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:07.560318    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e1fc0ffe-xvmjr"]
Oct 31 12:47:07.628036 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:07.613935    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7wsmb\""
Oct 31 12:47:07.628036 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:07.614003    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 12:47:07.719322 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:07.719275    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ssfnb\" (UniqueName: \"kubernetes.io/projected/8126e807-51f6-4f81-82a2-23d9d1a33742-kube-api-access-ssfnb\") pod \"maas-api-revoke-keys-e2e-mt-e1fc0ffe-xvmjr\" (UID: \"8126e807-51f6-4f81-82a2-23d9d1a33742\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e1fc0ffe-xvmjr"
Oct 31 12:47:07.719553 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:07.719418    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8126e807-51f6-4f81-82a2-23d9d1a33742-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-e1fc0ffe-xvmjr\" (UID: \"8126e807-51f6-4f81-82a2-23d9d1a33742\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e1fc0ffe-xvmjr"
Oct 31 12:47:07.719553 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:07.719453    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8126e807-51f6-4f81-82a2-23d9d1a33742-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-e1fc0ffe-xvmjr\" (UID: \"8126e807-51f6-4f81-82a2-23d9d1a33742\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e1fc0ffe-xvmjr"
Oct 31 12:47:07.820877 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:07.820837    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8126e807-51f6-4f81-82a2-23d9d1a33742-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-e1fc0ffe-xvmjr\" (UID: \"8126e807-51f6-4f81-82a2-23d9d1a33742\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e1fc0ffe-xvmjr"
Oct 31 12:47:07.821095 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:07.820904    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8126e807-51f6-4f81-82a2-23d9d1a33742-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-e1fc0ffe-xvmjr\" (UID: \"8126e807-51f6-4f81-82a2-23d9d1a33742\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e1fc0ffe-xvmjr"
Oct 31 12:47:07.821272 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:07.821250    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ssfnb\" (UniqueName: \"kubernetes.io/projected/8126e807-51f6-4f81-82a2-23d9d1a33742-kube-api-access-ssfnb\") pod \"maas-api-revoke-keys-e2e-mt-e1fc0ffe-xvmjr\" (UID: \"8126e807-51f6-4f81-82a2-23d9d1a33742\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e1fc0ffe-xvmjr"
Oct 31 12:47:07.821771 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:07.821738    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8126e807-51f6-4f81-82a2-23d9d1a33742-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-e1fc0ffe-xvmjr\" (UID: \"8126e807-51f6-4f81-82a2-23d9d1a33742\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e1fc0ffe-xvmjr"
Oct 31 12:47:07.823693 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:07.823665    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8126e807-51f6-4f81-82a2-23d9d1a33742-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-e1fc0ffe-xvmjr\" (UID: \"8126e807-51f6-4f81-82a2-23d9d1a33742\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e1fc0ffe-xvmjr"
Oct 31 12:47:07.833931 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:07.833903    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ssfnb\" (UniqueName: \"kubernetes.io/projected/8126e807-51f6-4f81-82a2-23d9d1a33742-kube-api-access-ssfnb\") pod \"maas-api-revoke-keys-e2e-mt-e1fc0ffe-xvmjr\" (UID: \"8126e807-51f6-4f81-82a2-23d9d1a33742\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e1fc0ffe-xvmjr"
Oct 31 12:47:07.849370 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:07.849335    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f22e7f-fw72r"]
Oct 31 12:47:07.893289 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:07.893246    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f22e7f-fw72r"]
Oct 31 12:47:07.893477 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:07.893395    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f22e7f-fw72r"
Oct 31 12:47:07.935841 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:07.935577    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e1fc0ffe-xvmjr"
Oct 31 12:47:08.026098 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:08.026050    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f86d8432-bab6-4a08-a85d-c61726d8c102-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f22e7f-fw72r\" (UID: \"f86d8432-bab6-4a08-a85d-c61726d8c102\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f22e7f-fw72r"
Oct 31 12:47:08.026276 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:08.026130    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f86d8432-bab6-4a08-a85d-c61726d8c102-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f22e7f-fw72r\" (UID: \"f86d8432-bab6-4a08-a85d-c61726d8c102\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f22e7f-fw72r"
Oct 31 12:47:08.026331 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:08.026308    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bv2hz\" (UniqueName: \"kubernetes.io/projected/f86d8432-bab6-4a08-a85d-c61726d8c102-kube-api-access-bv2hz\") pod \"maas-api-revoke-keys-e2e-api-a-f22e7f-fw72r\" (UID: \"f86d8432-bab6-4a08-a85d-c61726d8c102\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f22e7f-fw72r"
Oct 31 12:47:08.030578 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:08.028362    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa778ea2-dsjqx"]
Oct 31 12:47:08.035258 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:08.035234    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c5e4799-mjqt5"]
Oct 31 12:47:08.040852 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:08.040830    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa778ea2-dsjqx"]
Oct 31 12:47:08.047958 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:08.047928    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c5e4799-mjqt5"]
Oct 31 12:47:08.129274 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:08.128458    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f86d8432-bab6-4a08-a85d-c61726d8c102-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f22e7f-fw72r\" (UID: \"f86d8432-bab6-4a08-a85d-c61726d8c102\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f22e7f-fw72r"
Oct 31 12:47:08.129274 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:08.128568    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f86d8432-bab6-4a08-a85d-c61726d8c102-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f22e7f-fw72r\" (UID: \"f86d8432-bab6-4a08-a85d-c61726d8c102\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f22e7f-fw72r"
Oct 31 12:47:08.129274 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:08.128707    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bv2hz\" (UniqueName: \"kubernetes.io/projected/f86d8432-bab6-4a08-a85d-c61726d8c102-kube-api-access-bv2hz\") pod \"maas-api-revoke-keys-e2e-api-a-f22e7f-fw72r\" (UID: \"f86d8432-bab6-4a08-a85d-c61726d8c102\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f22e7f-fw72r"
Oct 31 12:47:08.129960 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:08.129931    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f86d8432-bab6-4a08-a85d-c61726d8c102-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f22e7f-fw72r\" (UID: \"f86d8432-bab6-4a08-a85d-c61726d8c102\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f22e7f-fw72r"
Oct 31 12:47:08.131426 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:08.131389    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f86d8432-bab6-4a08-a85d-c61726d8c102-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f22e7f-fw72r\" (UID: \"f86d8432-bab6-4a08-a85d-c61726d8c102\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f22e7f-fw72r"
Oct 31 12:47:08.141558 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:08.141523    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bv2hz\" (UniqueName: \"kubernetes.io/projected/f86d8432-bab6-4a08-a85d-c61726d8c102-kube-api-access-bv2hz\") pod \"maas-api-revoke-keys-e2e-api-a-f22e7f-fw72r\" (UID: \"f86d8432-bab6-4a08-a85d-c61726d8c102\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f22e7f-fw72r"
Oct 31 12:47:08.236705 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:08.236669    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f22e7f-fw72r"
Oct 31 12:47:08.278672 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:47:08.278635    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8126e807_51f6_4f81_82a2_23d9d1a33742.slice/crio-a17f8c812699fba875eb45da274cf653cc3d334e4487b4cd39f2d034eb8741eb WatchSource:0}: Error finding container a17f8c812699fba875eb45da274cf653cc3d334e4487b4cd39f2d034eb8741eb: Status 404 returned error can't find the container with id a17f8c812699fba875eb45da274cf653cc3d334e4487b4cd39f2d034eb8741eb
Oct 31 12:47:08.281927 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:08.281881    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e1fc0ffe-xvmjr"]
Oct 31 12:47:08.610896 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:08.610858    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e1fc0ffe-xvmjr" event={"ID":"8126e807-51f6-4f81-82a2-23d9d1a33742","Type":"ContainerStarted","Data":"a17f8c812699fba875eb45da274cf653cc3d334e4487b4cd39f2d034eb8741eb"}
Oct 31 12:47:08.882239 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:47:08.882204    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf86d8432_bab6_4a08_a85d_c61726d8c102.slice/crio-20cda14e60e8b5934d9862532faa230cf877e78a5a8c4922d94fb69fd72e8095 WatchSource:0}: Error finding container 20cda14e60e8b5934d9862532faa230cf877e78a5a8c4922d94fb69fd72e8095: Status 404 returned error can't find the container with id 20cda14e60e8b5934d9862532faa230cf877e78a5a8c4922d94fb69fd72e8095
Oct 31 12:47:08.884341 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:08.884315    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f22e7f-fw72r"]
Oct 31 12:47:09.051324 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:09.051287    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ac6a465-1595-4d84-b230-72b56e67a33e" path="/var/lib/kubelet/pods/0ac6a465-1595-4d84-b230-72b56e67a33e/volumes"
Oct 31 12:47:09.051999 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:09.051976    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5cb9104d-0b45-4ba5-80ca-bff8d339fd36" path="/var/lib/kubelet/pods/5cb9104d-0b45-4ba5-80ca-bff8d339fd36/volumes"
Oct 31 12:47:09.625813 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:09.625782    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f22e7f-fw72r" event={"ID":"f86d8432-bab6-4a08-a85d-c61726d8c102","Type":"ContainerStarted","Data":"4196ac853c00bbb370884f073952858bd6073a099824e57d1ba818f0a19289a0"}
Oct 31 12:47:09.625949 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:09.625835    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f22e7f-fw72r" event={"ID":"f86d8432-bab6-4a08-a85d-c61726d8c102","Type":"ContainerStarted","Data":"20cda14e60e8b5934d9862532faa230cf877e78a5a8c4922d94fb69fd72e8095"}
Oct 31 12:47:09.627817 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:09.627793    5060 generic.go:356] "Generic (PLEG): container finished" podID="8126e807-51f6-4f81-82a2-23d9d1a33742" containerID="9f840cc0629fb7cbbeecfa77fe1c24e19568541bbfac0a40ad8fec6cbe4b7aac" exitCode=0
Oct 31 12:47:09.627913 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:09.627873    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e1fc0ffe-xvmjr" event={"ID":"8126e807-51f6-4f81-82a2-23d9d1a33742","Type":"ContainerDied","Data":"9f840cc0629fb7cbbeecfa77fe1c24e19568541bbfac0a40ad8fec6cbe4b7aac"}
Oct 31 12:47:09.650117 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:09.650071    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f22e7f-fw72r" podStartSLOduration=2.6500559790000002 podStartE2EDuration="2.650055979s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:47:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:47:09.644877964 +0000 UTC m=+1192.747945677" watchObservedRunningTime="2026-10-31 12:47:09.650055979 +0000 UTC m=+1192.753123691"
Oct 31 12:47:10.643172 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:10.643135    5060 generic.go:356] "Generic (PLEG): container finished" podID="f86d8432-bab6-4a08-a85d-c61726d8c102" containerID="4196ac853c00bbb370884f073952858bd6073a099824e57d1ba818f0a19289a0" exitCode=0
Oct 31 12:47:10.643712 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:10.643219    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f22e7f-fw72r" event={"ID":"f86d8432-bab6-4a08-a85d-c61726d8c102","Type":"ContainerDied","Data":"4196ac853c00bbb370884f073952858bd6073a099824e57d1ba818f0a19289a0"}
Oct 31 12:47:11.102310 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:11.102273    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e1fc0ffe-xvmjr"
Oct 31 12:47:11.198513 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:11.196625    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8126e807-51f6-4f81-82a2-23d9d1a33742-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8126e807-51f6-4f81-82a2-23d9d1a33742-maas-api-openshift-ca\") pod \"8126e807-51f6-4f81-82a2-23d9d1a33742\" (UID: \"8126e807-51f6-4f81-82a2-23d9d1a33742\") "
Oct 31 12:47:11.198513 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:11.196693    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8126e807-51f6-4f81-82a2-23d9d1a33742-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8126e807-51f6-4f81-82a2-23d9d1a33742-maas-api-xks-ca\") pod \"8126e807-51f6-4f81-82a2-23d9d1a33742\" (UID: \"8126e807-51f6-4f81-82a2-23d9d1a33742\") "
Oct 31 12:47:11.198513 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:11.196884    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8126e807-51f6-4f81-82a2-23d9d1a33742-kube-api-access-ssfnb\" (UniqueName: \"kubernetes.io/projected/8126e807-51f6-4f81-82a2-23d9d1a33742-kube-api-access-ssfnb\") pod \"8126e807-51f6-4f81-82a2-23d9d1a33742\" (UID: \"8126e807-51f6-4f81-82a2-23d9d1a33742\") "
Oct 31 12:47:11.198513 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:11.198309    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8126e807-51f6-4f81-82a2-23d9d1a33742-maas-api-openshift-ca" pod "8126e807-51f6-4f81-82a2-23d9d1a33742" (UID: "8126e807-51f6-4f81-82a2-23d9d1a33742"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:47:11.235204 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:11.235163    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8126e807-51f6-4f81-82a2-23d9d1a33742-maas-api-xks-ca" pod "8126e807-51f6-4f81-82a2-23d9d1a33742" (UID: "8126e807-51f6-4f81-82a2-23d9d1a33742"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:47:11.235379 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:11.235351    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8126e807-51f6-4f81-82a2-23d9d1a33742-kube-api-access-ssfnb" pod "8126e807-51f6-4f81-82a2-23d9d1a33742" (UID: "8126e807-51f6-4f81-82a2-23d9d1a33742"). InnerVolumeSpecName "kube-api-access-ssfnb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:47:11.299247 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:11.299170    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8126e807-51f6-4f81-82a2-23d9d1a33742-maas-api-openshift-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:47:11.299247 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:11.299210    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8126e807-51f6-4f81-82a2-23d9d1a33742-maas-api-xks-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:47:11.299247 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:11.299224    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ssfnb\" (UniqueName: \"kubernetes.io/projected/8126e807-51f6-4f81-82a2-23d9d1a33742-kube-api-access-ssfnb\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:47:11.662304 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:11.662204    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e1fc0ffe-xvmjr"
Oct 31 12:47:11.662304 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:11.662235    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e1fc0ffe-xvmjr" event={"ID":"8126e807-51f6-4f81-82a2-23d9d1a33742","Type":"ContainerDied","Data":"a17f8c812699fba875eb45da274cf653cc3d334e4487b4cd39f2d034eb8741eb"}
Oct 31 12:47:11.662784 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:11.662311    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a17f8c812699fba875eb45da274cf653cc3d334e4487b4cd39f2d034eb8741eb"
Oct 31 12:47:12.105236 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:12.105197    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f22e7f-fw72r"
Oct 31 12:47:12.168763 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:12.168245    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f86d8432-bab6-4a08-a85d-c61726d8c102-kube-api-access-bv2hz\" (UniqueName: \"kubernetes.io/projected/f86d8432-bab6-4a08-a85d-c61726d8c102-kube-api-access-bv2hz\") pod \"f86d8432-bab6-4a08-a85d-c61726d8c102\" (UID: \"f86d8432-bab6-4a08-a85d-c61726d8c102\") "
Oct 31 12:47:12.168763 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:12.168368    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f86d8432-bab6-4a08-a85d-c61726d8c102-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f86d8432-bab6-4a08-a85d-c61726d8c102-maas-api-openshift-ca\") pod \"f86d8432-bab6-4a08-a85d-c61726d8c102\" (UID: \"f86d8432-bab6-4a08-a85d-c61726d8c102\") "
Oct 31 12:47:12.168763 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:12.168752    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f86d8432-bab6-4a08-a85d-c61726d8c102-maas-api-openshift-ca" pod "f86d8432-bab6-4a08-a85d-c61726d8c102" (UID: "f86d8432-bab6-4a08-a85d-c61726d8c102"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:47:12.169064 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:12.168919    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f86d8432-bab6-4a08-a85d-c61726d8c102-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f86d8432-bab6-4a08-a85d-c61726d8c102-maas-api-xks-ca\") pod \"f86d8432-bab6-4a08-a85d-c61726d8c102\" (UID: \"f86d8432-bab6-4a08-a85d-c61726d8c102\") "
Oct 31 12:47:12.171026 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:12.170999    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f86d8432-bab6-4a08-a85d-c61726d8c102-maas-api-openshift-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:47:12.187598 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:12.172888    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f86d8432-bab6-4a08-a85d-c61726d8c102-kube-api-access-bv2hz" pod "f86d8432-bab6-4a08-a85d-c61726d8c102" (UID: "f86d8432-bab6-4a08-a85d-c61726d8c102"). InnerVolumeSpecName "kube-api-access-bv2hz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:47:12.197274 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:12.197237    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f86d8432-bab6-4a08-a85d-c61726d8c102-maas-api-xks-ca" pod "f86d8432-bab6-4a08-a85d-c61726d8c102" (UID: "f86d8432-bab6-4a08-a85d-c61726d8c102"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:47:12.272579 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:12.272541    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bv2hz\" (UniqueName: \"kubernetes.io/projected/f86d8432-bab6-4a08-a85d-c61726d8c102-kube-api-access-bv2hz\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:47:12.272579 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:12.272576    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f86d8432-bab6-4a08-a85d-c61726d8c102-maas-api-xks-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:47:12.731600 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:12.729844    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f22e7f-fw72r"
Oct 31 12:47:12.731600 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:12.729901    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f22e7f-fw72r" event={"ID":"f86d8432-bab6-4a08-a85d-c61726d8c102","Type":"ContainerDied","Data":"20cda14e60e8b5934d9862532faa230cf877e78a5a8c4922d94fb69fd72e8095"}
Oct 31 12:47:12.731600 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:12.729966    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="20cda14e60e8b5934d9862532faa230cf877e78a5a8c4922d94fb69fd72e8095"
Oct 31 12:47:13.649618 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:13.649549    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3a5f961-mzrdk"]
Oct 31 12:47:13.650392 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:13.650373    5060 cpu_manager.go:414] "containerMap: removing container" podUID="f86d8432-bab6-4a08-a85d-c61726d8c102" containerName="revoke-keys"
Oct 31 12:47:13.650392 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:13.650393    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f86d8432-bab6-4a08-a85d-c61726d8c102" containerName="revoke-keys"
Oct 31 12:47:13.650392 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:13.650414    5060 cpu_manager.go:414] "containerMap: removing container" podUID="8126e807-51f6-4f81-82a2-23d9d1a33742" containerName="revoke-keys"
Oct 31 12:47:13.650392 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:13.650419    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8126e807-51f6-4f81-82a2-23d9d1a33742" containerName="revoke-keys"
Oct 31 12:47:13.650735 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:13.650566    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="f86d8432-bab6-4a08-a85d-c61726d8c102" containerName="revoke-keys"
Oct 31 12:47:13.650735 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:13.650578    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="8126e807-51f6-4f81-82a2-23d9d1a33742" containerName="revoke-keys"
Oct 31 12:47:13.691831 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:13.691781    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3a5f961-mzrdk"]
Oct 31 12:47:13.692024 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:13.691917    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3a5f961-mzrdk"
Oct 31 12:47:13.696211 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:13.696184    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 12:47:13.697057 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:13.697036    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 12:47:13.697183 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:13.697063    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 12:47:13.697311 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:13.697225    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7wsmb\""
Oct 31 12:47:13.844081 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:13.844045    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c370a0b-7c18-46a6-8748-92c92b46dca0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e3a5f961-mzrdk\" (UID: \"8c370a0b-7c18-46a6-8748-92c92b46dca0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3a5f961-mzrdk"
Oct 31 12:47:13.844627 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:13.844105    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wl27k\" (UniqueName: \"kubernetes.io/projected/8c370a0b-7c18-46a6-8748-92c92b46dca0-kube-api-access-wl27k\") pod \"maas-api-revoke-sub-e3a5f961-mzrdk\" (UID: \"8c370a0b-7c18-46a6-8748-92c92b46dca0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3a5f961-mzrdk"
Oct 31 12:47:13.844627 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:13.844465    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c370a0b-7c18-46a6-8748-92c92b46dca0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e3a5f961-mzrdk\" (UID: \"8c370a0b-7c18-46a6-8748-92c92b46dca0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3a5f961-mzrdk"
Oct 31 12:47:13.946164 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:13.946081    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c370a0b-7c18-46a6-8748-92c92b46dca0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e3a5f961-mzrdk\" (UID: \"8c370a0b-7c18-46a6-8748-92c92b46dca0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3a5f961-mzrdk"
Oct 31 12:47:13.946338 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:13.946267    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c370a0b-7c18-46a6-8748-92c92b46dca0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e3a5f961-mzrdk\" (UID: \"8c370a0b-7c18-46a6-8748-92c92b46dca0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3a5f961-mzrdk"
Oct 31 12:47:13.946338 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:13.946325    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wl27k\" (UniqueName: \"kubernetes.io/projected/8c370a0b-7c18-46a6-8748-92c92b46dca0-kube-api-access-wl27k\") pod \"maas-api-revoke-sub-e3a5f961-mzrdk\" (UID: \"8c370a0b-7c18-46a6-8748-92c92b46dca0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3a5f961-mzrdk"
Oct 31 12:47:13.946961 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:13.946930    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c370a0b-7c18-46a6-8748-92c92b46dca0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e3a5f961-mzrdk\" (UID: \"8c370a0b-7c18-46a6-8748-92c92b46dca0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3a5f961-mzrdk"
Oct 31 12:47:13.948871 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:13.948849    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c370a0b-7c18-46a6-8748-92c92b46dca0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e3a5f961-mzrdk\" (UID: \"8c370a0b-7c18-46a6-8748-92c92b46dca0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3a5f961-mzrdk"
Oct 31 12:47:13.972430 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:13.972400    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wl27k\" (UniqueName: \"kubernetes.io/projected/8c370a0b-7c18-46a6-8748-92c92b46dca0-kube-api-access-wl27k\") pod \"maas-api-revoke-sub-e3a5f961-mzrdk\" (UID: \"8c370a0b-7c18-46a6-8748-92c92b46dca0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3a5f961-mzrdk"
Oct 31 12:47:14.036463 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:14.036430    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3a5f961-mzrdk"
Oct 31 12:47:14.378806 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:14.378329    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3a5f961-mzrdk"]
Oct 31 12:47:14.763478 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:14.763446    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3a5f961-mzrdk" event={"ID":"8c370a0b-7c18-46a6-8748-92c92b46dca0","Type":"ContainerStarted","Data":"65bfd9751b6ae33da28125d41d2d5c8411aab3560b9e4629f64cc5fc5c08f431"}
Oct 31 12:47:15.780421 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:15.780369    5060 generic.go:356] "Generic (PLEG): container finished" podID="1af894f4-5dbb-495a-b896-670ce365742e" containerID="2f2f93ba9ba80135ad1f844874a72f1bfa14b5aeb5af28cf429848555803afdb" exitCode=0
Oct 31 12:47:15.780975 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:15.780440    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f22e7f-openshift-default-84f9758845-24j6m" event={"ID":"1af894f4-5dbb-495a-b896-670ce365742e","Type":"ContainerDied","Data":"2f2f93ba9ba80135ad1f844874a72f1bfa14b5aeb5af28cf429848555803afdb"}
Oct 31 12:47:15.781031 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:15.780981    5060 scope.go:122] "RemoveContainer" containerID="2f2f93ba9ba80135ad1f844874a72f1bfa14b5aeb5af28cf429848555803afdb"
Oct 31 12:47:15.782923 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:15.782890    5060 generic.go:356] "Generic (PLEG): container finished" podID="8c370a0b-7c18-46a6-8748-92c92b46dca0" containerID="b82573bb3be8c333d80ccab70db9a2931ba3f9f3baa15306106cb82bcc215295" exitCode=0
Oct 31 12:47:15.783104 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:15.782956    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3a5f961-mzrdk" event={"ID":"8c370a0b-7c18-46a6-8748-92c92b46dca0","Type":"ContainerDied","Data":"b82573bb3be8c333d80ccab70db9a2931ba3f9f3baa15306106cb82bcc215295"}
Oct 31 12:47:15.783214 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:15.783132    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:47:15.783289 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:15.783223    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:47:15.783289 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:15.783273    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:47:16.803378 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:16.803346    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f22e7f-openshift-default-84f9758845-24j6m" event={"ID":"1af894f4-5dbb-495a-b896-670ce365742e","Type":"ContainerStarted","Data":"baf84e1b4a76f84270c7514aa2ef620831892521cd9d798f67f36272ba414bf7"}
Oct 31 12:47:17.033653 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:17.033618    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6adf0260-kcxpx"]
Oct 31 12:47:17.065071 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:47:17.064976    5060 status_manager.go:1045] "Failed to get status for pod" err="pods \"maas-api-revoke-sub-6adf0260-kcxpx\" is forbidden: User \"system:node:ip-10-0-130-190.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"odh-ai-gateway-infra\": no relationship found between node 'ip-10-0-130-190.us-east-2.compute.internal' and this object" podUID="d21ba884-a8a6-4197-8b2a-f8373396ec51" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6adf0260-kcxpx"
Oct 31 12:47:17.099052 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:17.098814    5060 patch_prober.go:29] interesting pod/e2e-api-a-f22e7f-openshift-default-84f9758845-24j6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body=
Oct 31 12:47:17.099052 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:17.098888    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f22e7f-openshift-default-84f9758845-24j6m" podUID="1af894f4-5dbb-495a-b896-670ce365742e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused"
Oct 31 12:47:17.100861 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:17.100829    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6adf0260-kcxpx"]
Oct 31 12:47:17.101018 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:17.100884    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-f22e7f-openshift-default-84f9758845-24j6m"
Oct 31 12:47:17.101969 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:17.101947    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3a5f961-mzrdk"
Oct 31 12:47:17.244019 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:17.241928    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c370a0b-7c18-46a6-8748-92c92b46dca0-kube-api-access-wl27k\" (UniqueName: \"kubernetes.io/projected/8c370a0b-7c18-46a6-8748-92c92b46dca0-kube-api-access-wl27k\") pod \"8c370a0b-7c18-46a6-8748-92c92b46dca0\" (UID: \"8c370a0b-7c18-46a6-8748-92c92b46dca0\") "
Oct 31 12:47:17.244019 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:17.242305    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8c370a0b-7c18-46a6-8748-92c92b46dca0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c370a0b-7c18-46a6-8748-92c92b46dca0-maas-api-xks-ca\") pod \"8c370a0b-7c18-46a6-8748-92c92b46dca0\" (UID: \"8c370a0b-7c18-46a6-8748-92c92b46dca0\") "
Oct 31 12:47:17.244019 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:17.242387    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c370a0b-7c18-46a6-8748-92c92b46dca0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c370a0b-7c18-46a6-8748-92c92b46dca0-maas-api-openshift-ca\") pod \"8c370a0b-7c18-46a6-8748-92c92b46dca0\" (UID: \"8c370a0b-7c18-46a6-8748-92c92b46dca0\") "
Oct 31 12:47:17.413315 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:17.400965    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c370a0b-7c18-46a6-8748-92c92b46dca0-maas-api-openshift-ca" pod "8c370a0b-7c18-46a6-8748-92c92b46dca0" (UID: "8c370a0b-7c18-46a6-8748-92c92b46dca0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:47:17.421057 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:17.421015    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c370a0b-7c18-46a6-8748-92c92b46dca0-maas-api-xks-ca" pod "8c370a0b-7c18-46a6-8748-92c92b46dca0" (UID: "8c370a0b-7c18-46a6-8748-92c92b46dca0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:47:17.436083 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:17.436050    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c370a0b-7c18-46a6-8748-92c92b46dca0-kube-api-access-wl27k" pod "8c370a0b-7c18-46a6-8748-92c92b46dca0" (UID: "8c370a0b-7c18-46a6-8748-92c92b46dca0"). InnerVolumeSpecName "kube-api-access-wl27k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:47:17.451021 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:17.450992    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c370a0b-7c18-46a6-8748-92c92b46dca0-maas-api-xks-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:47:17.451178 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:17.451023    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c370a0b-7c18-46a6-8748-92c92b46dca0-maas-api-openshift-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:47:17.451178 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:17.451041    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wl27k\" (UniqueName: \"kubernetes.io/projected/8c370a0b-7c18-46a6-8748-92c92b46dca0-kube-api-access-wl27k\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:47:17.547692 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:17.547659    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pcvvm_f032909d-bc39-4c9e-99b7-f3d800842119/console-operator/2.log"
Oct 31 12:47:17.552852 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:17.552824    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pcvvm_f032909d-bc39-4c9e-99b7-f3d800842119/console-operator/2.log"
Oct 31 12:47:17.832553 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:17.832261    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3a5f961-mzrdk"
Oct 31 12:47:17.832553 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:17.832400    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3a5f961-mzrdk" event={"ID":"8c370a0b-7c18-46a6-8748-92c92b46dca0","Type":"ContainerDied","Data":"65bfd9751b6ae33da28125d41d2d5c8411aab3560b9e4629f64cc5fc5c08f431"}
Oct 31 12:47:17.832553 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:17.832439    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="65bfd9751b6ae33da28125d41d2d5c8411aab3560b9e4629f64cc5fc5c08f431"
Oct 31 12:47:18.099114 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:18.099034    5060 patch_prober.go:29] interesting pod/e2e-api-a-f22e7f-openshift-default-84f9758845-24j6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body=
Oct 31 12:47:18.099114 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:18.099091    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f22e7f-openshift-default-84f9758845-24j6m" podUID="1af894f4-5dbb-495a-b896-670ce365742e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused"
Oct 31 12:47:18.986244 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:18.977600    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-f22e7f-openshift-default-84f9758845-24j6m"
Oct 31 12:47:19.030641 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:19.030598    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fbc4381f-ctm7g"]
Oct 31 12:47:19.112706 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:19.106045    5060 patch_prober.go:29] interesting pod/e2e-api-a-f22e7f-openshift-default-84f9758845-24j6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body=
Oct 31 12:47:19.112706 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:19.106118    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f22e7f-openshift-default-84f9758845-24j6m" podUID="1af894f4-5dbb-495a-b896-670ce365742e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused"
Oct 31 12:47:19.119188 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:19.119144    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d21ba884-a8a6-4197-8b2a-f8373396ec51" path="/var/lib/kubelet/pods/d21ba884-a8a6-4197-8b2a-f8373396ec51/volumes"
Oct 31 12:47:19.119772 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:19.119745    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fbc4381f-ctm7g"]
Oct 31 12:47:20.101428 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:20.100886    5060 patch_prober.go:29] interesting pod/e2e-api-a-f22e7f-openshift-default-84f9758845-24j6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body=
Oct 31 12:47:20.101428 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:20.100956    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f22e7f-openshift-default-84f9758845-24j6m" podUID="1af894f4-5dbb-495a-b896-670ce365742e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused"
Oct 31 12:47:20.147932 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:20.147889    5060 scope.go:122] "RemoveContainer" containerID="b72780e0b86d0ec07ac0274ca6268ef2e0924a34df4b4ef8f569cee1b098f8f9"
Oct 31 12:47:20.181703 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:20.180642    5060 scope.go:122] "RemoveContainer" containerID="d4df205b92185a7af1fe7d1d0aa370404833b9ec7b2632649dab183491e6d297"
Oct 31 12:47:20.209744 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:20.209467    5060 scope.go:122] "RemoveContainer" containerID="b615ac3dcc628fa62bac5a336ad513972f03503c5b23495f22297bc1efd6eccb"
Oct 31 12:47:20.229025 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:20.229000    5060 scope.go:122] "RemoveContainer" containerID="27a99613806fcc2aac38ad5e01181eab7b713f52dd347608cf7fc78045b441ff"
Oct 31 12:47:20.250037 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:20.250017    5060 scope.go:122] "RemoveContainer" containerID="32249e761326f57144a6cce55f8bff9d6e63a69e7be5c737b65f7d73b52b8d8e"
Oct 31 12:47:20.317654 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:20.317627    5060 scope.go:122] "RemoveContainer" containerID="6dc01d41f18979e1ae088ac419af21a6e1c2691a0660b25e1a4f2ee98a2496b8"
Oct 31 12:47:21.099517 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:21.098631    5060 patch_prober.go:29] interesting pod/e2e-api-a-f22e7f-openshift-default-84f9758845-24j6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body=
Oct 31 12:47:21.099517 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:21.098705    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f22e7f-openshift-default-84f9758845-24j6m" podUID="1af894f4-5dbb-495a-b896-670ce365742e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused"
Oct 31 12:47:21.106310 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:21.106282    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99d41b7a-82d6-4a10-a059-954635516f68" path="/var/lib/kubelet/pods/99d41b7a-82d6-4a10-a059-954635516f68/volumes"
Oct 31 12:47:21.107030 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:21.107006    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a060601-ghnt5"]
Oct 31 12:47:21.109693 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:21.109627    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a060601-ghnt5"]
Oct 31 12:47:22.097816 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:22.097779    5060 patch_prober.go:29] interesting pod/e2e-api-a-f22e7f-openshift-default-84f9758845-24j6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body=
Oct 31 12:47:22.098008 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:22.097829    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f22e7f-openshift-default-84f9758845-24j6m" podUID="1af894f4-5dbb-495a-b896-670ce365742e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused"
Oct 31 12:47:23.051620 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:23.051580    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94144e6d-f757-4ebf-99c1-212dcf43fd9b" path="/var/lib/kubelet/pods/94144e6d-f757-4ebf-99c1-212dcf43fd9b/volumes"
Oct 31 12:47:23.305346 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:23.305267    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-f22e7f-openshift-default-84f9758845-24j6m"
Oct 31 12:47:23.307615 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:23.307590    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-f22e7f-openshift-default-84f9758845-24j6m"
Oct 31 12:47:26.520519 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:26.518153    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-787316-openshift-default-58b9b668c5-qpf4t"]
Oct 31 12:47:26.520519 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:26.519517    5060 cpu_manager.go:414] "containerMap: removing container" podUID="8c370a0b-7c18-46a6-8748-92c92b46dca0" containerName="revoke-keys"
Oct 31 12:47:26.520519 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:26.519542    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c370a0b-7c18-46a6-8748-92c92b46dca0" containerName="revoke-keys"
Oct 31 12:47:26.520519 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:26.519788    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c370a0b-7c18-46a6-8748-92c92b46dca0" containerName="revoke-keys"
Oct 31 12:47:26.737105 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:26.737075    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-787316-openshift-default-58b9b668c5-qpf4t"
Oct 31 12:47:26.745737 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:26.745705    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-787316-openshift-default-dockercfg-bnlw6\""
Oct 31 12:47:26.777588 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:26.777556    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-787316-openshift-default-58b9b668c5-qpf4t"]
Oct 31 12:47:26.845316 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:26.845005    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4e57789-6b1a-4d17-bdaf-925a3d3d0c43-workload-certs\") pod \"e2e-api-b-787316-openshift-default-58b9b668c5-qpf4t\" (UID: \"b4e57789-6b1a-4d17-bdaf-925a3d3d0c43\") " pod="openshift-ingress/e2e-api-b-787316-openshift-default-58b9b668c5-qpf4t"
Oct 31 12:47:26.845316 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:26.845077    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4e57789-6b1a-4d17-bdaf-925a3d3d0c43-workload-socket\") pod \"e2e-api-b-787316-openshift-default-58b9b668c5-qpf4t\" (UID: \"b4e57789-6b1a-4d17-bdaf-925a3d3d0c43\") " pod="openshift-ingress/e2e-api-b-787316-openshift-default-58b9b668c5-qpf4t"
Oct 31 12:47:26.845316 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:26.845199    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4e57789-6b1a-4d17-bdaf-925a3d3d0c43-istiod-ca-cert\") pod \"e2e-api-b-787316-openshift-default-58b9b668c5-qpf4t\" (UID: \"b4e57789-6b1a-4d17-bdaf-925a3d3d0c43\") " pod="openshift-ingress/e2e-api-b-787316-openshift-default-58b9b668c5-qpf4t"
Oct 31 12:47:26.845316 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:26.845268    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4e57789-6b1a-4d17-bdaf-925a3d3d0c43-istio-data\") pod \"e2e-api-b-787316-openshift-default-58b9b668c5-qpf4t\" (UID: \"b4e57789-6b1a-4d17-bdaf-925a3d3d0c43\") " pod="openshift-ingress/e2e-api-b-787316-openshift-default-58b9b668c5-qpf4t"
Oct 31 12:47:26.845316 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:26.845314    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4e57789-6b1a-4d17-bdaf-925a3d3d0c43-credential-socket\") pod \"e2e-api-b-787316-openshift-default-58b9b668c5-qpf4t\" (UID: \"b4e57789-6b1a-4d17-bdaf-925a3d3d0c43\") " pod="openshift-ingress/e2e-api-b-787316-openshift-default-58b9b668c5-qpf4t"
Oct 31 12:47:26.845819 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:26.845383    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5z4hd\" (UniqueName: \"kubernetes.io/projected/b4e57789-6b1a-4d17-bdaf-925a3d3d0c43-kube-api-access-5z4hd\") pod \"e2e-api-b-787316-openshift-default-58b9b668c5-qpf4t\" (UID: \"b4e57789-6b1a-4d17-bdaf-925a3d3d0c43\") " pod="openshift-ingress/e2e-api-b-787316-openshift-default-58b9b668c5-qpf4t"
Oct 31 12:47:26.845819 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:26.845456    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4e57789-6b1a-4d17-bdaf-925a3d3d0c43-istio-token\") pod \"e2e-api-b-787316-openshift-default-58b9b668c5-qpf4t\" (UID: \"b4e57789-6b1a-4d17-bdaf-925a3d3d0c43\") " pod="openshift-ingress/e2e-api-b-787316-openshift-default-58b9b668c5-qpf4t"
Oct 31 12:47:26.845819 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:26.845526    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4e57789-6b1a-4d17-bdaf-925a3d3d0c43-istio-envoy\") pod \"e2e-api-b-787316-openshift-default-58b9b668c5-qpf4t\" (UID: \"b4e57789-6b1a-4d17-bdaf-925a3d3d0c43\") " pod="openshift-ingress/e2e-api-b-787316-openshift-default-58b9b668c5-qpf4t"
Oct 31 12:47:26.845819 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:26.845689    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4e57789-6b1a-4d17-bdaf-925a3d3d0c43-istio-podinfo\") pod \"e2e-api-b-787316-openshift-default-58b9b668c5-qpf4t\" (UID: \"b4e57789-6b1a-4d17-bdaf-925a3d3d0c43\") " pod="openshift-ingress/e2e-api-b-787316-openshift-default-58b9b668c5-qpf4t"
Oct 31 12:47:26.949963 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:26.949924    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5z4hd\" (UniqueName: \"kubernetes.io/projected/b4e57789-6b1a-4d17-bdaf-925a3d3d0c43-kube-api-access-5z4hd\") pod \"e2e-api-b-787316-openshift-default-58b9b668c5-qpf4t\" (UID: \"b4e57789-6b1a-4d17-bdaf-925a3d3d0c43\") " pod="openshift-ingress/e2e-api-b-787316-openshift-default-58b9b668c5-qpf4t"
Oct 31 12:47:26.950704 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:26.950671    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4e57789-6b1a-4d17-bdaf-925a3d3d0c43-istio-token\") pod \"e2e-api-b-787316-openshift-default-58b9b668c5-qpf4t\" (UID: \"b4e57789-6b1a-4d17-bdaf-925a3d3d0c43\") " pod="openshift-ingress/e2e-api-b-787316-openshift-default-58b9b668c5-qpf4t"
Oct 31 12:47:26.951239 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:26.951219    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4e57789-6b1a-4d17-bdaf-925a3d3d0c43-istio-envoy\") pod \"e2e-api-b-787316-openshift-default-58b9b668c5-qpf4t\" (UID: \"b4e57789-6b1a-4d17-bdaf-925a3d3d0c43\") " pod="openshift-ingress/e2e-api-b-787316-openshift-default-58b9b668c5-qpf4t"
Oct 31 12:47:26.951631 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:26.951601    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4e57789-6b1a-4d17-bdaf-925a3d3d0c43-istio-podinfo\") pod \"e2e-api-b-787316-openshift-default-58b9b668c5-qpf4t\" (UID: \"b4e57789-6b1a-4d17-bdaf-925a3d3d0c43\") " pod="openshift-ingress/e2e-api-b-787316-openshift-default-58b9b668c5-qpf4t"
Oct 31 12:47:26.951851 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:26.951828    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4e57789-6b1a-4d17-bdaf-925a3d3d0c43-workload-certs\") pod \"e2e-api-b-787316-openshift-default-58b9b668c5-qpf4t\" (UID: \"b4e57789-6b1a-4d17-bdaf-925a3d3d0c43\") " pod="openshift-ingress/e2e-api-b-787316-openshift-default-58b9b668c5-qpf4t"
Oct 31 12:47:26.952024 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:26.952008    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4e57789-6b1a-4d17-bdaf-925a3d3d0c43-workload-socket\") pod \"e2e-api-b-787316-openshift-default-58b9b668c5-qpf4t\" (UID: \"b4e57789-6b1a-4d17-bdaf-925a3d3d0c43\") " pod="openshift-ingress/e2e-api-b-787316-openshift-default-58b9b668c5-qpf4t"
Oct 31 12:47:26.952235 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:26.952221    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4e57789-6b1a-4d17-bdaf-925a3d3d0c43-istiod-ca-cert\") pod \"e2e-api-b-787316-openshift-default-58b9b668c5-qpf4t\" (UID: \"b4e57789-6b1a-4d17-bdaf-925a3d3d0c43\") " pod="openshift-ingress/e2e-api-b-787316-openshift-default-58b9b668c5-qpf4t"
Oct 31 12:47:26.952390 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:26.952372    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4e57789-6b1a-4d17-bdaf-925a3d3d0c43-istio-data\") pod \"e2e-api-b-787316-openshift-default-58b9b668c5-qpf4t\" (UID: \"b4e57789-6b1a-4d17-bdaf-925a3d3d0c43\") " pod="openshift-ingress/e2e-api-b-787316-openshift-default-58b9b668c5-qpf4t"
Oct 31 12:47:26.952546 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:26.952530    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4e57789-6b1a-4d17-bdaf-925a3d3d0c43-credential-socket\") pod \"e2e-api-b-787316-openshift-default-58b9b668c5-qpf4t\" (UID: \"b4e57789-6b1a-4d17-bdaf-925a3d3d0c43\") " pod="openshift-ingress/e2e-api-b-787316-openshift-default-58b9b668c5-qpf4t"
Oct 31 12:47:26.957514 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:26.953253    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4e57789-6b1a-4d17-bdaf-925a3d3d0c43-credential-socket\") pod \"e2e-api-b-787316-openshift-default-58b9b668c5-qpf4t\" (UID: \"b4e57789-6b1a-4d17-bdaf-925a3d3d0c43\") " pod="openshift-ingress/e2e-api-b-787316-openshift-default-58b9b668c5-qpf4t"
Oct 31 12:47:26.962037 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:26.957939    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4e57789-6b1a-4d17-bdaf-925a3d3d0c43-workload-certs\") pod \"e2e-api-b-787316-openshift-default-58b9b668c5-qpf4t\" (UID: \"b4e57789-6b1a-4d17-bdaf-925a3d3d0c43\") " pod="openshift-ingress/e2e-api-b-787316-openshift-default-58b9b668c5-qpf4t"
Oct 31 12:47:26.962292 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:26.959882    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4e57789-6b1a-4d17-bdaf-925a3d3d0c43-workload-socket\") pod \"e2e-api-b-787316-openshift-default-58b9b668c5-qpf4t\" (UID: \"b4e57789-6b1a-4d17-bdaf-925a3d3d0c43\") " pod="openshift-ingress/e2e-api-b-787316-openshift-default-58b9b668c5-qpf4t"
Oct 31 12:47:26.966510 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:26.963678    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4e57789-6b1a-4d17-bdaf-925a3d3d0c43-istio-data\") pod \"e2e-api-b-787316-openshift-default-58b9b668c5-qpf4t\" (UID: \"b4e57789-6b1a-4d17-bdaf-925a3d3d0c43\") " pod="openshift-ingress/e2e-api-b-787316-openshift-default-58b9b668c5-qpf4t"
Oct 31 12:47:26.992551 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:26.971950    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4e57789-6b1a-4d17-bdaf-925a3d3d0c43-istiod-ca-cert\") pod \"e2e-api-b-787316-openshift-default-58b9b668c5-qpf4t\" (UID: \"b4e57789-6b1a-4d17-bdaf-925a3d3d0c43\") " pod="openshift-ingress/e2e-api-b-787316-openshift-default-58b9b668c5-qpf4t"
Oct 31 12:47:26.992551 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:26.986151    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4e57789-6b1a-4d17-bdaf-925a3d3d0c43-istio-envoy\") pod \"e2e-api-b-787316-openshift-default-58b9b668c5-qpf4t\" (UID: \"b4e57789-6b1a-4d17-bdaf-925a3d3d0c43\") " pod="openshift-ingress/e2e-api-b-787316-openshift-default-58b9b668c5-qpf4t"
Oct 31 12:47:26.994293 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:26.993700    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5z4hd\" (UniqueName: \"kubernetes.io/projected/b4e57789-6b1a-4d17-bdaf-925a3d3d0c43-kube-api-access-5z4hd\") pod \"e2e-api-b-787316-openshift-default-58b9b668c5-qpf4t\" (UID: \"b4e57789-6b1a-4d17-bdaf-925a3d3d0c43\") " pod="openshift-ingress/e2e-api-b-787316-openshift-default-58b9b668c5-qpf4t"
Oct 31 12:47:26.994293 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:26.994231    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4e57789-6b1a-4d17-bdaf-925a3d3d0c43-istio-podinfo\") pod \"e2e-api-b-787316-openshift-default-58b9b668c5-qpf4t\" (UID: \"b4e57789-6b1a-4d17-bdaf-925a3d3d0c43\") " pod="openshift-ingress/e2e-api-b-787316-openshift-default-58b9b668c5-qpf4t"
Oct 31 12:47:26.997241 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:26.997210    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4e57789-6b1a-4d17-bdaf-925a3d3d0c43-istio-token\") pod \"e2e-api-b-787316-openshift-default-58b9b668c5-qpf4t\" (UID: \"b4e57789-6b1a-4d17-bdaf-925a3d3d0c43\") " pod="openshift-ingress/e2e-api-b-787316-openshift-default-58b9b668c5-qpf4t"
Oct 31 12:47:27.035218 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:27.035140    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a614abb-6dw8n"]
Oct 31 12:47:27.057837 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:47:27.057767    5060 status_manager.go:1045] "Failed to get status for pod" err="pods \"maas-api-revoke-sub-8a614abb-6dw8n\" is forbidden: User \"system:node:ip-10-0-130-190.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"odh-ai-gateway-infra\": no relationship found between node 'ip-10-0-130-190.us-east-2.compute.internal' and this object" podUID="622ca715-506b-4684-96c1-a59c9ed0fe77" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a614abb-6dw8n"
Oct 31 12:47:27.067338 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:27.067305    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a614abb-6dw8n"]
Oct 31 12:47:27.070769 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:27.070743    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-787316-openshift-default-58b9b668c5-qpf4t"
Oct 31 12:47:27.409240 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:27.408552    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:47:27.409240 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:27.408655    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:47:27.409240 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:27.408702    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:47:27.424304 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:27.417232    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-787316-openshift-default-58b9b668c5-qpf4t"]
Oct 31 12:47:28.249510 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:28.249409    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-787316-openshift-default-58b9b668c5-qpf4t" event={"ID":"b4e57789-6b1a-4d17-bdaf-925a3d3d0c43","Type":"ContainerStarted","Data":"a4d096354cb95f71298f5e7a682f3e560e1fa6bb66078dbb6336fb4812ff473d"}
Oct 31 12:47:28.249510 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:28.249471    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-787316-openshift-default-58b9b668c5-qpf4t" event={"ID":"b4e57789-6b1a-4d17-bdaf-925a3d3d0c43","Type":"ContainerStarted","Data":"825945ecadf2fb01b08ecd079c5842bc1db0628fea8426e4dd4a234a5eb66a4c"}
Oct 31 12:47:28.281299 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:28.281244    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-787316-openshift-default-58b9b668c5-qpf4t" podStartSLOduration=2.281230051 podStartE2EDuration="2.281230051s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:47:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:47:28.279609328 +0000 UTC m=+1211.382677043" watchObservedRunningTime="2026-10-31 12:47:28.281230051 +0000 UTC m=+1211.384297826"
Oct 31 12:47:28.931525 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:28.926044    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c59c634d-jprkg"]
Oct 31 12:47:28.990331 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:28.985602    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c59c634d-jprkg"
Oct 31 12:47:29.017256 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:29.006084    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7wsmb\""
Oct 31 12:47:29.017256 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:29.006145    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 12:47:29.017256 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:29.014207    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 12:47:29.017256 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:29.014313    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 12:47:29.017256 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:29.014470    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c59c634d-jprkg"]
Oct 31 12:47:29.041817 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:29.037823    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3099e3c-9650-4241-93fd-a5236dbddd5c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c59c634d-jprkg\" (UID: \"e3099e3c-9650-4241-93fd-a5236dbddd5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c59c634d-jprkg"
Oct 31 12:47:29.041817 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:29.037953    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhpf6\" (UniqueName: \"kubernetes.io/projected/e3099e3c-9650-4241-93fd-a5236dbddd5c-kube-api-access-bhpf6\") pod \"maas-api-revoke-sub-c59c634d-jprkg\" (UID: \"e3099e3c-9650-4241-93fd-a5236dbddd5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c59c634d-jprkg"
Oct 31 12:47:29.041817 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:29.038023    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3099e3c-9650-4241-93fd-a5236dbddd5c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c59c634d-jprkg\" (UID: \"e3099e3c-9650-4241-93fd-a5236dbddd5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c59c634d-jprkg"
Oct 31 12:47:29.057121 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:29.057087    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="622ca715-506b-4684-96c1-a59c9ed0fe77" path="/var/lib/kubelet/pods/622ca715-506b-4684-96c1-a59c9ed0fe77/volumes"
Oct 31 12:47:29.071190 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:29.071162    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-787316-openshift-default-58b9b668c5-qpf4t"
Oct 31 12:47:29.077157 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:29.077129    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-787316-openshift-default-58b9b668c5-qpf4t"
Oct 31 12:47:29.139538 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:29.139459    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3099e3c-9650-4241-93fd-a5236dbddd5c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c59c634d-jprkg\" (UID: \"e3099e3c-9650-4241-93fd-a5236dbddd5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c59c634d-jprkg"
Oct 31 12:47:29.139795 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:29.139599    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhpf6\" (UniqueName: \"kubernetes.io/projected/e3099e3c-9650-4241-93fd-a5236dbddd5c-kube-api-access-bhpf6\") pod \"maas-api-revoke-sub-c59c634d-jprkg\" (UID: \"e3099e3c-9650-4241-93fd-a5236dbddd5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c59c634d-jprkg"
Oct 31 12:47:29.139795 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:29.139655    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3099e3c-9650-4241-93fd-a5236dbddd5c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c59c634d-jprkg\" (UID: \"e3099e3c-9650-4241-93fd-a5236dbddd5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c59c634d-jprkg"
Oct 31 12:47:29.140461 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:29.140435    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3099e3c-9650-4241-93fd-a5236dbddd5c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c59c634d-jprkg\" (UID: \"e3099e3c-9650-4241-93fd-a5236dbddd5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c59c634d-jprkg"
Oct 31 12:47:29.142024 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:29.142001    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3099e3c-9650-4241-93fd-a5236dbddd5c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c59c634d-jprkg\" (UID: \"e3099e3c-9650-4241-93fd-a5236dbddd5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c59c634d-jprkg"
Oct 31 12:47:29.152514 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:29.152467    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhpf6\" (UniqueName: \"kubernetes.io/projected/e3099e3c-9650-4241-93fd-a5236dbddd5c-kube-api-access-bhpf6\") pod \"maas-api-revoke-sub-c59c634d-jprkg\" (UID: \"e3099e3c-9650-4241-93fd-a5236dbddd5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c59c634d-jprkg"
Oct 31 12:47:29.264425 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:29.264338    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-787316-openshift-default-58b9b668c5-qpf4t"
Oct 31 12:47:29.265571 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:29.265540    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-787316-openshift-default-58b9b668c5-qpf4t"
Oct 31 12:47:29.356224 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:29.356187    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c59c634d-jprkg"
Oct 31 12:47:29.846160 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:47:29.846129    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3099e3c_9650_4241_93fd_a5236dbddd5c.slice/crio-d9b8255c99a580f3ec896b6b13809260a3c1a0ef623dc6a45b1e01f9253bc4b8 WatchSource:0}: Error finding container d9b8255c99a580f3ec896b6b13809260a3c1a0ef623dc6a45b1e01f9253bc4b8: Status 404 returned error can't find the container with id d9b8255c99a580f3ec896b6b13809260a3c1a0ef623dc6a45b1e01f9253bc4b8
Oct 31 12:47:29.847308 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:29.847281    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c59c634d-jprkg"]
Oct 31 12:47:30.672617 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:30.668949    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c59c634d-jprkg" event={"ID":"e3099e3c-9650-4241-93fd-a5236dbddd5c","Type":"ContainerStarted","Data":"d9b8255c99a580f3ec896b6b13809260a3c1a0ef623dc6a45b1e01f9253bc4b8"}
Oct 31 12:47:30.709412 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:47:30.709333    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7phb" containerName="extractor"
Oct 31 12:47:31.712811 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:31.710567    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c59c634d-jprkg" event={"ID":"e3099e3c-9650-4241-93fd-a5236dbddd5c","Type":"ContainerStarted","Data":"33255c13e0d2189ee73cca104bdebc8a142182d63396105123cd5b7e49177626"}
Oct 31 12:47:31.746957 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:31.746903    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c59c634d-jprkg" podStartSLOduration=3.746890314 podStartE2EDuration="3.746890314s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:47:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:47:31.744821533 +0000 UTC m=+1214.847889245" watchObservedRunningTime="2026-10-31 12:47:31.746890314 +0000 UTC m=+1214.849958027"
Oct 31 12:47:31.780526 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:47:31.779304    5060 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3099e3c_9650_4241_93fd_a5236dbddd5c.slice/crio-conmon-33255c13e0d2189ee73cca104bdebc8a142182d63396105123cd5b7e49177626.scope\": RecentStats: unable to find data in memory cache]"
Oct 31 12:47:31.780526 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:47:31.779461    5060 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3099e3c_9650_4241_93fd_a5236dbddd5c.slice/crio-conmon-33255c13e0d2189ee73cca104bdebc8a142182d63396105123cd5b7e49177626.scope\": RecentStats: unable to find data in memory cache]"
Oct 31 12:47:32.786852 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:32.786804    5060 generic.go:356] "Generic (PLEG): container finished" podID="e3099e3c-9650-4241-93fd-a5236dbddd5c" containerID="33255c13e0d2189ee73cca104bdebc8a142182d63396105123cd5b7e49177626" exitCode=0
Oct 31 12:47:32.788182 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:32.788155    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c59c634d-jprkg" event={"ID":"e3099e3c-9650-4241-93fd-a5236dbddd5c","Type":"ContainerDied","Data":"33255c13e0d2189ee73cca104bdebc8a142182d63396105123cd5b7e49177626"}
Oct 31 12:47:34.428167 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:34.427713    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c59c634d-jprkg"
Oct 31 12:47:34.584510 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:34.582806    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3099e3c-9650-4241-93fd-a5236dbddd5c-kube-api-access-bhpf6\" (UniqueName: \"kubernetes.io/projected/e3099e3c-9650-4241-93fd-a5236dbddd5c-kube-api-access-bhpf6\") pod \"e3099e3c-9650-4241-93fd-a5236dbddd5c\" (UID: \"e3099e3c-9650-4241-93fd-a5236dbddd5c\") "
Oct 31 12:47:34.662524 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:34.662457    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3099e3c-9650-4241-93fd-a5236dbddd5c-kube-api-access-bhpf6" pod "e3099e3c-9650-4241-93fd-a5236dbddd5c" (UID: "e3099e3c-9650-4241-93fd-a5236dbddd5c"). InnerVolumeSpecName "kube-api-access-bhpf6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:47:34.664279 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:34.664218    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e3099e3c-9650-4241-93fd-a5236dbddd5c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3099e3c-9650-4241-93fd-a5236dbddd5c-maas-api-xks-ca\") pod \"e3099e3c-9650-4241-93fd-a5236dbddd5c\" (UID: \"e3099e3c-9650-4241-93fd-a5236dbddd5c\") "
Oct 31 12:47:34.664731 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:34.664704    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3099e3c-9650-4241-93fd-a5236dbddd5c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3099e3c-9650-4241-93fd-a5236dbddd5c-maas-api-openshift-ca\") pod \"e3099e3c-9650-4241-93fd-a5236dbddd5c\" (UID: \"e3099e3c-9650-4241-93fd-a5236dbddd5c\") "
Oct 31 12:47:34.665864 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:34.665824    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bhpf6\" (UniqueName: \"kubernetes.io/projected/e3099e3c-9650-4241-93fd-a5236dbddd5c-kube-api-access-bhpf6\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:47:34.666501 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:34.666464    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3099e3c-9650-4241-93fd-a5236dbddd5c-maas-api-openshift-ca" pod "e3099e3c-9650-4241-93fd-a5236dbddd5c" (UID: "e3099e3c-9650-4241-93fd-a5236dbddd5c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:47:34.702742 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:34.683633    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3099e3c-9650-4241-93fd-a5236dbddd5c-maas-api-xks-ca" pod "e3099e3c-9650-4241-93fd-a5236dbddd5c" (UID: "e3099e3c-9650-4241-93fd-a5236dbddd5c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:47:34.771855 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:34.770717    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3099e3c-9650-4241-93fd-a5236dbddd5c-maas-api-openshift-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:47:34.773719 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:34.773651    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3099e3c-9650-4241-93fd-a5236dbddd5c-maas-api-xks-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:47:34.820392 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:34.820362    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c59c634d-jprkg"
Oct 31 12:47:34.820716 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:34.820682    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c59c634d-jprkg" event={"ID":"e3099e3c-9650-4241-93fd-a5236dbddd5c","Type":"ContainerDied","Data":"d9b8255c99a580f3ec896b6b13809260a3c1a0ef623dc6a45b1e01f9253bc4b8"}
Oct 31 12:47:34.820810 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:34.820738    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d9b8255c99a580f3ec896b6b13809260a3c1a0ef623dc6a45b1e01f9253bc4b8"
Oct 31 12:47:35.811592 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:35.811556    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5d310d-2h8b6"]
Oct 31 12:47:35.821972 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:35.821391    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5d310d-2h8b6"]
Oct 31 12:47:37.060251 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:37.060212    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ebdedf6-8a4c-46bf-b6b4-022472588c22" path="/var/lib/kubelet/pods/6ebdedf6-8a4c-46bf-b6b4-022472588c22/volumes"
Oct 31 12:47:38.189362 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:38.189315    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0258f09-9x2p4"]
Oct 31 12:47:38.290816 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:38.290783    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0258f09-9x2p4"]
Oct 31 12:47:39.054577 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:39.054538    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="930c4fa6-67c2-45f8-ad39-08235a3027ef" path="/var/lib/kubelet/pods/930c4fa6-67c2-45f8-ad39-08235a3027ef/volumes"
Oct 31 12:47:40.067662 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:40.067298    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-873f5159-ldjxk"]
Oct 31 12:47:40.106894 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:40.106820    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-873f5159-ldjxk"]
Oct 31 12:47:40.135420 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:40.135347    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6526221-b77v7"]
Oct 31 12:47:40.167239 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:40.167203    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6526221-b77v7"]
Oct 31 12:47:40.197383 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:40.197341    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x2qzv" podUID="c247ed2a-aa90-424b-9b7e-4f2936d67e3f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded"
Oct 31 12:47:40.466627 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:40.459851    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-2-w5-b25d3b-kserve-76f798d59f-vvzhg" podUID="394ed76c-84ac-4612-9acb-a7e36cb68d89" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": context deadline exceeded"
Oct 31 12:47:41.090515 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:41.087033    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/premium-w2-e6c79d-kserve-6468bf46d4-h66f8" podUID="cd6a266f-8d12-4089-83ee-c75c716bf26c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:47:41.132795 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:41.131938    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2264162-d632-485f-89c6-286c99b30f62" path="/var/lib/kubelet/pods/a2264162-d632-485f-89c6-286c99b30f62/volumes"
Oct 31 12:47:41.134013 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:41.133982    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e695a11f-7212-46ab-b537-a40a639e7a71" path="/var/lib/kubelet/pods/e695a11f-7212-46ab-b537-a40a639e7a71/volumes"
Oct 31 12:47:43.977534 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:43.971261    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-5d310d-openshift-default-7c8bb48c95-n2tkf"]
Oct 31 12:47:43.994871 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:43.994826    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-5d310d-openshift-default-7c8bb48c95-n2tkf" podUID="3e880a6e-3d1c-42e4-a6f3-a7a22c2cc35d" containerName="istio-proxy" containerID="cri-o://4fac2d86dad1ebc23ea5d3f832ba1934f3485799c582624d659485c7ff99a2bd" gracePeriod=30
Oct 31 12:47:44.233623 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:44.233536    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-711af2-mpngx"]
Oct 31 12:47:44.273570 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:44.268784    5060 cpu_manager.go:414] "containerMap: removing container" podUID="e3099e3c-9650-4241-93fd-a5236dbddd5c" containerName="revoke-keys"
Oct 31 12:47:44.273570 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:44.268817    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3099e3c-9650-4241-93fd-a5236dbddd5c" containerName="revoke-keys"
Oct 31 12:47:44.273570 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:44.269119    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3099e3c-9650-4241-93fd-a5236dbddd5c" containerName="revoke-keys"
Oct 31 12:47:44.429922 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:44.422162    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-711af2-mpngx"
Oct 31 12:47:44.433212 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:44.433179    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-711af2-mpngx"]
Oct 31 12:47:44.438567 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:44.438538    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 12:47:44.444563 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:44.443817    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7wsmb\""
Oct 31 12:47:44.444752 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:44.444026    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 12:47:44.448561 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:44.444846    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 12:47:44.680725 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:44.660752    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c874d2e9-5ace-4faf-a1dd-d497e34b73d3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-711af2-mpngx\" (UID: \"c874d2e9-5ace-4faf-a1dd-d497e34b73d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-711af2-mpngx"
Oct 31 12:47:44.685542 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:44.681170    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c874d2e9-5ace-4faf-a1dd-d497e34b73d3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-711af2-mpngx\" (UID: \"c874d2e9-5ace-4faf-a1dd-d497e34b73d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-711af2-mpngx"
Oct 31 12:47:44.714010 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:44.713965    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pndfs\" (UniqueName: \"kubernetes.io/projected/c874d2e9-5ace-4faf-a1dd-d497e34b73d3-kube-api-access-pndfs\") pod \"maas-api-revoke-keys-e2e-api-b-711af2-mpngx\" (UID: \"c874d2e9-5ace-4faf-a1dd-d497e34b73d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-711af2-mpngx"
Oct 31 12:47:44.913595 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:44.898434    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c874d2e9-5ace-4faf-a1dd-d497e34b73d3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-711af2-mpngx\" (UID: \"c874d2e9-5ace-4faf-a1dd-d497e34b73d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-711af2-mpngx"
Oct 31 12:47:44.913595 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:44.898925    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c874d2e9-5ace-4faf-a1dd-d497e34b73d3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-711af2-mpngx\" (UID: \"c874d2e9-5ace-4faf-a1dd-d497e34b73d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-711af2-mpngx"
Oct 31 12:47:44.913595 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:44.899060    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pndfs\" (UniqueName: \"kubernetes.io/projected/c874d2e9-5ace-4faf-a1dd-d497e34b73d3-kube-api-access-pndfs\") pod \"maas-api-revoke-keys-e2e-api-b-711af2-mpngx\" (UID: \"c874d2e9-5ace-4faf-a1dd-d497e34b73d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-711af2-mpngx"
Oct 31 12:47:44.995046 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:44.995005    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c874d2e9-5ace-4faf-a1dd-d497e34b73d3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-711af2-mpngx\" (UID: \"c874d2e9-5ace-4faf-a1dd-d497e34b73d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-711af2-mpngx"
Oct 31 12:47:45.035532 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:45.025293    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pndfs\" (UniqueName: \"kubernetes.io/projected/c874d2e9-5ace-4faf-a1dd-d497e34b73d3-kube-api-access-pndfs\") pod \"maas-api-revoke-keys-e2e-api-b-711af2-mpngx\" (UID: \"c874d2e9-5ace-4faf-a1dd-d497e34b73d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-711af2-mpngx"
Oct 31 12:47:45.061629 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:45.057991    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f22e7f-fw72r"]
Oct 31 12:47:45.110043 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:45.110009    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c874d2e9-5ace-4faf-a1dd-d497e34b73d3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-711af2-mpngx\" (UID: \"c874d2e9-5ace-4faf-a1dd-d497e34b73d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-711af2-mpngx"
Oct 31 12:47:45.210971 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:45.210616    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-711af2-mpngx"
Oct 31 12:47:45.475341 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:45.475251    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f22e7f-fw72r"]
Oct 31 12:47:46.562541 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:46.548072    5060 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-2-w5-b25d3b-kserve-76f798d59f-vvzhg" podUID="394ed76c-84ac-4612-9acb-a7e36cb68d89" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:47:46.839720 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:46.839638    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e1fc0ffe-xvmjr"]
Oct 31 12:47:46.849068 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:46.848987    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e1fc0ffe-xvmjr"]
Oct 31 12:47:47.033800 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:47.033751    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f97bf1ca-swdmw"]
Oct 31 12:47:47.039195 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:47.039168    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f97bf1ca-swdmw"]
Oct 31 12:47:47.071527 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:47.054720    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8126e807-51f6-4f81-82a2-23d9d1a33742" path="/var/lib/kubelet/pods/8126e807-51f6-4f81-82a2-23d9d1a33742/volumes"
Oct 31 12:47:47.128111 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:47.125931    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4db4fbd-149c-4d21-b64e-efcf12e2bc05" path="/var/lib/kubelet/pods/c4db4fbd-149c-4d21-b64e-efcf12e2bc05/volumes"
Oct 31 12:47:47.128111 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:47.126833    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f86d8432-bab6-4a08-a85d-c61726d8c102" path="/var/lib/kubelet/pods/f86d8432-bab6-4a08-a85d-c61726d8c102/volumes"
Oct 31 12:47:47.128111 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:47.127775    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-711af2-mpngx"]
Oct 31 12:47:47.131253 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:47:47.131221    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc874d2e9_5ace_4faf_a1dd_d497e34b73d3.slice/crio-4c46568d55802f7495f8422f5f73f9564f101aeb5c88eb6f5e5c5c00a2c61cbf WatchSource:0}: Error finding container 4c46568d55802f7495f8422f5f73f9564f101aeb5c88eb6f5e5c5c00a2c61cbf: Status 404 returned error can't find the container with id 4c46568d55802f7495f8422f5f73f9564f101aeb5c88eb6f5e5c5c00a2c61cbf
Oct 31 12:47:47.836277 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:47.836185    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-711af2-mpngx" event={"ID":"c874d2e9-5ace-4faf-a1dd-d497e34b73d3","Type":"ContainerStarted","Data":"e42fb51c5312380b1f24ee6c93eb46ca1a7a43bb3fcf2bff8253c47a2fd87f7e"}
Oct 31 12:47:47.836277 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:47.836249    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-711af2-mpngx" event={"ID":"c874d2e9-5ace-4faf-a1dd-d497e34b73d3","Type":"ContainerStarted","Data":"4c46568d55802f7495f8422f5f73f9564f101aeb5c88eb6f5e5c5c00a2c61cbf"}
Oct 31 12:47:47.954564 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:47.908110    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-711af2-mpngx" podStartSLOduration=3.908088364 podStartE2EDuration="3.908088364s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:47:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:47:47.853771765 +0000 UTC m=+1230.956839478" watchObservedRunningTime="2026-10-31 12:47:47.908088364 +0000 UTC m=+1231.011156079"
Oct 31 12:47:48.513113 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:47:48.513067    5060 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc874d2e9_5ace_4faf_a1dd_d497e34b73d3.slice/crio-conmon-e42fb51c5312380b1f24ee6c93eb46ca1a7a43bb3fcf2bff8253c47a2fd87f7e.scope\": RecentStats: unable to find data in memory cache]"
Oct 31 12:47:48.852419 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:48.852387    5060 generic.go:356] "Generic (PLEG): container finished" podID="c874d2e9-5ace-4faf-a1dd-d497e34b73d3" containerID="e42fb51c5312380b1f24ee6c93eb46ca1a7a43bb3fcf2bff8253c47a2fd87f7e" exitCode=0
Oct 31 12:47:48.852836 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:48.852539    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-711af2-mpngx" event={"ID":"c874d2e9-5ace-4faf-a1dd-d497e34b73d3","Type":"ContainerDied","Data":"e42fb51c5312380b1f24ee6c93eb46ca1a7a43bb3fcf2bff8253c47a2fd87f7e"}
Oct 31 12:47:49.587975 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:49.587178    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-5d310d-openshift-default-7c8bb48c95-n2tkf"
Oct 31 12:47:49.686351 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:49.686315    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e880a6e-3d1c-42e4-a6f3-a7a22c2cc35d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e880a6e-3d1c-42e4-a6f3-a7a22c2cc35d-istio-data\") pod \"3e880a6e-3d1c-42e4-a6f3-a7a22c2cc35d\" (UID: \"3e880a6e-3d1c-42e4-a6f3-a7a22c2cc35d\") "
Oct 31 12:47:49.686706 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:49.686682    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e880a6e-3d1c-42e4-a6f3-a7a22c2cc35d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e880a6e-3d1c-42e4-a6f3-a7a22c2cc35d-istio-envoy\") pod \"3e880a6e-3d1c-42e4-a6f3-a7a22c2cc35d\" (UID: \"3e880a6e-3d1c-42e4-a6f3-a7a22c2cc35d\") "
Oct 31 12:47:49.686994 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:49.686964    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3e880a6e-3d1c-42e4-a6f3-a7a22c2cc35d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e880a6e-3d1c-42e4-a6f3-a7a22c2cc35d-istio-podinfo\") pod \"3e880a6e-3d1c-42e4-a6f3-a7a22c2cc35d\" (UID: \"3e880a6e-3d1c-42e4-a6f3-a7a22c2cc35d\") "
Oct 31 12:47:49.687397 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:49.687378    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e880a6e-3d1c-42e4-a6f3-a7a22c2cc35d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e880a6e-3d1c-42e4-a6f3-a7a22c2cc35d-workload-certs\") pod \"3e880a6e-3d1c-42e4-a6f3-a7a22c2cc35d\" (UID: \"3e880a6e-3d1c-42e4-a6f3-a7a22c2cc35d\") "
Oct 31 12:47:49.687569 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:49.687551    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e880a6e-3d1c-42e4-a6f3-a7a22c2cc35d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e880a6e-3d1c-42e4-a6f3-a7a22c2cc35d-credential-socket\") pod \"3e880a6e-3d1c-42e4-a6f3-a7a22c2cc35d\" (UID: \"3e880a6e-3d1c-42e4-a6f3-a7a22c2cc35d\") "
Oct 31 12:47:49.687731 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:49.687712    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e880a6e-3d1c-42e4-a6f3-a7a22c2cc35d-istio-token\" (UniqueName: \"kubernetes.io/projected/3e880a6e-3d1c-42e4-a6f3-a7a22c2cc35d-istio-token\") pod \"3e880a6e-3d1c-42e4-a6f3-a7a22c2cc35d\" (UID: \"3e880a6e-3d1c-42e4-a6f3-a7a22c2cc35d\") "
Oct 31 12:47:49.687866 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:49.687851    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e880a6e-3d1c-42e4-a6f3-a7a22c2cc35d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e880a6e-3d1c-42e4-a6f3-a7a22c2cc35d-istiod-ca-cert\") pod \"3e880a6e-3d1c-42e4-a6f3-a7a22c2cc35d\" (UID: \"3e880a6e-3d1c-42e4-a6f3-a7a22c2cc35d\") "
Oct 31 12:47:49.688089 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:49.688058    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e880a6e-3d1c-42e4-a6f3-a7a22c2cc35d-kube-api-access-6hgrh\" (UniqueName: \"kubernetes.io/projected/3e880a6e-3d1c-42e4-a6f3-a7a22c2cc35d-kube-api-access-6hgrh\") pod \"3e880a6e-3d1c-42e4-a6f3-a7a22c2cc35d\" (UID: \"3e880a6e-3d1c-42e4-a6f3-a7a22c2cc35d\") "
Oct 31 12:47:49.688247 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:49.688231    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e880a6e-3d1c-42e4-a6f3-a7a22c2cc35d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e880a6e-3d1c-42e4-a6f3-a7a22c2cc35d-workload-socket\") pod \"3e880a6e-3d1c-42e4-a6f3-a7a22c2cc35d\" (UID: \"3e880a6e-3d1c-42e4-a6f3-a7a22c2cc35d\") "
Oct 31 12:47:49.689980 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:49.687731    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e880a6e-3d1c-42e4-a6f3-a7a22c2cc35d-istio-data" pod "3e880a6e-3d1c-42e4-a6f3-a7a22c2cc35d" (UID: "3e880a6e-3d1c-42e4-a6f3-a7a22c2cc35d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:47:49.689980 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:49.687900    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e880a6e-3d1c-42e4-a6f3-a7a22c2cc35d-workload-certs" pod "3e880a6e-3d1c-42e4-a6f3-a7a22c2cc35d" (UID: "3e880a6e-3d1c-42e4-a6f3-a7a22c2cc35d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:47:49.689980 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:49.689459    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e880a6e-3d1c-42e4-a6f3-a7a22c2cc35d-credential-socket" pod "3e880a6e-3d1c-42e4-a6f3-a7a22c2cc35d" (UID: "3e880a6e-3d1c-42e4-a6f3-a7a22c2cc35d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:47:49.689980 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:49.689760    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e880a6e-3d1c-42e4-a6f3-a7a22c2cc35d-istiod-ca-cert" pod "3e880a6e-3d1c-42e4-a6f3-a7a22c2cc35d" (UID: "3e880a6e-3d1c-42e4-a6f3-a7a22c2cc35d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:47:49.689980 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:49.689934    5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e880a6e-3d1c-42e4-a6f3-a7a22c2cc35d-istio-data\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:47:49.689980 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:49.689974    5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e880a6e-3d1c-42e4-a6f3-a7a22c2cc35d-workload-certs\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:47:49.690339 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:49.689990    5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e880a6e-3d1c-42e4-a6f3-a7a22c2cc35d-credential-socket\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:47:49.690339 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:49.690004    5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e880a6e-3d1c-42e4-a6f3-a7a22c2cc35d-istiod-ca-cert\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:47:49.691254 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:49.690528    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e880a6e-3d1c-42e4-a6f3-a7a22c2cc35d-workload-socket" pod "3e880a6e-3d1c-42e4-a6f3-a7a22c2cc35d" (UID: "3e880a6e-3d1c-42e4-a6f3-a7a22c2cc35d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:47:49.694842 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:49.694802    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e880a6e-3d1c-42e4-a6f3-a7a22c2cc35d-kube-api-access-6hgrh" pod "3e880a6e-3d1c-42e4-a6f3-a7a22c2cc35d" (UID: "3e880a6e-3d1c-42e4-a6f3-a7a22c2cc35d"). InnerVolumeSpecName "kube-api-access-6hgrh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:47:49.695462 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:49.695436    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e880a6e-3d1c-42e4-a6f3-a7a22c2cc35d-istio-token" pod "3e880a6e-3d1c-42e4-a6f3-a7a22c2cc35d" (UID: "3e880a6e-3d1c-42e4-a6f3-a7a22c2cc35d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:47:49.695596 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:49.695547    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3e880a6e-3d1c-42e4-a6f3-a7a22c2cc35d-istio-podinfo" pod "3e880a6e-3d1c-42e4-a6f3-a7a22c2cc35d" (UID: "3e880a6e-3d1c-42e4-a6f3-a7a22c2cc35d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 12:47:49.697376 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:49.697353    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e880a6e-3d1c-42e4-a6f3-a7a22c2cc35d-istio-envoy" pod "3e880a6e-3d1c-42e4-a6f3-a7a22c2cc35d" (UID: "3e880a6e-3d1c-42e4-a6f3-a7a22c2cc35d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:47:49.792217 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:49.792122    5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e880a6e-3d1c-42e4-a6f3-a7a22c2cc35d-istio-envoy\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:47:49.792217 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:49.792157    5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e880a6e-3d1c-42e4-a6f3-a7a22c2cc35d-istio-podinfo\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:47:49.792217 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:49.792169    5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e880a6e-3d1c-42e4-a6f3-a7a22c2cc35d-istio-token\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:47:49.792217 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:49.792179    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hgrh\" (UniqueName: \"kubernetes.io/projected/3e880a6e-3d1c-42e4-a6f3-a7a22c2cc35d-kube-api-access-6hgrh\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:47:49.792217 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:49.792188    5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e880a6e-3d1c-42e4-a6f3-a7a22c2cc35d-workload-socket\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:47:49.897809 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:49.897767    5060 generic.go:356] "Generic (PLEG): container finished" podID="3e880a6e-3d1c-42e4-a6f3-a7a22c2cc35d" containerID="4fac2d86dad1ebc23ea5d3f832ba1934f3485799c582624d659485c7ff99a2bd" exitCode=0
Oct 31 12:47:49.898460 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:49.897976    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5d310d-openshift-default-7c8bb48c95-n2tkf" event={"ID":"3e880a6e-3d1c-42e4-a6f3-a7a22c2cc35d","Type":"ContainerDied","Data":"4fac2d86dad1ebc23ea5d3f832ba1934f3485799c582624d659485c7ff99a2bd"}
Oct 31 12:47:49.898460 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:49.898040    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5d310d-openshift-default-7c8bb48c95-n2tkf" event={"ID":"3e880a6e-3d1c-42e4-a6f3-a7a22c2cc35d","Type":"ContainerDied","Data":"52d1909554c75a71bc00ec9ae2bc7752793b197eaf51c99f87aadd164f432fe6"}
Oct 31 12:47:49.898460 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:49.898072    5060 scope.go:122] "RemoveContainer" containerID="4fac2d86dad1ebc23ea5d3f832ba1934f3485799c582624d659485c7ff99a2bd"
Oct 31 12:47:49.898460 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:49.898256    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-5d310d-openshift-default-7c8bb48c95-n2tkf"
Oct 31 12:47:49.920517 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:49.920459    5060 scope.go:122] "RemoveContainer" containerID="a7f4a5da16afcef056b5a5fda046ab0306d428991771005d472a0b191d3ec202"
Oct 31 12:47:49.965576 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:49.965538    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-5d310d-openshift-default-7c8bb48c95-n2tkf"]
Oct 31 12:47:49.968574 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:49.968553    5060 scope.go:122] "RemoveContainer" containerID="4fac2d86dad1ebc23ea5d3f832ba1934f3485799c582624d659485c7ff99a2bd"
Oct 31 12:47:49.969197 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:47:49.969162    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4fac2d86dad1ebc23ea5d3f832ba1934f3485799c582624d659485c7ff99a2bd\": container with ID starting with 4fac2d86dad1ebc23ea5d3f832ba1934f3485799c582624d659485c7ff99a2bd not found: ID does not exist" containerID="4fac2d86dad1ebc23ea5d3f832ba1934f3485799c582624d659485c7ff99a2bd"
Oct 31 12:47:49.969296 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:49.969205    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4fac2d86dad1ebc23ea5d3f832ba1934f3485799c582624d659485c7ff99a2bd"} err="failed to get container status \"4fac2d86dad1ebc23ea5d3f832ba1934f3485799c582624d659485c7ff99a2bd\": rpc error: code = NotFound desc = could not find container \"4fac2d86dad1ebc23ea5d3f832ba1934f3485799c582624d659485c7ff99a2bd\": container with ID starting with 4fac2d86dad1ebc23ea5d3f832ba1934f3485799c582624d659485c7ff99a2bd not found: ID does not exist"
Oct 31 12:47:49.969296 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:49.969230    5060 scope.go:122] "RemoveContainer" containerID="a7f4a5da16afcef056b5a5fda046ab0306d428991771005d472a0b191d3ec202"
Oct 31 12:47:49.970033 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:47:49.970004    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7f4a5da16afcef056b5a5fda046ab0306d428991771005d472a0b191d3ec202\": container with ID starting with a7f4a5da16afcef056b5a5fda046ab0306d428991771005d472a0b191d3ec202 not found: ID does not exist" containerID="a7f4a5da16afcef056b5a5fda046ab0306d428991771005d472a0b191d3ec202"
Oct 31 12:47:49.970221 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:49.970041    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7f4a5da16afcef056b5a5fda046ab0306d428991771005d472a0b191d3ec202"} err="failed to get container status \"a7f4a5da16afcef056b5a5fda046ab0306d428991771005d472a0b191d3ec202\": rpc error: code = NotFound desc = could not find container \"a7f4a5da16afcef056b5a5fda046ab0306d428991771005d472a0b191d3ec202\": container with ID starting with a7f4a5da16afcef056b5a5fda046ab0306d428991771005d472a0b191d3ec202 not found: ID does not exist"
Oct 31 12:47:49.970659 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:49.970638    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-5d310d-openshift-default-7c8bb48c95-n2tkf"]
Oct 31 12:47:50.348152 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:50.348120    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-711af2-mpngx"
Oct 31 12:47:50.620585 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:50.605812    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c874d2e9-5ace-4faf-a1dd-d497e34b73d3-kube-api-access-pndfs\" (UniqueName: \"kubernetes.io/projected/c874d2e9-5ace-4faf-a1dd-d497e34b73d3-kube-api-access-pndfs\") pod \"c874d2e9-5ace-4faf-a1dd-d497e34b73d3\" (UID: \"c874d2e9-5ace-4faf-a1dd-d497e34b73d3\") "
Oct 31 12:47:50.620585 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:50.606083    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c874d2e9-5ace-4faf-a1dd-d497e34b73d3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c874d2e9-5ace-4faf-a1dd-d497e34b73d3-maas-api-xks-ca\") pod \"c874d2e9-5ace-4faf-a1dd-d497e34b73d3\" (UID: \"c874d2e9-5ace-4faf-a1dd-d497e34b73d3\") "
Oct 31 12:47:50.620585 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:50.606144    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c874d2e9-5ace-4faf-a1dd-d497e34b73d3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c874d2e9-5ace-4faf-a1dd-d497e34b73d3-maas-api-openshift-ca\") pod \"c874d2e9-5ace-4faf-a1dd-d497e34b73d3\" (UID: \"c874d2e9-5ace-4faf-a1dd-d497e34b73d3\") "
Oct 31 12:47:50.685694 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:50.679391    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c874d2e9-5ace-4faf-a1dd-d497e34b73d3-maas-api-openshift-ca" pod "c874d2e9-5ace-4faf-a1dd-d497e34b73d3" (UID: "c874d2e9-5ace-4faf-a1dd-d497e34b73d3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:47:50.686543 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:50.686468    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c874d2e9-5ace-4faf-a1dd-d497e34b73d3-kube-api-access-pndfs" pod "c874d2e9-5ace-4faf-a1dd-d497e34b73d3" (UID: "c874d2e9-5ace-4faf-a1dd-d497e34b73d3"). InnerVolumeSpecName "kube-api-access-pndfs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:47:50.703854 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:50.703750    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c874d2e9-5ace-4faf-a1dd-d497e34b73d3-maas-api-xks-ca" pod "c874d2e9-5ace-4faf-a1dd-d497e34b73d3" (UID: "c874d2e9-5ace-4faf-a1dd-d497e34b73d3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:47:50.719196 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:50.719140    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c874d2e9-5ace-4faf-a1dd-d497e34b73d3-maas-api-xks-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:47:50.719393 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:50.719203    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c874d2e9-5ace-4faf-a1dd-d497e34b73d3-maas-api-openshift-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:47:50.719393 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:50.719227    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pndfs\" (UniqueName: \"kubernetes.io/projected/c874d2e9-5ace-4faf-a1dd-d497e34b73d3-kube-api-access-pndfs\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:47:51.033996 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:51.033671    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-711af2-mpngx" event={"ID":"c874d2e9-5ace-4faf-a1dd-d497e34b73d3","Type":"ContainerDied","Data":"4c46568d55802f7495f8422f5f73f9564f101aeb5c88eb6f5e5c5c00a2c61cbf"}
Oct 31 12:47:51.034663 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:51.034631    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4c46568d55802f7495f8422f5f73f9564f101aeb5c88eb6f5e5c5c00a2c61cbf"
Oct 31 12:47:51.034881 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:51.034867    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-711af2-mpngx"
Oct 31 12:47:51.110693 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:51.110143    5060 patch_prober.go:29] interesting pod/e2e-api-b-bb3cf5-openshift-default-5fc978478f-mw7qm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 12:47:51.110693 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:51.110228    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-bb3cf5-openshift-default-5fc978478f-mw7qm" podUID="d9cb64fe-d6b4-439d-bc3a-8598b7cd512e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:47:51.119909 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:51.119872    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e880a6e-3d1c-42e4-a6f3-a7a22c2cc35d" path="/var/lib/kubelet/pods/3e880a6e-3d1c-42e4-a6f3-a7a22c2cc35d/volumes"
Oct 31 12:47:54.232564 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:54.232524    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-f22e7f-openshift-default-84f9758845-24j6m"]
Oct 31 12:47:54.233480 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:54.233442    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-f22e7f-openshift-default-84f9758845-24j6m" podUID="1af894f4-5dbb-495a-b896-670ce365742e" containerName="istio-proxy" containerID="cri-o://baf84e1b4a76f84270c7514aa2ef620831892521cd9d798f67f36272ba414bf7" gracePeriod=30
Oct 31 12:47:54.912414 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:54.903057    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bb3cf5-26qt6"]
Oct 31 12:47:54.940548 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:54.938722    5060 cpu_manager.go:414] "containerMap: removing container" podUID="c874d2e9-5ace-4faf-a1dd-d497e34b73d3" containerName="revoke-keys"
Oct 31 12:47:54.940548 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:54.938756    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c874d2e9-5ace-4faf-a1dd-d497e34b73d3" containerName="revoke-keys"
Oct 31 12:47:54.940548 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:54.938775    5060 cpu_manager.go:414] "containerMap: removing container" podUID="3e880a6e-3d1c-42e4-a6f3-a7a22c2cc35d" containerName="istio-proxy"
Oct 31 12:47:54.940548 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:54.938782    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e880a6e-3d1c-42e4-a6f3-a7a22c2cc35d" containerName="istio-proxy"
Oct 31 12:47:54.940548 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:54.938802    5060 cpu_manager.go:414] "containerMap: removing container" podUID="3e880a6e-3d1c-42e4-a6f3-a7a22c2cc35d" containerName="istio-proxy"
Oct 31 12:47:54.940548 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:54.938811    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e880a6e-3d1c-42e4-a6f3-a7a22c2cc35d" containerName="istio-proxy"
Oct 31 12:47:54.940548 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:54.939028    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="c874d2e9-5ace-4faf-a1dd-d497e34b73d3" containerName="revoke-keys"
Oct 31 12:47:54.940548 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:54.939047    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e880a6e-3d1c-42e4-a6f3-a7a22c2cc35d" containerName="istio-proxy"
Oct 31 12:47:54.940548 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:54.939057    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e880a6e-3d1c-42e4-a6f3-a7a22c2cc35d" containerName="istio-proxy"
Oct 31 12:47:54.976819 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:54.963141    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bb3cf5-26qt6"]
Oct 31 12:47:54.976819 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:54.963306    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bb3cf5-26qt6"
Oct 31 12:47:54.976819 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:54.975819    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 12:47:54.976819 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:54.976613    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 12:47:54.976819 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:54.976683    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 12:47:55.062598 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:55.060833    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z95gn\" (UniqueName: \"kubernetes.io/projected/de748339-be67-487e-ba7b-eeaa3ccbc356-kube-api-access-z95gn\") pod \"maas-api-revoke-keys-e2e-api-b-bb3cf5-26qt6\" (UID: \"de748339-be67-487e-ba7b-eeaa3ccbc356\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bb3cf5-26qt6"
Oct 31 12:47:55.062598 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:55.061197    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de748339-be67-487e-ba7b-eeaa3ccbc356-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-bb3cf5-26qt6\" (UID: \"de748339-be67-487e-ba7b-eeaa3ccbc356\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bb3cf5-26qt6"
Oct 31 12:47:55.062598 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:55.061287    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de748339-be67-487e-ba7b-eeaa3ccbc356-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-bb3cf5-26qt6\" (UID: \"de748339-be67-487e-ba7b-eeaa3ccbc356\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bb3cf5-26qt6"
Oct 31 12:47:55.066595 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:55.066564    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-341d3105-c772h"]
Oct 31 12:47:55.066727 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:55.066617    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-341d3105-c772h"]
Oct 31 12:47:55.080343 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:55.080277    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7wsmb\""
Oct 31 12:47:55.166590 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:55.165606    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de748339-be67-487e-ba7b-eeaa3ccbc356-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-bb3cf5-26qt6\" (UID: \"de748339-be67-487e-ba7b-eeaa3ccbc356\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bb3cf5-26qt6"
Oct 31 12:47:55.166590 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:55.165698    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de748339-be67-487e-ba7b-eeaa3ccbc356-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-bb3cf5-26qt6\" (UID: \"de748339-be67-487e-ba7b-eeaa3ccbc356\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bb3cf5-26qt6"
Oct 31 12:47:55.166590 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:55.165779    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z95gn\" (UniqueName: \"kubernetes.io/projected/de748339-be67-487e-ba7b-eeaa3ccbc356-kube-api-access-z95gn\") pod \"maas-api-revoke-keys-e2e-api-b-bb3cf5-26qt6\" (UID: \"de748339-be67-487e-ba7b-eeaa3ccbc356\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bb3cf5-26qt6"
Oct 31 12:47:55.166590 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:55.166300    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de748339-be67-487e-ba7b-eeaa3ccbc356-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-bb3cf5-26qt6\" (UID: \"de748339-be67-487e-ba7b-eeaa3ccbc356\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bb3cf5-26qt6"
Oct 31 12:47:55.250257 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:55.249152    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z95gn\" (UniqueName: \"kubernetes.io/projected/de748339-be67-487e-ba7b-eeaa3ccbc356-kube-api-access-z95gn\") pod \"maas-api-revoke-keys-e2e-api-b-bb3cf5-26qt6\" (UID: \"de748339-be67-487e-ba7b-eeaa3ccbc356\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bb3cf5-26qt6"
Oct 31 12:47:55.585146 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:55.585101    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de748339-be67-487e-ba7b-eeaa3ccbc356-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-bb3cf5-26qt6\" (UID: \"de748339-be67-487e-ba7b-eeaa3ccbc356\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bb3cf5-26qt6"
Oct 31 12:47:55.762294 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:55.761943    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bb3cf5-26qt6"
Oct 31 12:47:56.064660 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:56.064581    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06ffe023-2fswn"]
Oct 31 12:47:56.193913 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:56.108473    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06ffe023-2fswn"]
Oct 31 12:47:56.939350 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:56.938661    5060 patch_prober.go:29] interesting pod/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-hkvl5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 12:47:56.939350 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:56.938728    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-hkvl5" podUID="4e300ab1-6fe3-4548-90b3-0cbb8099e114" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:47:57.078174 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:57.057429    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d243b8a-d515-4b0e-ae52-d74de6e56550" path="/var/lib/kubelet/pods/7d243b8a-d515-4b0e-ae52-d74de6e56550/volumes"
Oct 31 12:47:57.219234 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:57.219149    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da0cf5fb-f52a-4cff-98b8-75ce9fcf48b9" path="/var/lib/kubelet/pods/da0cf5fb-f52a-4cff-98b8-75ce9fcf48b9/volumes"
Oct 31 12:47:57.760470 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:57.760433    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bb3cf5-26qt6"]
Oct 31 12:47:57.809771 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:47:57.809655    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde748339_be67_487e_ba7b_eeaa3ccbc356.slice/crio-781727245fd3327448603cbf3136bae57a39d7dccca07c67c2598641f5b55cff WatchSource:0}: Error finding container 781727245fd3327448603cbf3136bae57a39d7dccca07c67c2598641f5b55cff: Status 404 returned error can't find the container with id 781727245fd3327448603cbf3136bae57a39d7dccca07c67c2598641f5b55cff
Oct 31 12:47:58.241807 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:58.241760    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bb3cf5-26qt6" event={"ID":"de748339-be67-487e-ba7b-eeaa3ccbc356","Type":"ContainerStarted","Data":"781727245fd3327448603cbf3136bae57a39d7dccca07c67c2598641f5b55cff"}
Oct 31 12:47:59.253797 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:59.253761    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bb3cf5-26qt6" event={"ID":"de748339-be67-487e-ba7b-eeaa3ccbc356","Type":"ContainerStarted","Data":"a5457bbc2fa14cea13e8f2a52bd54a89bd65735191b1d3ee74a53726e7ca69fe"}
Oct 31 12:47:59.326215 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:47:59.326148    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bb3cf5-26qt6" podStartSLOduration=5.326128091 podStartE2EDuration="5.326128091s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:47:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:47:59.317656122 +0000 UTC m=+1242.420723836" watchObservedRunningTime="2026-10-31 12:47:59.326128091 +0000 UTC m=+1242.429195804"
Oct 31 12:48:00.231961 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:48:00.231923    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7phb" containerName="extractor"
Oct 31 12:48:00.352519 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:00.350105    5060 generic.go:356] "Generic (PLEG): container finished" podID="de748339-be67-487e-ba7b-eeaa3ccbc356" containerID="a5457bbc2fa14cea13e8f2a52bd54a89bd65735191b1d3ee74a53726e7ca69fe" exitCode=0
Oct 31 12:48:00.352519 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:00.350265    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bb3cf5-26qt6" event={"ID":"de748339-be67-487e-ba7b-eeaa3ccbc356","Type":"ContainerDied","Data":"a5457bbc2fa14cea13e8f2a52bd54a89bd65735191b1d3ee74a53726e7ca69fe"}
Oct 31 12:48:00.531670 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:00.529868    5060 generic.go:356] "Generic (PLEG): container finished" podID="1af894f4-5dbb-495a-b896-670ce365742e" containerID="baf84e1b4a76f84270c7514aa2ef620831892521cd9d798f67f36272ba414bf7" exitCode=0
Oct 31 12:48:00.531670 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:00.530083    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f22e7f-openshift-default-84f9758845-24j6m" event={"ID":"1af894f4-5dbb-495a-b896-670ce365742e","Type":"ContainerDied","Data":"baf84e1b4a76f84270c7514aa2ef620831892521cd9d798f67f36272ba414bf7"}
Oct 31 12:48:00.531670 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:00.530140    5060 scope.go:122] "RemoveContainer" containerID="2f2f93ba9ba80135ad1f844874a72f1bfa14b5aeb5af28cf429848555803afdb"
Oct 31 12:48:01.133394 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:01.133355    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c302cfbf-gt8wk"]
Oct 31 12:48:01.144465 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:01.144433    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-223f1a1d-l6hj5"]
Oct 31 12:48:01.153605 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:01.153572    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c302cfbf-gt8wk"]
Oct 31 12:48:01.194234 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:01.194183    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-223f1a1d-l6hj5"]
Oct 31 12:48:01.291548 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:01.291511    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f22e7f-openshift-default-84f9758845-24j6m"
Oct 31 12:48:01.315273 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:01.315202    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1af894f4-5dbb-495a-b896-670ce365742e-kube-api-access-pzxtw\" (UniqueName: \"kubernetes.io/projected/1af894f4-5dbb-495a-b896-670ce365742e-kube-api-access-pzxtw\") pod \"1af894f4-5dbb-495a-b896-670ce365742e\" (UID: \"1af894f4-5dbb-495a-b896-670ce365742e\") "
Oct 31 12:48:01.315436 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:01.315272    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1af894f4-5dbb-495a-b896-670ce365742e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1af894f4-5dbb-495a-b896-670ce365742e-istio-podinfo\") pod \"1af894f4-5dbb-495a-b896-670ce365742e\" (UID: \"1af894f4-5dbb-495a-b896-670ce365742e\") "
Oct 31 12:48:01.315436 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:01.315316    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1af894f4-5dbb-495a-b896-670ce365742e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1af894f4-5dbb-495a-b896-670ce365742e-workload-certs\") pod \"1af894f4-5dbb-495a-b896-670ce365742e\" (UID: \"1af894f4-5dbb-495a-b896-670ce365742e\") "
Oct 31 12:48:01.315436 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:01.315350    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1af894f4-5dbb-495a-b896-670ce365742e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1af894f4-5dbb-495a-b896-670ce365742e-istio-data\") pod \"1af894f4-5dbb-495a-b896-670ce365742e\" (UID: \"1af894f4-5dbb-495a-b896-670ce365742e\") "
Oct 31 12:48:01.315436 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:01.315396    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1af894f4-5dbb-495a-b896-670ce365742e-istio-token\" (UniqueName: \"kubernetes.io/projected/1af894f4-5dbb-495a-b896-670ce365742e-istio-token\") pod \"1af894f4-5dbb-495a-b896-670ce365742e\" (UID: \"1af894f4-5dbb-495a-b896-670ce365742e\") "
Oct 31 12:48:01.315436 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:01.315417    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1af894f4-5dbb-495a-b896-670ce365742e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1af894f4-5dbb-495a-b896-670ce365742e-credential-socket\") pod \"1af894f4-5dbb-495a-b896-670ce365742e\" (UID: \"1af894f4-5dbb-495a-b896-670ce365742e\") "
Oct 31 12:48:01.315764 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:01.315459    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1af894f4-5dbb-495a-b896-670ce365742e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1af894f4-5dbb-495a-b896-670ce365742e-istiod-ca-cert\") pod \"1af894f4-5dbb-495a-b896-670ce365742e\" (UID: \"1af894f4-5dbb-495a-b896-670ce365742e\") "
Oct 31 12:48:01.315764 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:01.315605    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1af894f4-5dbb-495a-b896-670ce365742e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1af894f4-5dbb-495a-b896-670ce365742e-istio-envoy\") pod \"1af894f4-5dbb-495a-b896-670ce365742e\" (UID: \"1af894f4-5dbb-495a-b896-670ce365742e\") "
Oct 31 12:48:01.315880 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:01.315771    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1af894f4-5dbb-495a-b896-670ce365742e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1af894f4-5dbb-495a-b896-670ce365742e-workload-socket\") pod \"1af894f4-5dbb-495a-b896-670ce365742e\" (UID: \"1af894f4-5dbb-495a-b896-670ce365742e\") "
Oct 31 12:48:01.316387 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:01.315776    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1af894f4-5dbb-495a-b896-670ce365742e-istio-data" pod "1af894f4-5dbb-495a-b896-670ce365742e" (UID: "1af894f4-5dbb-495a-b896-670ce365742e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:48:01.316387 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:01.316009    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1af894f4-5dbb-495a-b896-670ce365742e-istiod-ca-cert" pod "1af894f4-5dbb-495a-b896-670ce365742e" (UID: "1af894f4-5dbb-495a-b896-670ce365742e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:48:01.316387 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:01.316275    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1af894f4-5dbb-495a-b896-670ce365742e-workload-certs" pod "1af894f4-5dbb-495a-b896-670ce365742e" (UID: "1af894f4-5dbb-495a-b896-670ce365742e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:48:01.316387 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:01.316341    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1af894f4-5dbb-495a-b896-670ce365742e-credential-socket" pod "1af894f4-5dbb-495a-b896-670ce365742e" (UID: "1af894f4-5dbb-495a-b896-670ce365742e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:48:01.317058 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:01.316624    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1af894f4-5dbb-495a-b896-670ce365742e-workload-socket" pod "1af894f4-5dbb-495a-b896-670ce365742e" (UID: "1af894f4-5dbb-495a-b896-670ce365742e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:48:01.317058 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:01.316878    5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1af894f4-5dbb-495a-b896-670ce365742e-credential-socket\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:48:01.317058 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:01.316898    5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1af894f4-5dbb-495a-b896-670ce365742e-istiod-ca-cert\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:48:01.317058 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:01.316911    5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1af894f4-5dbb-495a-b896-670ce365742e-workload-socket\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:48:01.317058 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:01.316925    5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1af894f4-5dbb-495a-b896-670ce365742e-workload-certs\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:48:01.317058 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:01.316939    5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1af894f4-5dbb-495a-b896-670ce365742e-istio-data\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:48:01.318332 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:01.318243    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1af894f4-5dbb-495a-b896-670ce365742e-kube-api-access-pzxtw" pod "1af894f4-5dbb-495a-b896-670ce365742e" (UID: "1af894f4-5dbb-495a-b896-670ce365742e"). InnerVolumeSpecName "kube-api-access-pzxtw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:48:01.319801 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:01.319766    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1af894f4-5dbb-495a-b896-670ce365742e-istio-envoy" pod "1af894f4-5dbb-495a-b896-670ce365742e" (UID: "1af894f4-5dbb-495a-b896-670ce365742e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:48:01.332006 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:01.320161    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1af894f4-5dbb-495a-b896-670ce365742e-istio-podinfo" pod "1af894f4-5dbb-495a-b896-670ce365742e" (UID: "1af894f4-5dbb-495a-b896-670ce365742e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 12:48:01.332006 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:01.320761    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1af894f4-5dbb-495a-b896-670ce365742e-istio-token" pod "1af894f4-5dbb-495a-b896-670ce365742e" (UID: "1af894f4-5dbb-495a-b896-670ce365742e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:48:01.418469 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:01.418430    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pzxtw\" (UniqueName: \"kubernetes.io/projected/1af894f4-5dbb-495a-b896-670ce365742e-kube-api-access-pzxtw\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:48:01.418469 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:01.418468    5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1af894f4-5dbb-495a-b896-670ce365742e-istio-podinfo\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:48:01.418469 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:01.418478    5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1af894f4-5dbb-495a-b896-670ce365742e-istio-token\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:48:01.418469 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:01.418498    5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1af894f4-5dbb-495a-b896-670ce365742e-istio-envoy\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:48:01.545825 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:01.545783    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f22e7f-openshift-default-84f9758845-24j6m" event={"ID":"1af894f4-5dbb-495a-b896-670ce365742e","Type":"ContainerDied","Data":"d3c8dc1ad394e45c85cc49c0d32de83a2fcccd406b5f9d415126504ca9ecd1f8"}
Oct 31 12:48:01.546021 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:01.545829    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f22e7f-openshift-default-84f9758845-24j6m"
Oct 31 12:48:01.546021 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:01.545887    5060 scope.go:122] "RemoveContainer" containerID="baf84e1b4a76f84270c7514aa2ef620831892521cd9d798f67f36272ba414bf7"
Oct 31 12:48:01.616106 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:01.616062    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-f22e7f-openshift-default-84f9758845-24j6m"]
Oct 31 12:48:01.630224 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:01.630185    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-f22e7f-openshift-default-84f9758845-24j6m"]
Oct 31 12:48:01.898759 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:01.898669    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bb3cf5-26qt6"
Oct 31 12:48:01.930219 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:01.928384    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de748339-be67-487e-ba7b-eeaa3ccbc356-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de748339-be67-487e-ba7b-eeaa3ccbc356-maas-api-openshift-ca\") pod \"de748339-be67-487e-ba7b-eeaa3ccbc356\" (UID: \"de748339-be67-487e-ba7b-eeaa3ccbc356\") "
Oct 31 12:48:01.930219 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:01.928821    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de748339-be67-487e-ba7b-eeaa3ccbc356-kube-api-access-z95gn\" (UniqueName: \"kubernetes.io/projected/de748339-be67-487e-ba7b-eeaa3ccbc356-kube-api-access-z95gn\") pod \"de748339-be67-487e-ba7b-eeaa3ccbc356\" (UID: \"de748339-be67-487e-ba7b-eeaa3ccbc356\") "
Oct 31 12:48:01.930219 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:01.928936    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/de748339-be67-487e-ba7b-eeaa3ccbc356-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de748339-be67-487e-ba7b-eeaa3ccbc356-maas-api-xks-ca\") pod \"de748339-be67-487e-ba7b-eeaa3ccbc356\" (UID: \"de748339-be67-487e-ba7b-eeaa3ccbc356\") "
Oct 31 12:48:01.930726 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:01.930599    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de748339-be67-487e-ba7b-eeaa3ccbc356-maas-api-openshift-ca" pod "de748339-be67-487e-ba7b-eeaa3ccbc356" (UID: "de748339-be67-487e-ba7b-eeaa3ccbc356"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:48:01.971411 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:01.971340    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de748339-be67-487e-ba7b-eeaa3ccbc356-kube-api-access-z95gn" pod "de748339-be67-487e-ba7b-eeaa3ccbc356" (UID: "de748339-be67-487e-ba7b-eeaa3ccbc356"). InnerVolumeSpecName "kube-api-access-z95gn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:48:02.043250 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:02.033890    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de748339-be67-487e-ba7b-eeaa3ccbc356-maas-api-openshift-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:48:02.043250 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:02.033931    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z95gn\" (UniqueName: \"kubernetes.io/projected/de748339-be67-487e-ba7b-eeaa3ccbc356-kube-api-access-z95gn\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:48:02.060828 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:02.060781    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de748339-be67-487e-ba7b-eeaa3ccbc356-maas-api-xks-ca" pod "de748339-be67-487e-ba7b-eeaa3ccbc356" (UID: "de748339-be67-487e-ba7b-eeaa3ccbc356"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:48:02.135658 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:02.135612    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de748339-be67-487e-ba7b-eeaa3ccbc356-maas-api-xks-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:48:02.590918 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:02.562526    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bb3cf5-26qt6"
Oct 31 12:48:02.590918 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:02.562541    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bb3cf5-26qt6" event={"ID":"de748339-be67-487e-ba7b-eeaa3ccbc356","Type":"ContainerDied","Data":"781727245fd3327448603cbf3136bae57a39d7dccca07c67c2598641f5b55cff"}
Oct 31 12:48:02.590918 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:02.562600    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="781727245fd3327448603cbf3136bae57a39d7dccca07c67c2598641f5b55cff"
Oct 31 12:48:03.101233 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:03.101199    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1af894f4-5dbb-495a-b896-670ce365742e" path="/var/lib/kubelet/pods/1af894f4-5dbb-495a-b896-670ce365742e/volumes"
Oct 31 12:48:03.102253 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:03.102230    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e5cad97-aa60-45de-a22a-49ea851c9a38" path="/var/lib/kubelet/pods/5e5cad97-aa60-45de-a22a-49ea851c9a38/volumes"
Oct 31 12:48:03.103026 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:03.103006    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7aadf9e8-6412-439d-9b07-9f468cd6d91a" path="/var/lib/kubelet/pods/7aadf9e8-6412-439d-9b07-9f468cd6d91a/volumes"
Oct 31 12:48:05.995000 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:05.994614    5060 patch_prober.go:29] interesting pod/e2e-api-b-bb3cf5-openshift-default-5fc978478f-mw7qm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 12:48:05.995000 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:05.994680    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-bb3cf5-openshift-default-5fc978478f-mw7qm" podUID="d9cb64fe-d6b4-439d-bc3a-8598b7cd512e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:48:07.432679 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:07.432634    5060 patch_prober.go:29] interesting pod/e2e-api-b-711af2-openshift-default-6549b5d6f9-zdqxc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 12:48:07.433199 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:07.432718    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-711af2-openshift-default-6549b5d6f9-zdqxc" podUID="440e8c3e-006e-4a6b-82d1-5fb80f5e26d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:48:08.313957 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:08.313908    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-72mg2"]
Oct 31 12:48:08.315323 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:08.315279    5060 cpu_manager.go:414] "containerMap: removing container" podUID="1af894f4-5dbb-495a-b896-670ce365742e" containerName="istio-proxy"
Oct 31 12:48:08.315323 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:08.315310    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1af894f4-5dbb-495a-b896-670ce365742e" containerName="istio-proxy"
Oct 31 12:48:08.317348 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:08.315336    5060 cpu_manager.go:414] "containerMap: removing container" podUID="1af894f4-5dbb-495a-b896-670ce365742e" containerName="istio-proxy"
Oct 31 12:48:08.317348 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:08.315344    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1af894f4-5dbb-495a-b896-670ce365742e" containerName="istio-proxy"
Oct 31 12:48:08.317348 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:08.315390    5060 cpu_manager.go:414] "containerMap: removing container" podUID="de748339-be67-487e-ba7b-eeaa3ccbc356" containerName="revoke-keys"
Oct 31 12:48:08.317348 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:08.315397    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de748339-be67-487e-ba7b-eeaa3ccbc356" containerName="revoke-keys"
Oct 31 12:48:08.317348 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:08.315693    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="1af894f4-5dbb-495a-b896-670ce365742e" containerName="istio-proxy"
Oct 31 12:48:08.317348 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:08.315715    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="1af894f4-5dbb-495a-b896-670ce365742e" containerName="istio-proxy"
Oct 31 12:48:08.317348 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:08.315728    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="de748339-be67-487e-ba7b-eeaa3ccbc356" containerName="revoke-keys"
Oct 31 12:48:08.415443 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:08.415410    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-72mg2"]
Oct 31 12:48:08.415612 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:08.415567    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-72mg2"
Oct 31 12:48:08.583742 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:08.583653    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/346f4c45-c06a-4ae5-9fa0-c426467700f9-istio-envoy\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-72mg2\" (UID: \"346f4c45-c06a-4ae5-9fa0-c426467700f9\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-72mg2"
Oct 31 12:48:08.584262 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:08.583801    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/346f4c45-c06a-4ae5-9fa0-c426467700f9-workload-certs\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-72mg2\" (UID: \"346f4c45-c06a-4ae5-9fa0-c426467700f9\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-72mg2"
Oct 31 12:48:08.584262 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:08.583905    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/346f4c45-c06a-4ae5-9fa0-c426467700f9-istio-data\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-72mg2\" (UID: \"346f4c45-c06a-4ae5-9fa0-c426467700f9\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-72mg2"
Oct 31 12:48:08.584262 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:08.584012    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/346f4c45-c06a-4ae5-9fa0-c426467700f9-istio-podinfo\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-72mg2\" (UID: \"346f4c45-c06a-4ae5-9fa0-c426467700f9\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-72mg2"
Oct 31 12:48:08.584262 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:08.584115    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/346f4c45-c06a-4ae5-9fa0-c426467700f9-workload-socket\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-72mg2\" (UID: \"346f4c45-c06a-4ae5-9fa0-c426467700f9\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-72mg2"
Oct 31 12:48:08.584262 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:08.584156    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/346f4c45-c06a-4ae5-9fa0-c426467700f9-istiod-ca-cert\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-72mg2\" (UID: \"346f4c45-c06a-4ae5-9fa0-c426467700f9\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-72mg2"
Oct 31 12:48:08.584262 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:08.584189    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/346f4c45-c06a-4ae5-9fa0-c426467700f9-istio-token\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-72mg2\" (UID: \"346f4c45-c06a-4ae5-9fa0-c426467700f9\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-72mg2"
Oct 31 12:48:08.584581 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:08.584328    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqkjq\" (UniqueName: \"kubernetes.io/projected/346f4c45-c06a-4ae5-9fa0-c426467700f9-kube-api-access-wqkjq\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-72mg2\" (UID: \"346f4c45-c06a-4ae5-9fa0-c426467700f9\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-72mg2"
Oct 31 12:48:08.584581 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:08.584375    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/346f4c45-c06a-4ae5-9fa0-c426467700f9-credential-socket\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-72mg2\" (UID: \"346f4c45-c06a-4ae5-9fa0-c426467700f9\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-72mg2"
Oct 31 12:48:08.962507 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:08.685832    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/346f4c45-c06a-4ae5-9fa0-c426467700f9-istio-data\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-72mg2\" (UID: \"346f4c45-c06a-4ae5-9fa0-c426467700f9\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-72mg2"
Oct 31 12:48:08.962507 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:08.685923    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/346f4c45-c06a-4ae5-9fa0-c426467700f9-istio-podinfo\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-72mg2\" (UID: \"346f4c45-c06a-4ae5-9fa0-c426467700f9\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-72mg2"
Oct 31 12:48:08.962507 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:08.685952    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/346f4c45-c06a-4ae5-9fa0-c426467700f9-workload-socket\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-72mg2\" (UID: \"346f4c45-c06a-4ae5-9fa0-c426467700f9\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-72mg2"
Oct 31 12:48:08.962507 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:08.685983    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/346f4c45-c06a-4ae5-9fa0-c426467700f9-istiod-ca-cert\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-72mg2\" (UID: \"346f4c45-c06a-4ae5-9fa0-c426467700f9\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-72mg2"
Oct 31 12:48:08.962507 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:08.686011    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/346f4c45-c06a-4ae5-9fa0-c426467700f9-istio-token\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-72mg2\" (UID: \"346f4c45-c06a-4ae5-9fa0-c426467700f9\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-72mg2"
Oct 31 12:48:08.962507 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:08.686069    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqkjq\" (UniqueName: \"kubernetes.io/projected/346f4c45-c06a-4ae5-9fa0-c426467700f9-kube-api-access-wqkjq\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-72mg2\" (UID: \"346f4c45-c06a-4ae5-9fa0-c426467700f9\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-72mg2"
Oct 31 12:48:08.962507 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:08.686218    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/346f4c45-c06a-4ae5-9fa0-c426467700f9-credential-socket\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-72mg2\" (UID: \"346f4c45-c06a-4ae5-9fa0-c426467700f9\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-72mg2"
Oct 31 12:48:08.962507 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:08.686314    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/346f4c45-c06a-4ae5-9fa0-c426467700f9-istio-data\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-72mg2\" (UID: \"346f4c45-c06a-4ae5-9fa0-c426467700f9\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-72mg2"
Oct 31 12:48:08.962507 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:08.686369    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/346f4c45-c06a-4ae5-9fa0-c426467700f9-workload-socket\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-72mg2\" (UID: \"346f4c45-c06a-4ae5-9fa0-c426467700f9\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-72mg2"
Oct 31 12:48:08.962507 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:08.686462    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/346f4c45-c06a-4ae5-9fa0-c426467700f9-istio-envoy\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-72mg2\" (UID: \"346f4c45-c06a-4ae5-9fa0-c426467700f9\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-72mg2"
Oct 31 12:48:08.962507 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:08.686568    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/346f4c45-c06a-4ae5-9fa0-c426467700f9-workload-certs\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-72mg2\" (UID: \"346f4c45-c06a-4ae5-9fa0-c426467700f9\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-72mg2"
Oct 31 12:48:08.962507 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:08.686591    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/346f4c45-c06a-4ae5-9fa0-c426467700f9-credential-socket\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-72mg2\" (UID: \"346f4c45-c06a-4ae5-9fa0-c426467700f9\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-72mg2"
Oct 31 12:48:08.962507 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:08.686781    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/346f4c45-c06a-4ae5-9fa0-c426467700f9-istiod-ca-cert\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-72mg2\" (UID: \"346f4c45-c06a-4ae5-9fa0-c426467700f9\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-72mg2"
Oct 31 12:48:08.962507 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:08.686881    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/346f4c45-c06a-4ae5-9fa0-c426467700f9-workload-certs\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-72mg2\" (UID: \"346f4c45-c06a-4ae5-9fa0-c426467700f9\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-72mg2"
Oct 31 12:48:08.962507 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:08.688584    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/346f4c45-c06a-4ae5-9fa0-c426467700f9-istio-envoy\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-72mg2\" (UID: \"346f4c45-c06a-4ae5-9fa0-c426467700f9\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-72mg2"
Oct 31 12:48:08.962507 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:08.688725    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/346f4c45-c06a-4ae5-9fa0-c426467700f9-istio-podinfo\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-72mg2\" (UID: \"346f4c45-c06a-4ae5-9fa0-c426467700f9\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-72mg2"
Oct 31 12:48:08.962507 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:08.699091    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/346f4c45-c06a-4ae5-9fa0-c426467700f9-istio-token\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-72mg2\" (UID: \"346f4c45-c06a-4ae5-9fa0-c426467700f9\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-72mg2"
Oct 31 12:48:08.962507 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:08.699551    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqkjq\" (UniqueName: \"kubernetes.io/projected/346f4c45-c06a-4ae5-9fa0-c426467700f9-kube-api-access-wqkjq\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-72mg2\" (UID: \"346f4c45-c06a-4ae5-9fa0-c426467700f9\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-72mg2"
Oct 31 12:48:08.962507 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:08.749449    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-72mg2"
Oct 31 12:48:09.314186 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:09.313828    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-72mg2"]
Oct 31 12:48:09.314374 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:48:09.314207    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod346f4c45_c06a_4ae5_9fa0_c426467700f9.slice/crio-73f213b09a1900b7cd2db0acfaaf2b0e581dbcbc881d532dcdd7d3ad880d610a WatchSource:0}: Error finding container 73f213b09a1900b7cd2db0acfaaf2b0e581dbcbc881d532dcdd7d3ad880d610a: Status 404 returned error can't find the container with id 73f213b09a1900b7cd2db0acfaaf2b0e581dbcbc881d532dcdd7d3ad880d610a
Oct 31 12:48:09.320047 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:09.319462    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:48:09.320047 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:09.319643    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:48:09.320047 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:09.319685    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:48:09.860988 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:09.860955    5060 generic.go:356] "Generic (PLEG): container finished" podID="d9cb64fe-d6b4-439d-bc3a-8598b7cd512e" containerID="15e608e4b881d6dc2804ab6cfc24e27e38e7e9798db2a22b07b5bdd67fffaf78" exitCode=0
Oct 31 12:48:09.861549 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:09.861025    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-bb3cf5-openshift-default-5fc978478f-mw7qm" event={"ID":"d9cb64fe-d6b4-439d-bc3a-8598b7cd512e","Type":"ContainerDied","Data":"15e608e4b881d6dc2804ab6cfc24e27e38e7e9798db2a22b07b5bdd67fffaf78"}
Oct 31 12:48:09.861639 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:09.861614    5060 scope.go:122] "RemoveContainer" containerID="15e608e4b881d6dc2804ab6cfc24e27e38e7e9798db2a22b07b5bdd67fffaf78"
Oct 31 12:48:09.863447 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:09.863422    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-72mg2" event={"ID":"346f4c45-c06a-4ae5-9fa0-c426467700f9","Type":"ContainerStarted","Data":"ea8748ac2ab76bad1ceca1fd5047a546529d7eb073f2611aa57bbbb73e6ea836"}
Oct 31 12:48:09.863646 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:09.863623    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-72mg2" event={"ID":"346f4c45-c06a-4ae5-9fa0-c426467700f9","Type":"ContainerStarted","Data":"73f213b09a1900b7cd2db0acfaaf2b0e581dbcbc881d532dcdd7d3ad880d610a"}
Oct 31 12:48:09.863795 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:09.863773    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:48:09.863858 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:09.863833    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:48:09.863912 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:09.863868    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:48:09.866150 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:09.866129    5060 generic.go:356] "Generic (PLEG): container finished" podID="4e300ab1-6fe3-4548-90b3-0cbb8099e114" containerID="1c42065af8b553bdc7cf3f32752a5147a10231f9339cf04598ee8ae8db27912e" exitCode=0
Oct 31 12:48:09.866262 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:09.866203    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-hkvl5" event={"ID":"4e300ab1-6fe3-4548-90b3-0cbb8099e114","Type":"ContainerDied","Data":"1c42065af8b553bdc7cf3f32752a5147a10231f9339cf04598ee8ae8db27912e"}
Oct 31 12:48:09.866692 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:09.866671    5060 scope.go:122] "RemoveContainer" containerID="1c42065af8b553bdc7cf3f32752a5147a10231f9339cf04598ee8ae8db27912e"
Oct 31 12:48:09.868348 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:09.868315    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:48:09.868445 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:09.868395    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:48:09.868535 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:09.868448    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:48:10.137258 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:10.137183    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-72mg2" podStartSLOduration=2.137162446 podStartE2EDuration="2.137162446s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:48:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:48:09.975900382 +0000 UTC m=+1253.078968120" watchObservedRunningTime="2026-10-31 12:48:10.137162446 +0000 UTC m=+1253.240230159"
Oct 31 12:48:10.138686 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:10.138655    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f6c73a95-bcsgz"]
Oct 31 12:48:10.211239 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:10.211207    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f6c73a95-bcsgz"]
Oct 31 12:48:10.759515 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:10.754763    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-72mg2"
Oct 31 12:48:10.775709 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:10.764194    5060 patch_prober.go:29] interesting pod/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-72mg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" start-of-body=
Oct 31 12:48:10.775709 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:10.764257    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-72mg2" podUID="346f4c45-c06a-4ae5-9fa0-c426467700f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused"
Oct 31 12:48:11.166157 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:11.039479    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-hkvl5"
Oct 31 12:48:11.495570 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:11.478483    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e86e368c-5055-4433-abb6-f1171b74c21d" path="/var/lib/kubelet/pods/e86e368c-5055-4433-abb6-f1171b74c21d/volumes"
Oct 31 12:48:11.591573 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:11.584440    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-hkvl5" event={"ID":"4e300ab1-6fe3-4548-90b3-0cbb8099e114","Type":"ContainerStarted","Data":"ff3576ccdf821939a0cd5ae44ea67bdc37f3e3711c25c2282a554d05810389d6"}
Oct 31 12:48:11.634525 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:11.623354    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-bb3cf5-openshift-default-5fc978478f-mw7qm" event={"ID":"d9cb64fe-d6b4-439d-bc3a-8598b7cd512e","Type":"ContainerStarted","Data":"d4a8f0047375c3091a7328bd485e99dc745049f6dbe9a643546d3273f279c468"}
Oct 31 12:48:11.666989 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:11.665425    5060 patch_prober.go:29] interesting pod/e2e-api-b-bb3cf5-openshift-default-5fc978478f-mw7qm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body=
Oct 31 12:48:11.666989 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:11.665484    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-bb3cf5-openshift-default-5fc978478f-mw7qm" podUID="d9cb64fe-d6b4-439d-bc3a-8598b7cd512e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused"
Oct 31 12:48:11.666989 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:11.665815    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-bb3cf5-openshift-default-5fc978478f-mw7qm"
Oct 31 12:48:11.754416 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:11.754322    5060 patch_prober.go:29] interesting pod/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-72mg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" start-of-body=
Oct 31 12:48:11.754721 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:11.754691    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-72mg2" podUID="346f4c45-c06a-4ae5-9fa0-c426467700f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused"
Oct 31 12:48:11.775238 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:11.775194    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-hkvl5"
Oct 31 12:48:11.778258 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:11.775582    5060 patch_prober.go:29] interesting pod/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-hkvl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body=
Oct 31 12:48:11.778549 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:11.778519    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-hkvl5" podUID="4e300ab1-6fe3-4548-90b3-0cbb8099e114" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused"
Oct 31 12:48:12.665441 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:12.665390    5060 patch_prober.go:29] interesting pod/e2e-api-b-bb3cf5-openshift-default-5fc978478f-mw7qm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body=
Oct 31 12:48:12.666039 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:12.666004    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-bb3cf5-openshift-default-5fc978478f-mw7qm" podUID="d9cb64fe-d6b4-439d-bc3a-8598b7cd512e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused"
Oct 31 12:48:12.770191 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:12.769741    5060 patch_prober.go:29] interesting pod/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-72mg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" start-of-body=
Oct 31 12:48:12.770191 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:12.769801    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-72mg2" podUID="346f4c45-c06a-4ae5-9fa0-c426467700f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused"
Oct 31 12:48:12.780961 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:12.780837    5060 patch_prober.go:29] interesting pod/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-hkvl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body=
Oct 31 12:48:12.780961 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:12.780898    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-hkvl5" podUID="4e300ab1-6fe3-4548-90b3-0cbb8099e114" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused"
Oct 31 12:48:13.166227 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:13.165582    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/simulator-w4-c67de2-kserve-868679cf9b-gqjbx" podUID="e104d7da-1fb2-4c99-a480-b4ae84c4f273" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:48:13.266992 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:13.226706    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x2qzv" podUID="c247ed2a-aa90-424b-9b7e-4f2936d67e3f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded"
Oct 31 12:48:13.384220 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:13.384167    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5c5815-cbc4t"]
Oct 31 12:48:13.485610 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:13.485225    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5c5815-cbc4t"]
Oct 31 12:48:13.485610 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:13.485383    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5c5815-cbc4t"
Oct 31 12:48:13.503300 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:13.486698    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-2-w5-b25d3b-kserve-76f798d59f-vvzhg" podUID="394ed76c-84ac-4612-9acb-a7e36cb68d89" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": context deadline exceeded"
Oct 31 12:48:13.503300 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:13.491958    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 12:48:13.503300 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:13.492098    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7wsmb\""
Oct 31 12:48:13.503300 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:13.498948    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 12:48:13.503300 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:13.499067    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 12:48:13.659512 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:13.647318    5060 patch_prober.go:29] interesting pod/e2e-api-b-bb3cf5-openshift-default-5fc978478f-mw7qm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body=
Oct 31 12:48:13.659512 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:13.647379    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-bb3cf5-openshift-default-5fc978478f-mw7qm" podUID="d9cb64fe-d6b4-439d-bc3a-8598b7cd512e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused"
Oct 31 12:48:13.659512 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:13.654998    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5psx\" (UniqueName: \"kubernetes.io/projected/e39c020a-e0e5-4a0c-9c52-3d63241cc13c-kube-api-access-h5psx\") pod \"maas-api-revoke-keys-e2e-api-a-5c5815-cbc4t\" (UID: \"e39c020a-e0e5-4a0c-9c52-3d63241cc13c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5c5815-cbc4t"
Oct 31 12:48:13.659512 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:13.655102    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e39c020a-e0e5-4a0c-9c52-3d63241cc13c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-5c5815-cbc4t\" (UID: \"e39c020a-e0e5-4a0c-9c52-3d63241cc13c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5c5815-cbc4t"
Oct 31 12:48:13.659512 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:13.655225    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e39c020a-e0e5-4a0c-9c52-3d63241cc13c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-5c5815-cbc4t\" (UID: \"e39c020a-e0e5-4a0c-9c52-3d63241cc13c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5c5815-cbc4t"
Oct 31 12:48:13.754613 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:13.754479    5060 patch_prober.go:29] interesting pod/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-72mg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" start-of-body=
Oct 31 12:48:13.755237 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:13.755186    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-72mg2" podUID="346f4c45-c06a-4ae5-9fa0-c426467700f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused"
Oct 31 12:48:13.760048 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:13.760006    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5psx\" (UniqueName: \"kubernetes.io/projected/e39c020a-e0e5-4a0c-9c52-3d63241cc13c-kube-api-access-h5psx\") pod \"maas-api-revoke-keys-e2e-api-a-5c5815-cbc4t\" (UID: \"e39c020a-e0e5-4a0c-9c52-3d63241cc13c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5c5815-cbc4t"
Oct 31 12:48:13.760356 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:13.760325    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e39c020a-e0e5-4a0c-9c52-3d63241cc13c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-5c5815-cbc4t\" (UID: \"e39c020a-e0e5-4a0c-9c52-3d63241cc13c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5c5815-cbc4t"
Oct 31 12:48:13.763237 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:13.763199    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e39c020a-e0e5-4a0c-9c52-3d63241cc13c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-5c5815-cbc4t\" (UID: \"e39c020a-e0e5-4a0c-9c52-3d63241cc13c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5c5815-cbc4t"
Oct 31 12:48:13.768993 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:13.768956    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e39c020a-e0e5-4a0c-9c52-3d63241cc13c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-5c5815-cbc4t\" (UID: \"e39c020a-e0e5-4a0c-9c52-3d63241cc13c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5c5815-cbc4t"
Oct 31 12:48:13.788521 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:13.786118    5060 patch_prober.go:29] interesting pod/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-hkvl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body=
Oct 31 12:48:13.788521 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:13.786176    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-hkvl5" podUID="4e300ab1-6fe3-4548-90b3-0cbb8099e114" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused"
Oct 31 12:48:13.937982 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:13.937938    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5psx\" (UniqueName: \"kubernetes.io/projected/e39c020a-e0e5-4a0c-9c52-3d63241cc13c-kube-api-access-h5psx\") pod \"maas-api-revoke-keys-e2e-api-a-5c5815-cbc4t\" (UID: \"e39c020a-e0e5-4a0c-9c52-3d63241cc13c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5c5815-cbc4t"
Oct 31 12:48:13.946908 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:13.946861    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e39c020a-e0e5-4a0c-9c52-3d63241cc13c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-5c5815-cbc4t\" (UID: \"e39c020a-e0e5-4a0c-9c52-3d63241cc13c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5c5815-cbc4t"
Oct 31 12:48:14.174514 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:14.173756    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5c5815-cbc4t"
Oct 31 12:48:14.659511 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:14.648878    5060 patch_prober.go:29] interesting pod/e2e-api-b-bb3cf5-openshift-default-5fc978478f-mw7qm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body=
Oct 31 12:48:14.659511 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:14.648932    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-bb3cf5-openshift-default-5fc978478f-mw7qm" podUID="d9cb64fe-d6b4-439d-bc3a-8598b7cd512e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused"
Oct 31 12:48:14.777517 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:14.776829    5060 patch_prober.go:29] interesting pod/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-72mg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" start-of-body=
Oct 31 12:48:14.777517 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:14.776898    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-72mg2" podUID="346f4c45-c06a-4ae5-9fa0-c426467700f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused"
Oct 31 12:48:14.790515 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:14.785154    5060 patch_prober.go:29] interesting pod/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-hkvl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body=
Oct 31 12:48:14.790515 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:14.785207    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-hkvl5" podUID="4e300ab1-6fe3-4548-90b3-0cbb8099e114" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused"
Oct 31 12:48:15.617667 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:15.615079    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-e43d8b-openshift-default-5578dcfc7f-5f9ss"]
Oct 31 12:48:15.654418 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:15.653965    5060 patch_prober.go:29] interesting pod/e2e-api-b-bb3cf5-openshift-default-5fc978478f-mw7qm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body=
Oct 31 12:48:15.654418 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:15.654048    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-bb3cf5-openshift-default-5fc978478f-mw7qm" podUID="d9cb64fe-d6b4-439d-bc3a-8598b7cd512e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused"
Oct 31 12:48:15.755213 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:15.755177    5060 patch_prober.go:29] interesting pod/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-72mg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" start-of-body=
Oct 31 12:48:15.755478 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:15.755447    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-72mg2" podUID="346f4c45-c06a-4ae5-9fa0-c426467700f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused"
Oct 31 12:48:15.822514 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:15.780647    5060 patch_prober.go:29] interesting pod/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-hkvl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body=
Oct 31 12:48:15.822514 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:15.780710    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-hkvl5" podUID="4e300ab1-6fe3-4548-90b3-0cbb8099e114" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused"
Oct 31 12:48:15.945592 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:15.945230    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e43d8b-openshift-default-5578dcfc7f-5f9ss"
Oct 31 12:48:15.970595 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:15.962542    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-e43d8b-openshift-default-dockercfg-s49sm\""
Oct 31 12:48:16.008833 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:15.999123    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-e43d8b-openshift-default-5578dcfc7f-5f9ss"]
Oct 31 12:48:16.008833 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:16.002860    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0ba8ab3-65d2-4132-9896-80f1fb46b5c5-credential-socket\") pod \"e2e-inf-a-e43d8b-openshift-default-5578dcfc7f-5f9ss\" (UID: \"a0ba8ab3-65d2-4132-9896-80f1fb46b5c5\") " pod="openshift-ingress/e2e-inf-a-e43d8b-openshift-default-5578dcfc7f-5f9ss"
Oct 31 12:48:16.008833 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:16.002933    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0ba8ab3-65d2-4132-9896-80f1fb46b5c5-workload-socket\") pod \"e2e-inf-a-e43d8b-openshift-default-5578dcfc7f-5f9ss\" (UID: \"a0ba8ab3-65d2-4132-9896-80f1fb46b5c5\") " pod="openshift-ingress/e2e-inf-a-e43d8b-openshift-default-5578dcfc7f-5f9ss"
Oct 31 12:48:16.008833 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:16.003000    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0ba8ab3-65d2-4132-9896-80f1fb46b5c5-istio-data\") pod \"e2e-inf-a-e43d8b-openshift-default-5578dcfc7f-5f9ss\" (UID: \"a0ba8ab3-65d2-4132-9896-80f1fb46b5c5\") " pod="openshift-ingress/e2e-inf-a-e43d8b-openshift-default-5578dcfc7f-5f9ss"
Oct 31 12:48:16.008833 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:16.003053    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0ba8ab3-65d2-4132-9896-80f1fb46b5c5-istio-podinfo\") pod \"e2e-inf-a-e43d8b-openshift-default-5578dcfc7f-5f9ss\" (UID: \"a0ba8ab3-65d2-4132-9896-80f1fb46b5c5\") " pod="openshift-ingress/e2e-inf-a-e43d8b-openshift-default-5578dcfc7f-5f9ss"
Oct 31 12:48:16.008833 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:16.003141    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0ba8ab3-65d2-4132-9896-80f1fb46b5c5-istio-envoy\") pod \"e2e-inf-a-e43d8b-openshift-default-5578dcfc7f-5f9ss\" (UID: \"a0ba8ab3-65d2-4132-9896-80f1fb46b5c5\") " pod="openshift-ingress/e2e-inf-a-e43d8b-openshift-default-5578dcfc7f-5f9ss"
Oct 31 12:48:16.008833 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:16.003192    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cg6vs\" (UniqueName: \"kubernetes.io/projected/a0ba8ab3-65d2-4132-9896-80f1fb46b5c5-kube-api-access-cg6vs\") pod \"e2e-inf-a-e43d8b-openshift-default-5578dcfc7f-5f9ss\" (UID: \"a0ba8ab3-65d2-4132-9896-80f1fb46b5c5\") " pod="openshift-ingress/e2e-inf-a-e43d8b-openshift-default-5578dcfc7f-5f9ss"
Oct 31 12:48:16.008833 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:16.003276    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0ba8ab3-65d2-4132-9896-80f1fb46b5c5-istio-token\") pod \"e2e-inf-a-e43d8b-openshift-default-5578dcfc7f-5f9ss\" (UID: \"a0ba8ab3-65d2-4132-9896-80f1fb46b5c5\") " pod="openshift-ingress/e2e-inf-a-e43d8b-openshift-default-5578dcfc7f-5f9ss"
Oct 31 12:48:16.008833 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:16.003430    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0ba8ab3-65d2-4132-9896-80f1fb46b5c5-istiod-ca-cert\") pod \"e2e-inf-a-e43d8b-openshift-default-5578dcfc7f-5f9ss\" (UID: \"a0ba8ab3-65d2-4132-9896-80f1fb46b5c5\") " pod="openshift-ingress/e2e-inf-a-e43d8b-openshift-default-5578dcfc7f-5f9ss"
Oct 31 12:48:16.008833 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:16.003536    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0ba8ab3-65d2-4132-9896-80f1fb46b5c5-workload-certs\") pod \"e2e-inf-a-e43d8b-openshift-default-5578dcfc7f-5f9ss\" (UID: \"a0ba8ab3-65d2-4132-9896-80f1fb46b5c5\") " pod="openshift-ingress/e2e-inf-a-e43d8b-openshift-default-5578dcfc7f-5f9ss"
Oct 31 12:48:16.099832 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:16.099573    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/premium-w2-e6c79d-kserve-6468bf46d4-h66f8" podUID="cd6a266f-8d12-4089-83ee-c75c716bf26c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:48:16.130737 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:16.119976    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0ba8ab3-65d2-4132-9896-80f1fb46b5c5-credential-socket\") pod \"e2e-inf-a-e43d8b-openshift-default-5578dcfc7f-5f9ss\" (UID: \"a0ba8ab3-65d2-4132-9896-80f1fb46b5c5\") " pod="openshift-ingress/e2e-inf-a-e43d8b-openshift-default-5578dcfc7f-5f9ss"
Oct 31 12:48:16.130737 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:16.120053    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0ba8ab3-65d2-4132-9896-80f1fb46b5c5-workload-socket\") pod \"e2e-inf-a-e43d8b-openshift-default-5578dcfc7f-5f9ss\" (UID: \"a0ba8ab3-65d2-4132-9896-80f1fb46b5c5\") " pod="openshift-ingress/e2e-inf-a-e43d8b-openshift-default-5578dcfc7f-5f9ss"
Oct 31 12:48:16.130737 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:16.120111    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0ba8ab3-65d2-4132-9896-80f1fb46b5c5-istio-data\") pod \"e2e-inf-a-e43d8b-openshift-default-5578dcfc7f-5f9ss\" (UID: \"a0ba8ab3-65d2-4132-9896-80f1fb46b5c5\") " pod="openshift-ingress/e2e-inf-a-e43d8b-openshift-default-5578dcfc7f-5f9ss"
Oct 31 12:48:16.130737 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:16.120142    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0ba8ab3-65d2-4132-9896-80f1fb46b5c5-istio-podinfo\") pod \"e2e-inf-a-e43d8b-openshift-default-5578dcfc7f-5f9ss\" (UID: \"a0ba8ab3-65d2-4132-9896-80f1fb46b5c5\") " pod="openshift-ingress/e2e-inf-a-e43d8b-openshift-default-5578dcfc7f-5f9ss"
Oct 31 12:48:16.130737 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:16.120219    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0ba8ab3-65d2-4132-9896-80f1fb46b5c5-istio-envoy\") pod \"e2e-inf-a-e43d8b-openshift-default-5578dcfc7f-5f9ss\" (UID: \"a0ba8ab3-65d2-4132-9896-80f1fb46b5c5\") " pod="openshift-ingress/e2e-inf-a-e43d8b-openshift-default-5578dcfc7f-5f9ss"
Oct 31 12:48:16.130737 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:16.120280    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cg6vs\" (UniqueName: \"kubernetes.io/projected/a0ba8ab3-65d2-4132-9896-80f1fb46b5c5-kube-api-access-cg6vs\") pod \"e2e-inf-a-e43d8b-openshift-default-5578dcfc7f-5f9ss\" (UID: \"a0ba8ab3-65d2-4132-9896-80f1fb46b5c5\") " pod="openshift-ingress/e2e-inf-a-e43d8b-openshift-default-5578dcfc7f-5f9ss"
Oct 31 12:48:16.130737 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:16.120349    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0ba8ab3-65d2-4132-9896-80f1fb46b5c5-istio-token\") pod \"e2e-inf-a-e43d8b-openshift-default-5578dcfc7f-5f9ss\" (UID: \"a0ba8ab3-65d2-4132-9896-80f1fb46b5c5\") " pod="openshift-ingress/e2e-inf-a-e43d8b-openshift-default-5578dcfc7f-5f9ss"
Oct 31 12:48:16.130737 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:16.120481    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0ba8ab3-65d2-4132-9896-80f1fb46b5c5-istiod-ca-cert\") pod \"e2e-inf-a-e43d8b-openshift-default-5578dcfc7f-5f9ss\" (UID: \"a0ba8ab3-65d2-4132-9896-80f1fb46b5c5\") " pod="openshift-ingress/e2e-inf-a-e43d8b-openshift-default-5578dcfc7f-5f9ss"
Oct 31 12:48:16.130737 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:16.120583    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0ba8ab3-65d2-4132-9896-80f1fb46b5c5-workload-certs\") pod \"e2e-inf-a-e43d8b-openshift-default-5578dcfc7f-5f9ss\" (UID: \"a0ba8ab3-65d2-4132-9896-80f1fb46b5c5\") " pod="openshift-ingress/e2e-inf-a-e43d8b-openshift-default-5578dcfc7f-5f9ss"
Oct 31 12:48:16.130737 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:16.121117    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0ba8ab3-65d2-4132-9896-80f1fb46b5c5-workload-certs\") pod \"e2e-inf-a-e43d8b-openshift-default-5578dcfc7f-5f9ss\" (UID: \"a0ba8ab3-65d2-4132-9896-80f1fb46b5c5\") " pod="openshift-ingress/e2e-inf-a-e43d8b-openshift-default-5578dcfc7f-5f9ss"
Oct 31 12:48:16.130737 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:16.121331    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0ba8ab3-65d2-4132-9896-80f1fb46b5c5-credential-socket\") pod \"e2e-inf-a-e43d8b-openshift-default-5578dcfc7f-5f9ss\" (UID: \"a0ba8ab3-65d2-4132-9896-80f1fb46b5c5\") " pod="openshift-ingress/e2e-inf-a-e43d8b-openshift-default-5578dcfc7f-5f9ss"
Oct 31 12:48:16.130737 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:16.121576    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0ba8ab3-65d2-4132-9896-80f1fb46b5c5-workload-socket\") pod \"e2e-inf-a-e43d8b-openshift-default-5578dcfc7f-5f9ss\" (UID: \"a0ba8ab3-65d2-4132-9896-80f1fb46b5c5\") " pod="openshift-ingress/e2e-inf-a-e43d8b-openshift-default-5578dcfc7f-5f9ss"
Oct 31 12:48:16.130737 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:16.121785    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0ba8ab3-65d2-4132-9896-80f1fb46b5c5-istio-data\") pod \"e2e-inf-a-e43d8b-openshift-default-5578dcfc7f-5f9ss\" (UID: \"a0ba8ab3-65d2-4132-9896-80f1fb46b5c5\") " pod="openshift-ingress/e2e-inf-a-e43d8b-openshift-default-5578dcfc7f-5f9ss"
Oct 31 12:48:16.152467 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:16.152204    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0ba8ab3-65d2-4132-9896-80f1fb46b5c5-istio-envoy\") pod \"e2e-inf-a-e43d8b-openshift-default-5578dcfc7f-5f9ss\" (UID: \"a0ba8ab3-65d2-4132-9896-80f1fb46b5c5\") " pod="openshift-ingress/e2e-inf-a-e43d8b-openshift-default-5578dcfc7f-5f9ss"
Oct 31 12:48:16.220586 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:16.176597    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0ba8ab3-65d2-4132-9896-80f1fb46b5c5-istiod-ca-cert\") pod \"e2e-inf-a-e43d8b-openshift-default-5578dcfc7f-5f9ss\" (UID: \"a0ba8ab3-65d2-4132-9896-80f1fb46b5c5\") " pod="openshift-ingress/e2e-inf-a-e43d8b-openshift-default-5578dcfc7f-5f9ss"
Oct 31 12:48:16.348873 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:16.176693    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/simulator-w4-c67de2-kserve-868679cf9b-gqjbx" podUID="e104d7da-1fb2-4c99-a480-b4ae84c4f273" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:48:16.427514 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:16.410349    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0ba8ab3-65d2-4132-9896-80f1fb46b5c5-istio-podinfo\") pod \"e2e-inf-a-e43d8b-openshift-default-5578dcfc7f-5f9ss\" (UID: \"a0ba8ab3-65d2-4132-9896-80f1fb46b5c5\") " pod="openshift-ingress/e2e-inf-a-e43d8b-openshift-default-5578dcfc7f-5f9ss"
Oct 31 12:48:16.439612 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:16.438677    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x2qzv" podUID="c247ed2a-aa90-424b-9b7e-4f2936d67e3f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:48:16.481940 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:16.481798    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0ba8ab3-65d2-4132-9896-80f1fb46b5c5-istio-token\") pod \"e2e-inf-a-e43d8b-openshift-default-5578dcfc7f-5f9ss\" (UID: \"a0ba8ab3-65d2-4132-9896-80f1fb46b5c5\") " pod="openshift-ingress/e2e-inf-a-e43d8b-openshift-default-5578dcfc7f-5f9ss"
Oct 31 12:48:16.550986 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:16.513950    5060 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-2-w5-b25d3b-kserve-76f798d59f-vvzhg" podUID="394ed76c-84ac-4612-9acb-a7e36cb68d89" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/health\": context deadline exceeded"
Oct 31 12:48:16.590259 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:16.590063    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5c5815-cbc4t"]
Oct 31 12:48:16.613621 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:16.590695    5060 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-c67de2/simulator-w4-c67de2-kserve-868679cf9b-gqjbx" podUID="e104d7da-1fb2-4c99-a480-b4ae84c4f273" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:48:16.958525 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:16.950633    5060 patch_prober.go:29] interesting pod/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-hkvl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body=
Oct 31 12:48:16.958525 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:16.950694    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-hkvl5" podUID="4e300ab1-6fe3-4548-90b3-0cbb8099e114" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused"
Oct 31 12:48:16.958525 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:16.952605    5060 patch_prober.go:29] interesting pod/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-72mg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" start-of-body=
Oct 31 12:48:16.958525 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:16.952655    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-72mg2" podUID="346f4c45-c06a-4ae5-9fa0-c426467700f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused"
Oct 31 12:48:16.958525 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:16.952735    5060 patch_prober.go:29] interesting pod/e2e-api-b-bb3cf5-openshift-default-5fc978478f-mw7qm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body=
Oct 31 12:48:16.958525 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:16.952759    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-bb3cf5-openshift-default-5fc978478f-mw7qm" podUID="d9cb64fe-d6b4-439d-bc3a-8598b7cd512e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused"
Oct 31 12:48:17.365061 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:17.363793    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/premium-w2-e6c79d-kserve-6468bf46d4-h66f8" podUID="cd6a266f-8d12-4089-83ee-c75c716bf26c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Oct 31 12:48:17.408713 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:17.395718    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cg6vs\" (UniqueName: \"kubernetes.io/projected/a0ba8ab3-65d2-4132-9896-80f1fb46b5c5-kube-api-access-cg6vs\") pod \"e2e-inf-a-e43d8b-openshift-default-5578dcfc7f-5f9ss\" (UID: \"a0ba8ab3-65d2-4132-9896-80f1fb46b5c5\") " pod="openshift-ingress/e2e-inf-a-e43d8b-openshift-default-5578dcfc7f-5f9ss"
Oct 31 12:48:17.464532 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:17.418320    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/simulator-w4-c67de2-kserve-868679cf9b-gqjbx" podUID="e104d7da-1fb2-4c99-a480-b4ae84c4f273" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:48:17.555576 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:17.458798    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x2qzv" podUID="c247ed2a-aa90-424b-9b7e-4f2936d67e3f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded"
Oct 31 12:48:17.555576 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:17.459950    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-2-w5-b25d3b-kserve-76f798d59f-vvzhg" podUID="394ed76c-84ac-4612-9acb-a7e36cb68d89" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:48:17.649119 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:17.649082    5060 patch_prober.go:29] interesting pod/e2e-api-b-bb3cf5-openshift-default-5fc978478f-mw7qm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body=
Oct 31 12:48:17.649393 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:17.649365    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-bb3cf5-openshift-default-5fc978478f-mw7qm" podUID="d9cb64fe-d6b4-439d-bc3a-8598b7cd512e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused"
Oct 31 12:48:17.727516 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:17.716309    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-e43d8b-openshift-default-dockercfg-s49sm\""
Oct 31 12:48:17.760259 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:17.760221    5060 patch_prober.go:29] interesting pod/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-72mg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" start-of-body=
Oct 31 12:48:17.760589 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:17.760555    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-72mg2" podUID="346f4c45-c06a-4ae5-9fa0-c426467700f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused"
Oct 31 12:48:17.790912 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:17.776420    5060 patch_prober.go:29] interesting pod/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-hkvl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body=
Oct 31 12:48:17.871537 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:17.868700    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-hkvl5" podUID="4e300ab1-6fe3-4548-90b3-0cbb8099e114" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused"
Oct 31 12:48:17.871537 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:17.799841    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e43d8b-openshift-default-5578dcfc7f-5f9ss"
Oct 31 12:48:18.028520 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:18.028144    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5c5815-cbc4t" event={"ID":"e39c020a-e0e5-4a0c-9c52-3d63241cc13c","Type":"ContainerStarted","Data":"54d14f684f085cf769afb2c96a3844a33ad03c6e0b7de11cbf51ab9579ce1d7f"}
Oct 31 12:48:18.028520 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:18.028255    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e6c79d/premium-w2-e6c79d-kserve-6468bf46d4-h66f8"
Oct 31 12:48:18.028520 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:18.028296    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c67de2/simulator-w4-c67de2-kserve-868679cf9b-gqjbx"
Oct 31 12:48:18.028520 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:18.028331    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x2qzv"
Oct 31 12:48:18.028520 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:18.028364    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c67de2/simulator-w4-c67de2-kserve-868679cf9b-gqjbx"
Oct 31 12:48:18.367231 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:18.367188    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/premium-w2-e6c79d-kserve-6468bf46d4-h66f8" podUID="cd6a266f-8d12-4089-83ee-c75c716bf26c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:48:18.581569 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:18.572957    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x2qzv" podUID="c247ed2a-aa90-424b-9b7e-4f2936d67e3f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:48:18.678537 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:18.657879    5060 patch_prober.go:29] interesting pod/e2e-api-b-bb3cf5-openshift-default-5fc978478f-mw7qm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body=
Oct 31 12:48:18.678537 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:18.657937    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-bb3cf5-openshift-default-5fc978478f-mw7qm" podUID="d9cb64fe-d6b4-439d-bc3a-8598b7cd512e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused"
Oct 31 12:48:18.775196 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:18.774194    5060 patch_prober.go:29] interesting pod/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-hkvl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body=
Oct 31 12:48:18.775196 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:18.774252    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-hkvl5" podUID="4e300ab1-6fe3-4548-90b3-0cbb8099e114" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused"
Oct 31 12:48:18.775196 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:18.775094    5060 patch_prober.go:29] interesting pod/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-72mg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" start-of-body=
Oct 31 12:48:18.775196 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:18.775133    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-72mg2" podUID="346f4c45-c06a-4ae5-9fa0-c426467700f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused"
Oct 31 12:48:18.812888 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:18.812543    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d250eff7-d4jg7"]
Oct 31 12:48:18.945908 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:18.945818    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5c5815-cbc4t" event={"ID":"e39c020a-e0e5-4a0c-9c52-3d63241cc13c","Type":"ContainerStarted","Data":"4c8798e6a22a01eb0ee9bdfd352175b6ac22c4444cfd5d89c4e08e492a2f20d6"}
Oct 31 12:48:18.988544 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:18.975904    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d250eff7-d4jg7"]
Oct 31 12:48:18.995039 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:18.994900    5060 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-e6c79d/premium-w2-e6c79d-kserve-6468bf46d4-h66f8" podUID="cd6a266f-8d12-4089-83ee-c75c716bf26c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:48:19.220567 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:19.189255    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/simulator-w4-c67de2-kserve-868679cf9b-gqjbx" podUID="e104d7da-1fb2-4c99-a480-b4ae84c4f273" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded"
Oct 31 12:48:19.386516 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:19.368943    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/premium-w2-e6c79d-kserve-6468bf46d4-h66f8" podUID="cd6a266f-8d12-4089-83ee-c75c716bf26c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:48:19.499562 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:19.495642    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9baea4fa-5a62-48d0-b065-8167e31b8236" path="/var/lib/kubelet/pods/9baea4fa-5a62-48d0-b065-8167e31b8236/volumes"
Oct 31 12:48:19.513513 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:19.507589    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x2qzv"
Oct 31 12:48:19.548521 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:19.547838    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e6c79d/premium-w2-e6c79d-kserve-6468bf46d4-h66f8"
Oct 31 12:48:19.662871 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:19.662832    5060 patch_prober.go:29] interesting pod/e2e-api-b-bb3cf5-openshift-default-5fc978478f-mw7qm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body=
Oct 31 12:48:19.670631 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:19.670588    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-bb3cf5-openshift-default-5fc978478f-mw7qm" podUID="d9cb64fe-d6b4-439d-bc3a-8598b7cd512e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused"
Oct 31 12:48:19.751235 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:19.751156    5060 patch_prober.go:29] interesting pod/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-72mg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" start-of-body=
Oct 31 12:48:19.751921 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:19.751871    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-72mg2" podUID="346f4c45-c06a-4ae5-9fa0-c426467700f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused"
Oct 31 12:48:19.928757 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:19.928699    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5c5815-cbc4t" podStartSLOduration=6.928677398 podStartE2EDuration="6.928677398s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:48:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:48:19.011957672 +0000 UTC m=+1262.115025386" watchObservedRunningTime="2026-10-31 12:48:19.928677398 +0000 UTC m=+1263.031745111"
Oct 31 12:48:20.015571 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:20.015454    5060 patch_prober.go:29] interesting pod/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-hkvl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body=
Oct 31 12:48:20.105547 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:20.015533    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-hkvl5" podUID="4e300ab1-6fe3-4548-90b3-0cbb8099e114" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused"
Oct 31 12:48:20.105547 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:20.026680    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-bb3cf5-openshift-default-5fc978478f-mw7qm"
Oct 31 12:48:20.452859 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:20.452807    5060 generic.go:356] "Generic (PLEG): container finished" podID="e39c020a-e0e5-4a0c-9c52-3d63241cc13c" containerID="4c8798e6a22a01eb0ee9bdfd352175b6ac22c4444cfd5d89c4e08e492a2f20d6" exitCode=0
Oct 31 12:48:20.454328 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:20.453437    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5c5815-cbc4t" event={"ID":"e39c020a-e0e5-4a0c-9c52-3d63241cc13c","Type":"ContainerDied","Data":"4c8798e6a22a01eb0ee9bdfd352175b6ac22c4444cfd5d89c4e08e492a2f20d6"}
Oct 31 12:48:20.481517 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:20.480671    5060 scope.go:122] "RemoveContainer" containerID="f77ceda4cc9823ffcb2589768f5fc6160d621266ba1e4eca73b51850f51a0f7e"
Oct 31 12:48:20.644762 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:20.644634    5060 patch_prober.go:29] interesting pod/e2e-api-b-bb3cf5-openshift-default-5fc978478f-mw7qm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body=
Oct 31 12:48:20.644762 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:20.644702    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-bb3cf5-openshift-default-5fc978478f-mw7qm" podUID="d9cb64fe-d6b4-439d-bc3a-8598b7cd512e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused"
Oct 31 12:48:20.646678 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:20.645234    5060 scope.go:122] "RemoveContainer" containerID="84545d817f673029f76bdfd04e79ff41953b6497153d89a6be592d06ce23846c"
Oct 31 12:48:20.732694 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:20.732643    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-e43d8b-openshift-default-5578dcfc7f-5f9ss"]
Oct 31 12:48:20.740529 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:20.740354    5060 scope.go:122] "RemoveContainer" containerID="ace316680fc1ffa9d4eee2000889ae0154c1fe4e42e7eac3928cbabd872d31c3"
Oct 31 12:48:20.808660 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:20.805928    5060 patch_prober.go:29] interesting pod/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-hkvl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body=
Oct 31 12:48:20.808660 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:20.808279    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-hkvl5" podUID="4e300ab1-6fe3-4548-90b3-0cbb8099e114" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused"
Oct 31 12:48:20.808660 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:20.807812    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:48:20.808974 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:20.808682    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:48:20.808974 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:20.808952    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:48:20.888161 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:20.888129    5060 scope.go:122] "RemoveContainer" containerID="df534668b3fcb84b8d19acb83414417e4ceefa93fe467899e36586ec15857337"
Oct 31 12:48:20.942731 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:20.941743    5060 scope.go:122] "RemoveContainer" containerID="363780b2bd626836b7147ef4cc43c5ea9d278e3a1a8061ddab1f83c3d3405877"
Oct 31 12:48:21.604768 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:21.604737    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-72mg2"
Oct 31 12:48:21.605326 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:21.605305    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-711af2-mpngx"]
Oct 31 12:48:21.605438 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:21.605427    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-711af2-mpngx"]
Oct 31 12:48:21.607248 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:21.607110    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-72mg2"
Oct 31 12:48:21.640089 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:21.640053    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-72mg2"
Oct 31 12:48:21.774933 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:21.774896    5060 patch_prober.go:29] interesting pod/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-hkvl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body=
Oct 31 12:48:21.775324 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:21.775285    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-hkvl5" podUID="4e300ab1-6fe3-4548-90b3-0cbb8099e114" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused"
Oct 31 12:48:21.862626 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:21.860603    5060 scope.go:122] "RemoveContainer" containerID="b877904e4fff437fe599080c1e9653a1423e9b402fc56e4bae1812d795831b8a"
Oct 31 12:48:21.957056 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:21.956970    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e43d8b-openshift-default-5578dcfc7f-5f9ss" event={"ID":"a0ba8ab3-65d2-4132-9896-80f1fb46b5c5","Type":"ContainerStarted","Data":"2b6b4095ec442fb7975d5b4fb0c6e4e415f999034ec793b89ea631ebaaf571d1"}
Oct 31 12:48:22.163690 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:22.163637    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/simulator-w4-c67de2-kserve-868679cf9b-gqjbx" podUID="e104d7da-1fb2-4c99-a480-b4ae84c4f273" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:48:22.223954 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:22.223924    5060 scope.go:122] "RemoveContainer" containerID="63024a86359dffc58137ad85d25b325d6b23be61a1eebf1de36f0f738c9cb4e9"
Oct 31 12:48:22.271207 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:22.270304    5060 scope.go:122] "RemoveContainer" containerID="e0c4693aff8c581be0c33040fe2aaf18843e8f01e4f5f5c9ce79dd1ca7ebebd0"
Oct 31 12:48:22.332090 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:22.327195    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-bb3cf5-openshift-default-5fc978478f-mw7qm"
Oct 31 12:48:22.337309 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:22.337241    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-bb3cf5-openshift-default-5fc978478f-mw7qm"
Oct 31 12:48:22.342610 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:22.341964    5060 scope.go:122] "RemoveContainer" containerID="60a0b48f406bbffd3817a09f05f357296de87247b995a466a99345eb83d4ef05"
Oct 31 12:48:22.405053 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:22.404644    5060 scope.go:122] "RemoveContainer" containerID="6d08814ed4ebc1038e3e29f1caaaf69061150f60d4bbd52a2af4921e29e514a1"
Oct 31 12:48:22.449564 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:22.448674    5060 patch_prober.go:29] interesting pod/e2e-api-b-711af2-openshift-default-6549b5d6f9-zdqxc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 12:48:22.449564 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:22.448786    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-711af2-openshift-default-6549b5d6f9-zdqxc" podUID="440e8c3e-006e-4a6b-82d1-5fb80f5e26d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:48:22.457171 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:22.456788    5060 scope.go:122] "RemoveContainer" containerID="7a043a877747c7ce8331704c988beb5ea63e41a2b8d6ca0fe9460811ec1174e2"
Oct 31 12:48:22.521448 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:22.521420    5060 scope.go:122] "RemoveContainer" containerID="f168eb5d6e15c60542c75f1826b8ba6c0d5646b25cd64214fd3630a623aa0d0c"
Oct 31 12:48:22.587527 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:22.587135    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5c5815-cbc4t"
Oct 31 12:48:22.659673 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:22.654242    5060 scope.go:122] "RemoveContainer" containerID="971de069a484c920446fef53a5cb36ac5446585d39329505dd716333b77d09cb"
Oct 31 12:48:22.691956 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:22.691637    5060 scope.go:122] "RemoveContainer" containerID="1ddef93f9665b2e472ee2ef78c968665ca2dc31a0208f750b3cfb1c9fb258c58"
Oct 31 12:48:22.692096 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:22.692052    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e39c020a-e0e5-4a0c-9c52-3d63241cc13c-kube-api-access-h5psx\" (UniqueName: \"kubernetes.io/projected/e39c020a-e0e5-4a0c-9c52-3d63241cc13c-kube-api-access-h5psx\") pod \"e39c020a-e0e5-4a0c-9c52-3d63241cc13c\" (UID: \"e39c020a-e0e5-4a0c-9c52-3d63241cc13c\") "
Oct 31 12:48:22.692168 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:22.692116    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e39c020a-e0e5-4a0c-9c52-3d63241cc13c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e39c020a-e0e5-4a0c-9c52-3d63241cc13c-maas-api-xks-ca\") pod \"e39c020a-e0e5-4a0c-9c52-3d63241cc13c\" (UID: \"e39c020a-e0e5-4a0c-9c52-3d63241cc13c\") "
Oct 31 12:48:22.692586 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:22.692247    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e39c020a-e0e5-4a0c-9c52-3d63241cc13c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e39c020a-e0e5-4a0c-9c52-3d63241cc13c-maas-api-openshift-ca\") pod \"e39c020a-e0e5-4a0c-9c52-3d63241cc13c\" (UID: \"e39c020a-e0e5-4a0c-9c52-3d63241cc13c\") "
Oct 31 12:48:22.694831 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:22.694806    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e39c020a-e0e5-4a0c-9c52-3d63241cc13c-maas-api-openshift-ca" pod "e39c020a-e0e5-4a0c-9c52-3d63241cc13c" (UID: "e39c020a-e0e5-4a0c-9c52-3d63241cc13c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:48:22.696549 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:22.696512    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e39c020a-e0e5-4a0c-9c52-3d63241cc13c-maas-api-xks-ca" pod "e39c020a-e0e5-4a0c-9c52-3d63241cc13c" (UID: "e39c020a-e0e5-4a0c-9c52-3d63241cc13c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:48:22.697971 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:22.697938    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e39c020a-e0e5-4a0c-9c52-3d63241cc13c-kube-api-access-h5psx" pod "e39c020a-e0e5-4a0c-9c52-3d63241cc13c" (UID: "e39c020a-e0e5-4a0c-9c52-3d63241cc13c"). InnerVolumeSpecName "kube-api-access-h5psx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:48:22.780800 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:22.780770    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-hkvl5"
Oct 31 12:48:22.782674 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:22.782646    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-hkvl5"
Oct 31 12:48:22.783895 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:22.783819    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-hkvl5"
Oct 31 12:48:22.798780 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:22.795336    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5psx\" (UniqueName: \"kubernetes.io/projected/e39c020a-e0e5-4a0c-9c52-3d63241cc13c-kube-api-access-h5psx\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:48:22.798780 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:22.795377    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e39c020a-e0e5-4a0c-9c52-3d63241cc13c-maas-api-xks-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:48:22.798780 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:22.795394    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e39c020a-e0e5-4a0c-9c52-3d63241cc13c-maas-api-openshift-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:48:22.983716 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:22.983681    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e43d8b-openshift-default-5578dcfc7f-5f9ss" event={"ID":"a0ba8ab3-65d2-4132-9896-80f1fb46b5c5","Type":"ContainerStarted","Data":"643c50baeec7939f03f59f571525c956c954acc4db8b464eb73930c894d69b75"}
Oct 31 12:48:22.986662 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:22.986639    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5c5815-cbc4t"
Oct 31 12:48:22.990334 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:22.989224    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5c5815-cbc4t" event={"ID":"e39c020a-e0e5-4a0c-9c52-3d63241cc13c","Type":"ContainerDied","Data":"54d14f684f085cf769afb2c96a3844a33ad03c6e0b7de11cbf51ab9579ce1d7f"}
Oct 31 12:48:22.990334 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:22.989276    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="54d14f684f085cf769afb2c96a3844a33ad03c6e0b7de11cbf51ab9579ce1d7f"
Oct 31 12:48:23.026120 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:23.026046    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-e43d8b-openshift-default-5578dcfc7f-5f9ss" podStartSLOduration=9.026025843 podStartE2EDuration="9.026025843s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:48:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:48:23.018681377 +0000 UTC m=+1266.121749124" watchObservedRunningTime="2026-10-31 12:48:23.026025843 +0000 UTC m=+1266.129093556"
Oct 31 12:48:23.056975 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:23.056935    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c874d2e9-5ace-4faf-a1dd-d497e34b73d3" path="/var/lib/kubelet/pods/c874d2e9-5ace-4faf-a1dd-d497e34b73d3/volumes"
Oct 31 12:48:23.713871 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:23.713186    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-sd68f"]
Oct 31 12:48:23.713871 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:23.713419    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-sd68f" podUID="4129ab4f-7756-4a6d-8d7f-9dff5d0f1c57" containerName="istio-proxy" containerID="cri-o://b93a12d0d99a0df2c3d5d6cf01e704001b745c2d41bb5bb35ff426730500162d" gracePeriod=30
Oct 31 12:48:23.800389 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:23.800249    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-e43d8b-openshift-default-5578dcfc7f-5f9ss"
Oct 31 12:48:23.846886 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:23.834265    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-e43d8b-openshift-default-5578dcfc7f-5f9ss"
Oct 31 12:48:24.001834 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:24.001796    5060 generic.go:356] "Generic (PLEG): container finished" podID="440e8c3e-006e-4a6b-82d1-5fb80f5e26d1" containerID="47e7f181c25f923c5479a64b7786d08ed6b5f4e7a0e117ea035685d1dd2d25eb" exitCode=0
Oct 31 12:48:24.002229 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:24.001880    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-711af2-openshift-default-6549b5d6f9-zdqxc" event={"ID":"440e8c3e-006e-4a6b-82d1-5fb80f5e26d1","Type":"ContainerDied","Data":"47e7f181c25f923c5479a64b7786d08ed6b5f4e7a0e117ea035685d1dd2d25eb"}
Oct 31 12:48:24.003165 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:24.003098    5060 scope.go:122] "RemoveContainer" containerID="47e7f181c25f923c5479a64b7786d08ed6b5f4e7a0e117ea035685d1dd2d25eb"
Oct 31 12:48:24.003520 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:24.003446    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-e43d8b-openshift-default-5578dcfc7f-5f9ss"
Oct 31 12:48:24.004970 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:24.004950    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-e43d8b-openshift-default-5578dcfc7f-5f9ss"
Oct 31 12:48:24.005137 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:24.005106    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:48:24.005202 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:24.005178    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:48:24.005262 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:24.005216    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:48:24.066219 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:24.066194    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-bbjl7"]
Oct 31 12:48:24.066379 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:24.066361    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-bbjl7" podUID="09b9b34e-eb53-4829-aed9-abb0b81675d0" containerName="istio-proxy" containerID="cri-o://07912adfcfe0dac99f69adef566eb21ca4e3b211b8b78efefb0506f1e7a058db" gracePeriod=30
Oct 31 12:48:25.017783 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:25.017747    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-711af2-openshift-default-6549b5d6f9-zdqxc" event={"ID":"440e8c3e-006e-4a6b-82d1-5fb80f5e26d1","Type":"ContainerStarted","Data":"786c9569efb95b9c63b956c308e0ae69c7468bd9c0e9bddb3bf4e16532c1578b"}
Oct 31 12:48:25.389908 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:25.389867    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-711af2-openshift-default-6549b5d6f9-zdqxc"
Oct 31 12:48:25.390110 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:25.389999    5060 patch_prober.go:29] interesting pod/e2e-api-b-711af2-openshift-default-6549b5d6f9-zdqxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body=
Oct 31 12:48:25.390110 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:25.390066    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-711af2-openshift-default-6549b5d6f9-zdqxc" podUID="440e8c3e-006e-4a6b-82d1-5fb80f5e26d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused"
Oct 31 12:48:26.395062 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:26.391937    5060 patch_prober.go:29] interesting pod/e2e-api-b-711af2-openshift-default-6549b5d6f9-zdqxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body=
Oct 31 12:48:26.395062 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:26.392015    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-711af2-openshift-default-6549b5d6f9-zdqxc" podUID="440e8c3e-006e-4a6b-82d1-5fb80f5e26d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused"
Oct 31 12:48:27.392066 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:27.392029    5060 patch_prober.go:29] interesting pod/e2e-api-b-711af2-openshift-default-6549b5d6f9-zdqxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body=
Oct 31 12:48:27.392367 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:27.392334    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-711af2-openshift-default-6549b5d6f9-zdqxc" podUID="440e8c3e-006e-4a6b-82d1-5fb80f5e26d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused"
Oct 31 12:48:28.097415 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:28.097215    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44c7d33e-8f9zt"]
Oct 31 12:48:28.106093 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:28.106062    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44c7d33e-8f9zt"]
Oct 31 12:48:28.391697 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:28.391601    5060 patch_prober.go:29] interesting pod/e2e-api-b-711af2-openshift-default-6549b5d6f9-zdqxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body=
Oct 31 12:48:28.391996 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:28.391949    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-711af2-openshift-default-6549b5d6f9-zdqxc" podUID="440e8c3e-006e-4a6b-82d1-5fb80f5e26d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused"
Oct 31 12:48:29.086268 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:29.086182    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a85046f-2e8f-4dbf-8ce1-26f18bfda012" path="/var/lib/kubelet/pods/5a85046f-2e8f-4dbf-8ce1-26f18bfda012/volumes"
Oct 31 12:48:29.152389 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:29.152356    5060 generic.go:356] "Generic (PLEG): container finished" podID="4129ab4f-7756-4a6d-8d7f-9dff5d0f1c57" containerID="b93a12d0d99a0df2c3d5d6cf01e704001b745c2d41bb5bb35ff426730500162d" exitCode=0
Oct 31 12:48:29.152726 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:29.152423    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-sd68f" event={"ID":"4129ab4f-7756-4a6d-8d7f-9dff5d0f1c57","Type":"ContainerDied","Data":"b93a12d0d99a0df2c3d5d6cf01e704001b745c2d41bb5bb35ff426730500162d"}
Oct 31 12:48:29.225919 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:29.225891    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-sd68f"
Oct 31 12:48:29.326437 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:29.318084    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4129ab4f-7756-4a6d-8d7f-9dff5d0f1c57-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4129ab4f-7756-4a6d-8d7f-9dff5d0f1c57-workload-certs\") pod \"4129ab4f-7756-4a6d-8d7f-9dff5d0f1c57\" (UID: \"4129ab4f-7756-4a6d-8d7f-9dff5d0f1c57\") "
Oct 31 12:48:29.326437 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:29.318141    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4129ab4f-7756-4a6d-8d7f-9dff5d0f1c57-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4129ab4f-7756-4a6d-8d7f-9dff5d0f1c57-istio-envoy\") pod \"4129ab4f-7756-4a6d-8d7f-9dff5d0f1c57\" (UID: \"4129ab4f-7756-4a6d-8d7f-9dff5d0f1c57\") "
Oct 31 12:48:29.326437 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:29.318245    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4129ab4f-7756-4a6d-8d7f-9dff5d0f1c57-istio-token\" (UniqueName: \"kubernetes.io/projected/4129ab4f-7756-4a6d-8d7f-9dff5d0f1c57-istio-token\") pod \"4129ab4f-7756-4a6d-8d7f-9dff5d0f1c57\" (UID: \"4129ab4f-7756-4a6d-8d7f-9dff5d0f1c57\") "
Oct 31 12:48:29.326437 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:29.318346    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4129ab4f-7756-4a6d-8d7f-9dff5d0f1c57-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4129ab4f-7756-4a6d-8d7f-9dff5d0f1c57-istio-data\") pod \"4129ab4f-7756-4a6d-8d7f-9dff5d0f1c57\" (UID: \"4129ab4f-7756-4a6d-8d7f-9dff5d0f1c57\") "
Oct 31 12:48:29.326437 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:29.318438    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4129ab4f-7756-4a6d-8d7f-9dff5d0f1c57-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4129ab4f-7756-4a6d-8d7f-9dff5d0f1c57-istiod-ca-cert\") pod \"4129ab4f-7756-4a6d-8d7f-9dff5d0f1c57\" (UID: \"4129ab4f-7756-4a6d-8d7f-9dff5d0f1c57\") "
Oct 31 12:48:29.326437 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:29.318476    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4129ab4f-7756-4a6d-8d7f-9dff5d0f1c57-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4129ab4f-7756-4a6d-8d7f-9dff5d0f1c57-workload-socket\") pod \"4129ab4f-7756-4a6d-8d7f-9dff5d0f1c57\" (UID: \"4129ab4f-7756-4a6d-8d7f-9dff5d0f1c57\") "
Oct 31 12:48:29.326437 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:29.318577    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4129ab4f-7756-4a6d-8d7f-9dff5d0f1c57-kube-api-access-h546b\" (UniqueName: \"kubernetes.io/projected/4129ab4f-7756-4a6d-8d7f-9dff5d0f1c57-kube-api-access-h546b\") pod \"4129ab4f-7756-4a6d-8d7f-9dff5d0f1c57\" (UID: \"4129ab4f-7756-4a6d-8d7f-9dff5d0f1c57\") "
Oct 31 12:48:29.326437 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:29.318638    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4129ab4f-7756-4a6d-8d7f-9dff5d0f1c57-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4129ab4f-7756-4a6d-8d7f-9dff5d0f1c57-credential-socket\") pod \"4129ab4f-7756-4a6d-8d7f-9dff5d0f1c57\" (UID: \"4129ab4f-7756-4a6d-8d7f-9dff5d0f1c57\") "
Oct 31 12:48:29.326437 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:29.318722    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4129ab4f-7756-4a6d-8d7f-9dff5d0f1c57-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4129ab4f-7756-4a6d-8d7f-9dff5d0f1c57-istio-podinfo\") pod \"4129ab4f-7756-4a6d-8d7f-9dff5d0f1c57\" (UID: \"4129ab4f-7756-4a6d-8d7f-9dff5d0f1c57\") "
Oct 31 12:48:29.326437 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:29.320241    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4129ab4f-7756-4a6d-8d7f-9dff5d0f1c57-istio-data" pod "4129ab4f-7756-4a6d-8d7f-9dff5d0f1c57" (UID: "4129ab4f-7756-4a6d-8d7f-9dff5d0f1c57"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:48:29.326437 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:29.322390    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4129ab4f-7756-4a6d-8d7f-9dff5d0f1c57-workload-socket" pod "4129ab4f-7756-4a6d-8d7f-9dff5d0f1c57" (UID: "4129ab4f-7756-4a6d-8d7f-9dff5d0f1c57"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:48:29.326437 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:29.322605    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4129ab4f-7756-4a6d-8d7f-9dff5d0f1c57-credential-socket" pod "4129ab4f-7756-4a6d-8d7f-9dff5d0f1c57" (UID: "4129ab4f-7756-4a6d-8d7f-9dff5d0f1c57"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:48:29.326437 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:29.322768    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4129ab4f-7756-4a6d-8d7f-9dff5d0f1c57-istiod-ca-cert" pod "4129ab4f-7756-4a6d-8d7f-9dff5d0f1c57" (UID: "4129ab4f-7756-4a6d-8d7f-9dff5d0f1c57"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:48:29.326437 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:29.322821    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4129ab4f-7756-4a6d-8d7f-9dff5d0f1c57-workload-certs" pod "4129ab4f-7756-4a6d-8d7f-9dff5d0f1c57" (UID: "4129ab4f-7756-4a6d-8d7f-9dff5d0f1c57"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:48:29.326437 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:29.325224    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4129ab4f-7756-4a6d-8d7f-9dff5d0f1c57-istio-token" pod "4129ab4f-7756-4a6d-8d7f-9dff5d0f1c57" (UID: "4129ab4f-7756-4a6d-8d7f-9dff5d0f1c57"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:48:29.328743 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:29.328657    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4129ab4f-7756-4a6d-8d7f-9dff5d0f1c57-kube-api-access-h546b" pod "4129ab4f-7756-4a6d-8d7f-9dff5d0f1c57" (UID: "4129ab4f-7756-4a6d-8d7f-9dff5d0f1c57"). InnerVolumeSpecName "kube-api-access-h546b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:48:29.329274 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:29.329239    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4129ab4f-7756-4a6d-8d7f-9dff5d0f1c57-istio-podinfo" pod "4129ab4f-7756-4a6d-8d7f-9dff5d0f1c57" (UID: "4129ab4f-7756-4a6d-8d7f-9dff5d0f1c57"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 12:48:29.329384 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:29.329330    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4129ab4f-7756-4a6d-8d7f-9dff5d0f1c57-istio-envoy" pod "4129ab4f-7756-4a6d-8d7f-9dff5d0f1c57" (UID: "4129ab4f-7756-4a6d-8d7f-9dff5d0f1c57"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:48:29.422818 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:29.421581    5060 patch_prober.go:29] interesting pod/e2e-api-b-711af2-openshift-default-6549b5d6f9-zdqxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body=
Oct 31 12:48:29.422818 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:29.421637    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-711af2-openshift-default-6549b5d6f9-zdqxc" podUID="440e8c3e-006e-4a6b-82d1-5fb80f5e26d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused"
Oct 31 12:48:29.423097 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:29.422838    5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4129ab4f-7756-4a6d-8d7f-9dff5d0f1c57-istio-data\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:48:29.423097 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:29.422867    5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4129ab4f-7756-4a6d-8d7f-9dff5d0f1c57-istiod-ca-cert\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:48:29.423097 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:29.422882    5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4129ab4f-7756-4a6d-8d7f-9dff5d0f1c57-workload-socket\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:48:29.423097 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:29.422896    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h546b\" (UniqueName: \"kubernetes.io/projected/4129ab4f-7756-4a6d-8d7f-9dff5d0f1c57-kube-api-access-h546b\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:48:29.423097 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:29.422912    5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4129ab4f-7756-4a6d-8d7f-9dff5d0f1c57-credential-socket\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:48:29.423097 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:29.422926    5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4129ab4f-7756-4a6d-8d7f-9dff5d0f1c57-istio-podinfo\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:48:29.423097 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:29.422939    5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4129ab4f-7756-4a6d-8d7f-9dff5d0f1c57-workload-certs\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:48:29.423097 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:29.422951    5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4129ab4f-7756-4a6d-8d7f-9dff5d0f1c57-istio-envoy\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:48:29.423097 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:29.422991    5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4129ab4f-7756-4a6d-8d7f-9dff5d0f1c57-istio-token\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:48:29.514271 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:29.514177    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-bbjl7"
Oct 31 12:48:29.525037 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:29.524025    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09b9b34e-eb53-4829-aed9-abb0b81675d0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09b9b34e-eb53-4829-aed9-abb0b81675d0-istiod-ca-cert\") pod \"09b9b34e-eb53-4829-aed9-abb0b81675d0\" (UID: \"09b9b34e-eb53-4829-aed9-abb0b81675d0\") "
Oct 31 12:48:29.525037 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:29.524073    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09b9b34e-eb53-4829-aed9-abb0b81675d0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09b9b34e-eb53-4829-aed9-abb0b81675d0-workload-certs\") pod \"09b9b34e-eb53-4829-aed9-abb0b81675d0\" (UID: \"09b9b34e-eb53-4829-aed9-abb0b81675d0\") "
Oct 31 12:48:29.525037 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:29.524119    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09b9b34e-eb53-4829-aed9-abb0b81675d0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09b9b34e-eb53-4829-aed9-abb0b81675d0-istio-envoy\") pod \"09b9b34e-eb53-4829-aed9-abb0b81675d0\" (UID: \"09b9b34e-eb53-4829-aed9-abb0b81675d0\") "
Oct 31 12:48:29.525037 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:29.524141    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/09b9b34e-eb53-4829-aed9-abb0b81675d0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09b9b34e-eb53-4829-aed9-abb0b81675d0-istio-podinfo\") pod \"09b9b34e-eb53-4829-aed9-abb0b81675d0\" (UID: \"09b9b34e-eb53-4829-aed9-abb0b81675d0\") "
Oct 31 12:48:29.525037 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:29.524168    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09b9b34e-eb53-4829-aed9-abb0b81675d0-istio-token\" (UniqueName: \"kubernetes.io/projected/09b9b34e-eb53-4829-aed9-abb0b81675d0-istio-token\") pod \"09b9b34e-eb53-4829-aed9-abb0b81675d0\" (UID: \"09b9b34e-eb53-4829-aed9-abb0b81675d0\") "
Oct 31 12:48:29.525037 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:29.524205    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09b9b34e-eb53-4829-aed9-abb0b81675d0-kube-api-access-wsxn6\" (UniqueName: \"kubernetes.io/projected/09b9b34e-eb53-4829-aed9-abb0b81675d0-kube-api-access-wsxn6\") pod \"09b9b34e-eb53-4829-aed9-abb0b81675d0\" (UID: \"09b9b34e-eb53-4829-aed9-abb0b81675d0\") "
Oct 31 12:48:29.525037 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:29.524267    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09b9b34e-eb53-4829-aed9-abb0b81675d0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09b9b34e-eb53-4829-aed9-abb0b81675d0-credential-socket\") pod \"09b9b34e-eb53-4829-aed9-abb0b81675d0\" (UID: \"09b9b34e-eb53-4829-aed9-abb0b81675d0\") "
Oct 31 12:48:29.525037 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:29.524294    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09b9b34e-eb53-4829-aed9-abb0b81675d0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09b9b34e-eb53-4829-aed9-abb0b81675d0-workload-socket\") pod \"09b9b34e-eb53-4829-aed9-abb0b81675d0\" (UID: \"09b9b34e-eb53-4829-aed9-abb0b81675d0\") "
Oct 31 12:48:29.525037 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:29.524360    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09b9b34e-eb53-4829-aed9-abb0b81675d0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09b9b34e-eb53-4829-aed9-abb0b81675d0-istio-data\") pod \"09b9b34e-eb53-4829-aed9-abb0b81675d0\" (UID: \"09b9b34e-eb53-4829-aed9-abb0b81675d0\") "
Oct 31 12:48:29.525037 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:29.524730    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09b9b34e-eb53-4829-aed9-abb0b81675d0-workload-certs" pod "09b9b34e-eb53-4829-aed9-abb0b81675d0" (UID: "09b9b34e-eb53-4829-aed9-abb0b81675d0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:48:29.525037 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:29.524767    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09b9b34e-eb53-4829-aed9-abb0b81675d0-istiod-ca-cert" pod "09b9b34e-eb53-4829-aed9-abb0b81675d0" (UID: "09b9b34e-eb53-4829-aed9-abb0b81675d0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:48:29.525037 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:29.525007    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09b9b34e-eb53-4829-aed9-abb0b81675d0-istio-data" pod "09b9b34e-eb53-4829-aed9-abb0b81675d0" (UID: "09b9b34e-eb53-4829-aed9-abb0b81675d0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:48:29.525976 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:29.525389    5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09b9b34e-eb53-4829-aed9-abb0b81675d0-istiod-ca-cert\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:48:29.525976 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:29.525413    5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09b9b34e-eb53-4829-aed9-abb0b81675d0-workload-certs\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:48:29.525976 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:29.525426    5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09b9b34e-eb53-4829-aed9-abb0b81675d0-istio-data\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:48:29.525976 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:29.525561    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09b9b34e-eb53-4829-aed9-abb0b81675d0-credential-socket" pod "09b9b34e-eb53-4829-aed9-abb0b81675d0" (UID: "09b9b34e-eb53-4829-aed9-abb0b81675d0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:48:29.525976 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:29.525616    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09b9b34e-eb53-4829-aed9-abb0b81675d0-workload-socket" pod "09b9b34e-eb53-4829-aed9-abb0b81675d0" (UID: "09b9b34e-eb53-4829-aed9-abb0b81675d0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:48:29.527840 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:29.527807    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09b9b34e-eb53-4829-aed9-abb0b81675d0-istio-envoy" pod "09b9b34e-eb53-4829-aed9-abb0b81675d0" (UID: "09b9b34e-eb53-4829-aed9-abb0b81675d0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:48:29.528073 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:29.528049    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09b9b34e-eb53-4829-aed9-abb0b81675d0-kube-api-access-wsxn6" pod "09b9b34e-eb53-4829-aed9-abb0b81675d0" (UID: "09b9b34e-eb53-4829-aed9-abb0b81675d0"). InnerVolumeSpecName "kube-api-access-wsxn6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:48:29.540267 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:29.528089    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09b9b34e-eb53-4829-aed9-abb0b81675d0-istio-token" pod "09b9b34e-eb53-4829-aed9-abb0b81675d0" (UID: "09b9b34e-eb53-4829-aed9-abb0b81675d0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:48:29.579571 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:29.577873    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/09b9b34e-eb53-4829-aed9-abb0b81675d0-istio-podinfo" pod "09b9b34e-eb53-4829-aed9-abb0b81675d0" (UID: "09b9b34e-eb53-4829-aed9-abb0b81675d0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 12:48:29.743432 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:29.743165    5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09b9b34e-eb53-4829-aed9-abb0b81675d0-credential-socket\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:48:29.743432 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:29.743208    5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09b9b34e-eb53-4829-aed9-abb0b81675d0-workload-socket\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:48:29.743432 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:29.743225    5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09b9b34e-eb53-4829-aed9-abb0b81675d0-istio-envoy\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:48:29.743432 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:29.743239    5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09b9b34e-eb53-4829-aed9-abb0b81675d0-istio-podinfo\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:48:29.743432 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:29.743328    5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09b9b34e-eb53-4829-aed9-abb0b81675d0-istio-token\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:48:29.743432 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:29.743345    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wsxn6\" (UniqueName: \"kubernetes.io/projected/09b9b34e-eb53-4829-aed9-abb0b81675d0-kube-api-access-wsxn6\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:48:30.032661 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:48:30.032623    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7phb" containerName="extractor"
Oct 31 12:48:30.199575 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:30.197417    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-sd68f"
Oct 31 12:48:30.199575 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:30.197473    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-sd68f" event={"ID":"4129ab4f-7756-4a6d-8d7f-9dff5d0f1c57","Type":"ContainerDied","Data":"977487d36393140b45690701ef80d87813f658a59a4c69355a10d0a42803f1c3"}
Oct 31 12:48:30.199575 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:30.197546    5060 scope.go:122] "RemoveContainer" containerID="b93a12d0d99a0df2c3d5d6cf01e704001b745c2d41bb5bb35ff426730500162d"
Oct 31 12:48:30.204698 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:30.204661    5060 generic.go:356] "Generic (PLEG): container finished" podID="09b9b34e-eb53-4829-aed9-abb0b81675d0" containerID="07912adfcfe0dac99f69adef566eb21ca4e3b211b8b78efefb0506f1e7a058db" exitCode=0
Oct 31 12:48:30.204839 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:30.204746    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-bbjl7" event={"ID":"09b9b34e-eb53-4829-aed9-abb0b81675d0","Type":"ContainerDied","Data":"07912adfcfe0dac99f69adef566eb21ca4e3b211b8b78efefb0506f1e7a058db"}
Oct 31 12:48:30.204839 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:30.204809    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-bbjl7"
Oct 31 12:48:30.204980 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:30.204872    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-bbjl7" event={"ID":"09b9b34e-eb53-4829-aed9-abb0b81675d0","Type":"ContainerDied","Data":"32887e49b1db12e505b63ee10bc42a6305fe40c25b8050fcca1f35cd7d28d1c9"}
Oct 31 12:48:30.220902 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:30.220881    5060 scope.go:122] "RemoveContainer" containerID="07912adfcfe0dac99f69adef566eb21ca4e3b211b8b78efefb0506f1e7a058db"
Oct 31 12:48:30.235399 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:30.235373    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-sd68f"]
Oct 31 12:48:30.237245 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:30.237222    5060 scope.go:122] "RemoveContainer" containerID="07912adfcfe0dac99f69adef566eb21ca4e3b211b8b78efefb0506f1e7a058db"
Oct 31 12:48:30.237548 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:48:30.237525    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07912adfcfe0dac99f69adef566eb21ca4e3b211b8b78efefb0506f1e7a058db\": container with ID starting with 07912adfcfe0dac99f69adef566eb21ca4e3b211b8b78efefb0506f1e7a058db not found: ID does not exist" containerID="07912adfcfe0dac99f69adef566eb21ca4e3b211b8b78efefb0506f1e7a058db"
Oct 31 12:48:30.237616 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:30.237558    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07912adfcfe0dac99f69adef566eb21ca4e3b211b8b78efefb0506f1e7a058db"} err="failed to get container status \"07912adfcfe0dac99f69adef566eb21ca4e3b211b8b78efefb0506f1e7a058db\": rpc error: code = NotFound desc = could not find container \"07912adfcfe0dac99f69adef566eb21ca4e3b211b8b78efefb0506f1e7a058db\": container with ID starting with 07912adfcfe0dac99f69adef566eb21ca4e3b211b8b78efefb0506f1e7a058db not found: ID does not exist"
Oct 31 12:48:30.244637 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:30.244614    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-sd68f"]
Oct 31 12:48:30.252820 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:30.252792    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-bbjl7"]
Oct 31 12:48:30.290997 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:30.290903    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-bbjl7"]
Oct 31 12:48:30.329814 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:48:30.329775    5060 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4129ab4f_7756_4a6d_8d7f_9dff5d0f1c57.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4129ab4f_7756_4a6d_8d7f_9dff5d0f1c57.slice/crio-977487d36393140b45690701ef80d87813f658a59a4c69355a10d0a42803f1c3\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09b9b34e_eb53_4829_aed9_abb0b81675d0.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09b9b34e_eb53_4829_aed9_abb0b81675d0.slice/crio-32887e49b1db12e505b63ee10bc42a6305fe40c25b8050fcca1f35cd7d28d1c9\": RecentStats: unable to find data in memory cache]"
Oct 31 12:48:30.390539 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:30.390478    5060 patch_prober.go:29] interesting pod/e2e-api-b-711af2-openshift-default-6549b5d6f9-zdqxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body=
Oct 31 12:48:30.390742 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:30.390556    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-711af2-openshift-default-6549b5d6f9-zdqxc" podUID="440e8c3e-006e-4a6b-82d1-5fb80f5e26d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused"
Oct 31 12:48:31.128516 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:31.107253    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09b9b34e-eb53-4829-aed9-abb0b81675d0" path="/var/lib/kubelet/pods/09b9b34e-eb53-4829-aed9-abb0b81675d0/volumes"
Oct 31 12:48:31.134199 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:31.133421    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4129ab4f-7756-4a6d-8d7f-9dff5d0f1c57" path="/var/lib/kubelet/pods/4129ab4f-7756-4a6d-8d7f-9dff5d0f1c57/volumes"
Oct 31 12:48:31.392295 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:31.392198    5060 patch_prober.go:29] interesting pod/e2e-api-b-711af2-openshift-default-6549b5d6f9-zdqxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body=
Oct 31 12:48:31.392295 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:31.392272    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-711af2-openshift-default-6549b5d6f9-zdqxc" podUID="440e8c3e-006e-4a6b-82d1-5fb80f5e26d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused"
Oct 31 12:48:31.505909 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:31.505866    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-711af2-openshift-default-6549b5d6f9-zdqxc"]
Oct 31 12:48:31.506107 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:31.506077    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-711af2-openshift-default-6549b5d6f9-zdqxc" podUID="440e8c3e-006e-4a6b-82d1-5fb80f5e26d1" containerName="istio-proxy" containerID="cri-o://786c9569efb95b9c63b956c308e0ae69c7468bd9c0e9bddb3bf4e16532c1578b" gracePeriod=30
Oct 31 12:48:32.807803 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:32.797663    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-vxzjg"]
Oct 31 12:48:32.807803 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:32.798927    5060 cpu_manager.go:414] "containerMap: removing container" podUID="e39c020a-e0e5-4a0c-9c52-3d63241cc13c" containerName="revoke-keys"
Oct 31 12:48:32.807803 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:32.798952    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e39c020a-e0e5-4a0c-9c52-3d63241cc13c" containerName="revoke-keys"
Oct 31 12:48:32.807803 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:32.799012    5060 cpu_manager.go:414] "containerMap: removing container" podUID="09b9b34e-eb53-4829-aed9-abb0b81675d0" containerName="istio-proxy"
Oct 31 12:48:32.807803 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:32.799020    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09b9b34e-eb53-4829-aed9-abb0b81675d0" containerName="istio-proxy"
Oct 31 12:48:32.807803 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:32.799038    5060 cpu_manager.go:414] "containerMap: removing container" podUID="4129ab4f-7756-4a6d-8d7f-9dff5d0f1c57" containerName="istio-proxy"
Oct 31 12:48:32.807803 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:32.799047    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4129ab4f-7756-4a6d-8d7f-9dff5d0f1c57" containerName="istio-proxy"
Oct 31 12:48:32.807803 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:32.799224    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="09b9b34e-eb53-4829-aed9-abb0b81675d0" containerName="istio-proxy"
Oct 31 12:48:32.807803 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:32.799235    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="4129ab4f-7756-4a6d-8d7f-9dff5d0f1c57" containerName="istio-proxy"
Oct 31 12:48:32.807803 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:32.799251    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="e39c020a-e0e5-4a0c-9c52-3d63241cc13c" containerName="revoke-keys"
Oct 31 12:48:32.877514 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:32.873946    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-vxzjg"
Oct 31 12:48:32.906066 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:32.905796    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-4e8eb9-openshift-default-dockercfg-sczrg\""
Oct 31 12:48:33.032619 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:33.008812    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20fa8b33-c9e6-4fb0-9f3c-c8254dece2f6-workload-socket\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-vxzjg\" (UID: \"20fa8b33-c9e6-4fb0-9f3c-c8254dece2f6\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-vxzjg"
Oct 31 12:48:33.036788 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:33.033156    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20fa8b33-c9e6-4fb0-9f3c-c8254dece2f6-istio-envoy\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-vxzjg\" (UID: \"20fa8b33-c9e6-4fb0-9f3c-c8254dece2f6\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-vxzjg"
Oct 31 12:48:33.039943 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:33.037131    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20fa8b33-c9e6-4fb0-9f3c-c8254dece2f6-workload-certs\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-vxzjg\" (UID: \"20fa8b33-c9e6-4fb0-9f3c-c8254dece2f6\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-vxzjg"
Oct 31 12:48:33.088801 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:33.088699    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20fa8b33-c9e6-4fb0-9f3c-c8254dece2f6-credential-socket\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-vxzjg\" (UID: \"20fa8b33-c9e6-4fb0-9f3c-c8254dece2f6\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-vxzjg"
Oct 31 12:48:33.089073 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:33.089047    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtllc\" (UniqueName: \"kubernetes.io/projected/20fa8b33-c9e6-4fb0-9f3c-c8254dece2f6-kube-api-access-xtllc\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-vxzjg\" (UID: \"20fa8b33-c9e6-4fb0-9f3c-c8254dece2f6\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-vxzjg"
Oct 31 12:48:33.089246 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:33.089227    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20fa8b33-c9e6-4fb0-9f3c-c8254dece2f6-istio-data\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-vxzjg\" (UID: \"20fa8b33-c9e6-4fb0-9f3c-c8254dece2f6\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-vxzjg"
Oct 31 12:48:33.089359 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:33.089343    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20fa8b33-c9e6-4fb0-9f3c-c8254dece2f6-istio-podinfo\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-vxzjg\" (UID: \"20fa8b33-c9e6-4fb0-9f3c-c8254dece2f6\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-vxzjg"
Oct 31 12:48:33.089538 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:33.089519    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20fa8b33-c9e6-4fb0-9f3c-c8254dece2f6-istiod-ca-cert\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-vxzjg\" (UID: \"20fa8b33-c9e6-4fb0-9f3c-c8254dece2f6\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-vxzjg"
Oct 31 12:48:33.089846 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:33.089828    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20fa8b33-c9e6-4fb0-9f3c-c8254dece2f6-istio-token\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-vxzjg\" (UID: \"20fa8b33-c9e6-4fb0-9f3c-c8254dece2f6\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-vxzjg"
Oct 31 12:48:33.247109 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:33.240881    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20fa8b33-c9e6-4fb0-9f3c-c8254dece2f6-istio-token\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-vxzjg\" (UID: \"20fa8b33-c9e6-4fb0-9f3c-c8254dece2f6\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-vxzjg"
Oct 31 12:48:33.322440 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:33.317717    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20fa8b33-c9e6-4fb0-9f3c-c8254dece2f6-workload-socket\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-vxzjg\" (UID: \"20fa8b33-c9e6-4fb0-9f3c-c8254dece2f6\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-vxzjg"
Oct 31 12:48:33.322440 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:33.317842    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20fa8b33-c9e6-4fb0-9f3c-c8254dece2f6-istio-envoy\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-vxzjg\" (UID: \"20fa8b33-c9e6-4fb0-9f3c-c8254dece2f6\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-vxzjg"
Oct 31 12:48:33.322440 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:33.317910    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20fa8b33-c9e6-4fb0-9f3c-c8254dece2f6-workload-certs\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-vxzjg\" (UID: \"20fa8b33-c9e6-4fb0-9f3c-c8254dece2f6\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-vxzjg"
Oct 31 12:48:33.322440 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:33.317950    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20fa8b33-c9e6-4fb0-9f3c-c8254dece2f6-credential-socket\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-vxzjg\" (UID: \"20fa8b33-c9e6-4fb0-9f3c-c8254dece2f6\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-vxzjg"
Oct 31 12:48:33.322440 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:33.317980    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtllc\" (UniqueName: \"kubernetes.io/projected/20fa8b33-c9e6-4fb0-9f3c-c8254dece2f6-kube-api-access-xtllc\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-vxzjg\" (UID: \"20fa8b33-c9e6-4fb0-9f3c-c8254dece2f6\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-vxzjg"
Oct 31 12:48:33.322440 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:33.318034    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20fa8b33-c9e6-4fb0-9f3c-c8254dece2f6-istio-data\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-vxzjg\" (UID: \"20fa8b33-c9e6-4fb0-9f3c-c8254dece2f6\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-vxzjg"
Oct 31 12:48:33.322440 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:33.318058    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20fa8b33-c9e6-4fb0-9f3c-c8254dece2f6-istio-podinfo\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-vxzjg\" (UID: \"20fa8b33-c9e6-4fb0-9f3c-c8254dece2f6\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-vxzjg"
Oct 31 12:48:33.322440 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:33.318102    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20fa8b33-c9e6-4fb0-9f3c-c8254dece2f6-istiod-ca-cert\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-vxzjg\" (UID: \"20fa8b33-c9e6-4fb0-9f3c-c8254dece2f6\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-vxzjg"
Oct 31 12:48:33.322440 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:33.319576    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20fa8b33-c9e6-4fb0-9f3c-c8254dece2f6-istiod-ca-cert\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-vxzjg\" (UID: \"20fa8b33-c9e6-4fb0-9f3c-c8254dece2f6\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-vxzjg"
Oct 31 12:48:33.322440 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:33.279385    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-vxzjg"]
Oct 31 12:48:33.322440 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:33.320221    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20fa8b33-c9e6-4fb0-9f3c-c8254dece2f6-workload-socket\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-vxzjg\" (UID: \"20fa8b33-c9e6-4fb0-9f3c-c8254dece2f6\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-vxzjg"
Oct 31 12:48:33.323781 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:33.323346    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20fa8b33-c9e6-4fb0-9f3c-c8254dece2f6-workload-certs\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-vxzjg\" (UID: \"20fa8b33-c9e6-4fb0-9f3c-c8254dece2f6\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-vxzjg"
Oct 31 12:48:33.323781 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:33.323519    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20fa8b33-c9e6-4fb0-9f3c-c8254dece2f6-credential-socket\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-vxzjg\" (UID: \"20fa8b33-c9e6-4fb0-9f3c-c8254dece2f6\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-vxzjg"
Oct 31 12:48:33.323781 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:33.323673    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20fa8b33-c9e6-4fb0-9f3c-c8254dece2f6-istio-data\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-vxzjg\" (UID: \"20fa8b33-c9e6-4fb0-9f3c-c8254dece2f6\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-vxzjg"
Oct 31 12:48:33.329030 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:33.328008    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20fa8b33-c9e6-4fb0-9f3c-c8254dece2f6-istio-podinfo\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-vxzjg\" (UID: \"20fa8b33-c9e6-4fb0-9f3c-c8254dece2f6\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-vxzjg"
Oct 31 12:48:33.336408 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:33.336379    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20fa8b33-c9e6-4fb0-9f3c-c8254dece2f6-istio-envoy\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-vxzjg\" (UID: \"20fa8b33-c9e6-4fb0-9f3c-c8254dece2f6\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-vxzjg"
Oct 31 12:48:33.343172 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:33.343060    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20fa8b33-c9e6-4fb0-9f3c-c8254dece2f6-istio-token\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-vxzjg\" (UID: \"20fa8b33-c9e6-4fb0-9f3c-c8254dece2f6\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-vxzjg"
Oct 31 12:48:33.344705 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:33.344675    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtllc\" (UniqueName: \"kubernetes.io/projected/20fa8b33-c9e6-4fb0-9f3c-c8254dece2f6-kube-api-access-xtllc\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-vxzjg\" (UID: \"20fa8b33-c9e6-4fb0-9f3c-c8254dece2f6\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-vxzjg"
Oct 31 12:48:33.395678 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:33.395645    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-vxzjg"
Oct 31 12:48:33.743113 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:33.743027    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-vxzjg"]
Oct 31 12:48:33.749844 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:48:33.749809    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20fa8b33_c9e6_4fb0_9f3c_c8254dece2f6.slice/crio-7eee1d352b0d46eb4f360488182eb6f14acce1becd45890a1dea9a0b5e31c334 WatchSource:0}: Error finding container 7eee1d352b0d46eb4f360488182eb6f14acce1becd45890a1dea9a0b5e31c334: Status 404 returned error can't find the container with id 7eee1d352b0d46eb4f360488182eb6f14acce1becd45890a1dea9a0b5e31c334
Oct 31 12:48:33.754059 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:33.754017    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:48:33.754201 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:33.754109    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:48:33.754201 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:33.754160    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:48:33.754802 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:33.754728    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bb3cf5-26qt6"]
Oct 31 12:48:33.763981 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:33.763942    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bb3cf5-26qt6"]
Oct 31 12:48:34.375197 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:34.375146    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-vxzjg" event={"ID":"20fa8b33-c9e6-4fb0-9f3c-c8254dece2f6","Type":"ContainerStarted","Data":"cec778106ab298c2c59e82c8abfe4bb666ce3948bf26f0b090cf4e8426f6b03a"}
Oct 31 12:48:34.375915 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:34.375791    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-vxzjg" event={"ID":"20fa8b33-c9e6-4fb0-9f3c-c8254dece2f6","Type":"ContainerStarted","Data":"7eee1d352b0d46eb4f360488182eb6f14acce1becd45890a1dea9a0b5e31c334"}
Oct 31 12:48:34.414519 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:34.401237    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-vxzjg"
Oct 31 12:48:34.415690 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:34.415506    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-vxzjg" podStartSLOduration=2.415468631 podStartE2EDuration="2.415468631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:48:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:48:34.399920913 +0000 UTC m=+1277.502988650" watchObservedRunningTime="2026-10-31 12:48:34.415468631 +0000 UTC m=+1277.518536345"
Oct 31 12:48:35.060762 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:35.060722    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de748339-be67-487e-ba7b-eeaa3ccbc356" path="/var/lib/kubelet/pods/de748339-be67-487e-ba7b-eeaa3ccbc356/volumes"
Oct 31 12:48:35.645044 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:35.644976    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-vxzjg"
Oct 31 12:48:36.413182 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:36.413137    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-711af2-openshift-default-6549b5d6f9-zdqxc"
Oct 31 12:48:36.547520 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:36.547447    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e1fc0ffe-mfq7z"]
Oct 31 12:48:36.571521 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:36.566446    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-vxzjg"
Oct 31 12:48:36.571521 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:36.566553    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-vxzjg"
Oct 31 12:48:36.571521 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:36.566924    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e1fc0ffe-mfq7z"
Oct 31 12:48:36.581418 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:36.576943    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 12:48:36.581418 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:36.577024    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7wsmb\""
Oct 31 12:48:36.581418 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:36.577051    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 12:48:36.581418 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:36.577124    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 12:48:36.600518 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:36.583267    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e1fc0ffe-mfq7z"]
Oct 31 12:48:36.642570 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:36.633712    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7110c247-4202-4b54-bc1b-3d4cf7ae05d1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-e1fc0ffe-mfq7z\" (UID: \"7110c247-4202-4b54-bc1b-3d4cf7ae05d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e1fc0ffe-mfq7z"
Oct 31 12:48:36.642570 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:36.633955    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7110c247-4202-4b54-bc1b-3d4cf7ae05d1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-e1fc0ffe-mfq7z\" (UID: \"7110c247-4202-4b54-bc1b-3d4cf7ae05d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e1fc0ffe-mfq7z"
Oct 31 12:48:36.642570 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:36.634049    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mxkd\" (UniqueName: \"kubernetes.io/projected/7110c247-4202-4b54-bc1b-3d4cf7ae05d1-kube-api-access-8mxkd\") pod \"maas-api-revoke-keys-e2e-mt-e1fc0ffe-mfq7z\" (UID: \"7110c247-4202-4b54-bc1b-3d4cf7ae05d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e1fc0ffe-mfq7z"
Oct 31 12:48:36.810992 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:36.802659    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7110c247-4202-4b54-bc1b-3d4cf7ae05d1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-e1fc0ffe-mfq7z\" (UID: \"7110c247-4202-4b54-bc1b-3d4cf7ae05d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e1fc0ffe-mfq7z"
Oct 31 12:48:36.810992 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:36.803752    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7110c247-4202-4b54-bc1b-3d4cf7ae05d1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-e1fc0ffe-mfq7z\" (UID: \"7110c247-4202-4b54-bc1b-3d4cf7ae05d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e1fc0ffe-mfq7z"
Oct 31 12:48:36.810992 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:36.803834    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mxkd\" (UniqueName: \"kubernetes.io/projected/7110c247-4202-4b54-bc1b-3d4cf7ae05d1-kube-api-access-8mxkd\") pod \"maas-api-revoke-keys-e2e-mt-e1fc0ffe-mfq7z\" (UID: \"7110c247-4202-4b54-bc1b-3d4cf7ae05d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e1fc0ffe-mfq7z"
Oct 31 12:48:36.810992 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:36.806229    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7110c247-4202-4b54-bc1b-3d4cf7ae05d1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-e1fc0ffe-mfq7z\" (UID: \"7110c247-4202-4b54-bc1b-3d4cf7ae05d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e1fc0ffe-mfq7z"
Oct 31 12:48:36.810992 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:36.810030    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7110c247-4202-4b54-bc1b-3d4cf7ae05d1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-e1fc0ffe-mfq7z\" (UID: \"7110c247-4202-4b54-bc1b-3d4cf7ae05d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e1fc0ffe-mfq7z"
Oct 31 12:48:36.897513 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:36.896710    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mxkd\" (UniqueName: \"kubernetes.io/projected/7110c247-4202-4b54-bc1b-3d4cf7ae05d1-kube-api-access-8mxkd\") pod \"maas-api-revoke-keys-e2e-mt-e1fc0ffe-mfq7z\" (UID: \"7110c247-4202-4b54-bc1b-3d4cf7ae05d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e1fc0ffe-mfq7z"
Oct 31 12:48:37.194075 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:37.192800    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e1fc0ffe-mfq7z"
Oct 31 12:48:38.438801 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:38.433205    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e1fc0ffe-mfq7z"]
Oct 31 12:48:38.646005 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:38.645965    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e1fc0ffe-mfq7z" event={"ID":"7110c247-4202-4b54-bc1b-3d4cf7ae05d1","Type":"ContainerStarted","Data":"f0e74808b375171edc712c98b87bcd474caa2cd49cbdf9ed7cbb628d16b478ba"}
Oct 31 12:48:38.649430 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:38.649404    5060 generic.go:356] "Generic (PLEG): container finished" podID="440e8c3e-006e-4a6b-82d1-5fb80f5e26d1" containerID="786c9569efb95b9c63b956c308e0ae69c7468bd9c0e9bddb3bf4e16532c1578b" exitCode=0
Oct 31 12:48:38.650615 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:38.650595    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-711af2-openshift-default-6549b5d6f9-zdqxc" event={"ID":"440e8c3e-006e-4a6b-82d1-5fb80f5e26d1","Type":"ContainerDied","Data":"786c9569efb95b9c63b956c308e0ae69c7468bd9c0e9bddb3bf4e16532c1578b"}
Oct 31 12:48:38.650782 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:38.650769    5060 scope.go:122] "RemoveContainer" containerID="47e7f181c25f923c5479a64b7786d08ed6b5f4e7a0e117ea035685d1dd2d25eb"
Oct 31 12:48:39.195640 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:39.188647    5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-645cf464d7-v2sv5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 12:48:39.195640 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:39.188705    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-v2sv5" podUID="1e33b790-15e5-4754-929f-c0a7330b9361" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:48:39.237001 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:39.226569    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/simulator-w4-c67de2-kserve-868679cf9b-gqjbx" podUID="e104d7da-1fb2-4c99-a480-b4ae84c4f273" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded"
Oct 31 12:48:39.374698 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:39.374600    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/premium-w2-e6c79d-kserve-6468bf46d4-h66f8" podUID="cd6a266f-8d12-4089-83ee-c75c716bf26c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:48:39.615308 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:39.583212    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x2qzv" podUID="c247ed2a-aa90-424b-9b7e-4f2936d67e3f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded"
Oct 31 12:48:39.615308 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:39.599657    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-2-w5-b25d3b-kserve-76f798d59f-vvzhg" podUID="394ed76c-84ac-4612-9acb-a7e36cb68d89" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": context deadline exceeded"
Oct 31 12:48:40.396202 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:40.396084    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/simulator-w4-c67de2-kserve-868679cf9b-gqjbx" podUID="e104d7da-1fb2-4c99-a480-b4ae84c4f273" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded"
Oct 31 12:48:40.416624 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:40.397170    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-711af2-openshift-default-6549b5d6f9-zdqxc"
Oct 31 12:48:40.416954 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:40.403745    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c67de2/simulator-w4-c67de2-kserve-868679cf9b-gqjbx"
Oct 31 12:48:40.429268 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:40.425723    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/premium-w2-e6c79d-kserve-6468bf46d4-h66f8" podUID="cd6a266f-8d12-4089-83ee-c75c716bf26c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:48:40.430081 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:40.430059    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e6c79d/premium-w2-e6c79d-kserve-6468bf46d4-h66f8"
Oct 31 12:48:40.467512 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:40.465198    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e6c79d/premium-w2-e6c79d-kserve-6468bf46d4-h66f8"
Oct 31 12:48:40.467512 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:40.465279    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c67de2/simulator-w4-c67de2-kserve-868679cf9b-gqjbx"
Oct 31 12:48:40.565514 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:40.547530    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/440e8c3e-006e-4a6b-82d1-5fb80f5e26d1-istio-data" pod "440e8c3e-006e-4a6b-82d1-5fb80f5e26d1" (UID: "440e8c3e-006e-4a6b-82d1-5fb80f5e26d1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:48:40.565514 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:40.548455    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/440e8c3e-006e-4a6b-82d1-5fb80f5e26d1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/440e8c3e-006e-4a6b-82d1-5fb80f5e26d1-istio-data\") pod \"440e8c3e-006e-4a6b-82d1-5fb80f5e26d1\" (UID: \"440e8c3e-006e-4a6b-82d1-5fb80f5e26d1\") "
Oct 31 12:48:40.565514 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:40.548618    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/440e8c3e-006e-4a6b-82d1-5fb80f5e26d1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/440e8c3e-006e-4a6b-82d1-5fb80f5e26d1-istio-envoy\") pod \"440e8c3e-006e-4a6b-82d1-5fb80f5e26d1\" (UID: \"440e8c3e-006e-4a6b-82d1-5fb80f5e26d1\") "
Oct 31 12:48:40.565514 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:40.548651    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/440e8c3e-006e-4a6b-82d1-5fb80f5e26d1-kube-api-access-mnpqj\" (UniqueName: \"kubernetes.io/projected/440e8c3e-006e-4a6b-82d1-5fb80f5e26d1-kube-api-access-mnpqj\") pod \"440e8c3e-006e-4a6b-82d1-5fb80f5e26d1\" (UID: \"440e8c3e-006e-4a6b-82d1-5fb80f5e26d1\") "
Oct 31 12:48:40.565514 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:40.548764    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/440e8c3e-006e-4a6b-82d1-5fb80f5e26d1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/440e8c3e-006e-4a6b-82d1-5fb80f5e26d1-istiod-ca-cert\") pod \"440e8c3e-006e-4a6b-82d1-5fb80f5e26d1\" (UID: \"440e8c3e-006e-4a6b-82d1-5fb80f5e26d1\") "
Oct 31 12:48:40.565514 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:40.548824    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/440e8c3e-006e-4a6b-82d1-5fb80f5e26d1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/440e8c3e-006e-4a6b-82d1-5fb80f5e26d1-credential-socket\") pod \"440e8c3e-006e-4a6b-82d1-5fb80f5e26d1\" (UID: \"440e8c3e-006e-4a6b-82d1-5fb80f5e26d1\") "
Oct 31 12:48:40.565514 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:40.548951    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/440e8c3e-006e-4a6b-82d1-5fb80f5e26d1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/440e8c3e-006e-4a6b-82d1-5fb80f5e26d1-workload-socket\") pod \"440e8c3e-006e-4a6b-82d1-5fb80f5e26d1\" (UID: \"440e8c3e-006e-4a6b-82d1-5fb80f5e26d1\") "
Oct 31 12:48:40.565514 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:40.549013    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/440e8c3e-006e-4a6b-82d1-5fb80f5e26d1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/440e8c3e-006e-4a6b-82d1-5fb80f5e26d1-istio-podinfo\") pod \"440e8c3e-006e-4a6b-82d1-5fb80f5e26d1\" (UID: \"440e8c3e-006e-4a6b-82d1-5fb80f5e26d1\") "
Oct 31 12:48:40.565514 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:40.549154    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/440e8c3e-006e-4a6b-82d1-5fb80f5e26d1-istio-token\" (UniqueName: \"kubernetes.io/projected/440e8c3e-006e-4a6b-82d1-5fb80f5e26d1-istio-token\") pod \"440e8c3e-006e-4a6b-82d1-5fb80f5e26d1\" (UID: \"440e8c3e-006e-4a6b-82d1-5fb80f5e26d1\") "
Oct 31 12:48:40.565514 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:40.549206    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/440e8c3e-006e-4a6b-82d1-5fb80f5e26d1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/440e8c3e-006e-4a6b-82d1-5fb80f5e26d1-workload-certs\") pod \"440e8c3e-006e-4a6b-82d1-5fb80f5e26d1\" (UID: \"440e8c3e-006e-4a6b-82d1-5fb80f5e26d1\") "
Oct 31 12:48:40.565514 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:40.551344    5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/440e8c3e-006e-4a6b-82d1-5fb80f5e26d1-istio-data\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:48:40.616704 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:40.616375    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/440e8c3e-006e-4a6b-82d1-5fb80f5e26d1-istiod-ca-cert" pod "440e8c3e-006e-4a6b-82d1-5fb80f5e26d1" (UID: "440e8c3e-006e-4a6b-82d1-5fb80f5e26d1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:48:40.663544 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:40.655611    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/440e8c3e-006e-4a6b-82d1-5fb80f5e26d1-credential-socket" pod "440e8c3e-006e-4a6b-82d1-5fb80f5e26d1" (UID: "440e8c3e-006e-4a6b-82d1-5fb80f5e26d1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:48:40.663544 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:40.660447    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/440e8c3e-006e-4a6b-82d1-5fb80f5e26d1-workload-certs" pod "440e8c3e-006e-4a6b-82d1-5fb80f5e26d1" (UID: "440e8c3e-006e-4a6b-82d1-5fb80f5e26d1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:48:40.678512 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:40.671223    5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/440e8c3e-006e-4a6b-82d1-5fb80f5e26d1-istiod-ca-cert\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:48:40.678512 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:40.671260    5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/440e8c3e-006e-4a6b-82d1-5fb80f5e26d1-credential-socket\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:48:40.678512 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:40.671277    5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/440e8c3e-006e-4a6b-82d1-5fb80f5e26d1-workload-certs\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:48:40.697691 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:40.697602    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/440e8c3e-006e-4a6b-82d1-5fb80f5e26d1-workload-socket" pod "440e8c3e-006e-4a6b-82d1-5fb80f5e26d1" (UID: "440e8c3e-006e-4a6b-82d1-5fb80f5e26d1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:48:40.754337 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:40.754280    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/440e8c3e-006e-4a6b-82d1-5fb80f5e26d1-istio-token" pod "440e8c3e-006e-4a6b-82d1-5fb80f5e26d1" (UID: "440e8c3e-006e-4a6b-82d1-5fb80f5e26d1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:48:40.803888 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:40.802768    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/440e8c3e-006e-4a6b-82d1-5fb80f5e26d1-kube-api-access-mnpqj" pod "440e8c3e-006e-4a6b-82d1-5fb80f5e26d1" (UID: "440e8c3e-006e-4a6b-82d1-5fb80f5e26d1"). InnerVolumeSpecName "kube-api-access-mnpqj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:48:40.832169 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:40.832129    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/440e8c3e-006e-4a6b-82d1-5fb80f5e26d1-istio-envoy" pod "440e8c3e-006e-4a6b-82d1-5fb80f5e26d1" (UID: "440e8c3e-006e-4a6b-82d1-5fb80f5e26d1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:48:40.842736 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:40.842699    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/440e8c3e-006e-4a6b-82d1-5fb80f5e26d1-istio-podinfo" pod "440e8c3e-006e-4a6b-82d1-5fb80f5e26d1" (UID: "440e8c3e-006e-4a6b-82d1-5fb80f5e26d1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 12:48:40.909759 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:40.854880    5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/440e8c3e-006e-4a6b-82d1-5fb80f5e26d1-istio-envoy\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:48:40.989599 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:40.928252    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mnpqj\" (UniqueName: \"kubernetes.io/projected/440e8c3e-006e-4a6b-82d1-5fb80f5e26d1-kube-api-access-mnpqj\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:48:40.989599 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:40.928292    5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/440e8c3e-006e-4a6b-82d1-5fb80f5e26d1-workload-socket\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:48:40.989599 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:40.928312    5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/440e8c3e-006e-4a6b-82d1-5fb80f5e26d1-istio-podinfo\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:48:40.989599 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:40.928334    5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/440e8c3e-006e-4a6b-82d1-5fb80f5e26d1-istio-token\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:48:41.376517 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:41.366523    5060 generic.go:356] "Generic (PLEG): container finished" podID="7110c247-4202-4b54-bc1b-3d4cf7ae05d1" containerID="2fa664b61741f11c1b98eb0639d5b30fd6365667ab609cc869b5d321b65c5e71" exitCode=0
Oct 31 12:48:41.376517 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:41.370717    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e1fc0ffe-mfq7z" event={"ID":"7110c247-4202-4b54-bc1b-3d4cf7ae05d1","Type":"ContainerDied","Data":"2fa664b61741f11c1b98eb0639d5b30fd6365667ab609cc869b5d321b65c5e71"}
Oct 31 12:48:41.441442 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:41.441338    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-711af2-openshift-default-6549b5d6f9-zdqxc"
Oct 31 12:48:41.442048 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:41.441704    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-711af2-openshift-default-6549b5d6f9-zdqxc" event={"ID":"440e8c3e-006e-4a6b-82d1-5fb80f5e26d1","Type":"ContainerDied","Data":"4615eb5c1733c4e9bbf25ec01476fd43a6ecbc7d66a3c176acb4019cd39e7417"}
Oct 31 12:48:41.442048 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:41.441765    5060 scope.go:122] "RemoveContainer" containerID="786c9569efb95b9c63b956c308e0ae69c7468bd9c0e9bddb3bf4e16532c1578b"
Oct 31 12:48:41.473523 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:48:41.472940    5060 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7110c247_4202_4b54_bc1b_3d4cf7ae05d1.slice/crio-2fa664b61741f11c1b98eb0639d5b30fd6365667ab609cc869b5d321b65c5e71.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod440e8c3e_006e_4a6b_82d1_5fb80f5e26d1.slice/crio-4615eb5c1733c4e9bbf25ec01476fd43a6ecbc7d66a3c176acb4019cd39e7417\": RecentStats: unable to find data in memory cache]"
Oct 31 12:48:41.542580 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:41.542544    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-711af2-openshift-default-6549b5d6f9-zdqxc"]
Oct 31 12:48:41.591517 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:41.588578    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-711af2-openshift-default-6549b5d6f9-zdqxc"]
Oct 31 12:48:42.752028 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:42.751968    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e1fc0ffe-mfq7z"
Oct 31 12:48:42.820183 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:42.820154    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7110c247-4202-4b54-bc1b-3d4cf7ae05d1-kube-api-access-8mxkd\" (UniqueName: \"kubernetes.io/projected/7110c247-4202-4b54-bc1b-3d4cf7ae05d1-kube-api-access-8mxkd\") pod \"7110c247-4202-4b54-bc1b-3d4cf7ae05d1\" (UID: \"7110c247-4202-4b54-bc1b-3d4cf7ae05d1\") "
Oct 31 12:48:42.820356 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:42.820341    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7110c247-4202-4b54-bc1b-3d4cf7ae05d1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7110c247-4202-4b54-bc1b-3d4cf7ae05d1-maas-api-xks-ca\") pod \"7110c247-4202-4b54-bc1b-3d4cf7ae05d1\" (UID: \"7110c247-4202-4b54-bc1b-3d4cf7ae05d1\") "
Oct 31 12:48:42.820467 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:42.820456    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7110c247-4202-4b54-bc1b-3d4cf7ae05d1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7110c247-4202-4b54-bc1b-3d4cf7ae05d1-maas-api-openshift-ca\") pod \"7110c247-4202-4b54-bc1b-3d4cf7ae05d1\" (UID: \"7110c247-4202-4b54-bc1b-3d4cf7ae05d1\") "
Oct 31 12:48:42.821342 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:42.821310    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7110c247-4202-4b54-bc1b-3d4cf7ae05d1-maas-api-openshift-ca" pod "7110c247-4202-4b54-bc1b-3d4cf7ae05d1" (UID: "7110c247-4202-4b54-bc1b-3d4cf7ae05d1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:48:42.823399 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:42.823377    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7110c247-4202-4b54-bc1b-3d4cf7ae05d1-kube-api-access-8mxkd" pod "7110c247-4202-4b54-bc1b-3d4cf7ae05d1" (UID: "7110c247-4202-4b54-bc1b-3d4cf7ae05d1"). InnerVolumeSpecName "kube-api-access-8mxkd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:48:42.825902 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:42.825878    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7110c247-4202-4b54-bc1b-3d4cf7ae05d1-maas-api-xks-ca" pod "7110c247-4202-4b54-bc1b-3d4cf7ae05d1" (UID: "7110c247-4202-4b54-bc1b-3d4cf7ae05d1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:48:42.923660 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:42.923611    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8mxkd\" (UniqueName: \"kubernetes.io/projected/7110c247-4202-4b54-bc1b-3d4cf7ae05d1-kube-api-access-8mxkd\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:48:42.923660 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:42.923646    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7110c247-4202-4b54-bc1b-3d4cf7ae05d1-maas-api-xks-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:48:42.923660 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:42.923662    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7110c247-4202-4b54-bc1b-3d4cf7ae05d1-maas-api-openshift-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:48:43.098256 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:43.098216    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="440e8c3e-006e-4a6b-82d1-5fb80f5e26d1" path="/var/lib/kubelet/pods/440e8c3e-006e-4a6b-82d1-5fb80f5e26d1/volumes"
Oct 31 12:48:43.502239 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:43.502151    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e1fc0ffe-mfq7z"
Oct 31 12:48:43.502239 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:43.502177    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e1fc0ffe-mfq7z" event={"ID":"7110c247-4202-4b54-bc1b-3d4cf7ae05d1","Type":"ContainerDied","Data":"f0e74808b375171edc712c98b87bcd474caa2cd49cbdf9ed7cbb628d16b478ba"}
Oct 31 12:48:43.502239 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:43.502236    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f0e74808b375171edc712c98b87bcd474caa2cd49cbdf9ed7cbb628d16b478ba"
Oct 31 12:48:44.810511 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:44.810208    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-bb3cf5-openshift-default-5fc978478f-mw7qm"]
Oct 31 12:48:44.811022 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:44.810525    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-bb3cf5-openshift-default-5fc978478f-mw7qm" podUID="d9cb64fe-d6b4-439d-bc3a-8598b7cd512e" containerName="istio-proxy" containerID="cri-o://d4a8f0047375c3091a7328bd485e99dc745049f6dbe9a643546d3273f279c468" gracePeriod=30
Oct 31 12:48:45.336134 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:45.334941    5060 patch_prober.go:29] interesting pod/e2e-api-b-787316-openshift-default-58b9b668c5-qpf4t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.194:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 12:48:45.336134 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:45.335070    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-787316-openshift-default-58b9b668c5-qpf4t" podUID="b4e57789-6b1a-4d17-bdaf-925a3d3d0c43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.194:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:48:46.903355 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:46.903266    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-e98df8-openshift-default-779d98479-r2jbq"]
Oct 31 12:48:46.904656 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:46.904628    5060 cpu_manager.go:414] "containerMap: removing container" podUID="440e8c3e-006e-4a6b-82d1-5fb80f5e26d1" containerName="istio-proxy"
Oct 31 12:48:46.904784 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:46.904657    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="440e8c3e-006e-4a6b-82d1-5fb80f5e26d1" containerName="istio-proxy"
Oct 31 12:48:46.904784 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:46.904679    5060 cpu_manager.go:414] "containerMap: removing container" podUID="7110c247-4202-4b54-bc1b-3d4cf7ae05d1" containerName="revoke-keys"
Oct 31 12:48:46.904784 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:46.904686    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7110c247-4202-4b54-bc1b-3d4cf7ae05d1" containerName="revoke-keys"
Oct 31 12:48:46.904946 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:46.904915    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="440e8c3e-006e-4a6b-82d1-5fb80f5e26d1" containerName="istio-proxy"
Oct 31 12:48:46.904946 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:46.904927    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="7110c247-4202-4b54-bc1b-3d4cf7ae05d1" containerName="revoke-keys"
Oct 31 12:48:46.915741 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:46.912512    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e98df8-openshift-default-779d98479-r2jbq"
Oct 31 12:48:46.923580 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:46.923551    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-e98df8-openshift-default-779d98479-r2jbq"]
Oct 31 12:48:46.932771 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:46.926813    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-e98df8-openshift-default-dockercfg-z8hm9\""
Oct 31 12:48:47.014867 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:47.014181    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e08b08ff-4f40-474a-a103-2c499e999942-workload-certs\") pod \"e2e-inf-a-e98df8-openshift-default-779d98479-r2jbq\" (UID: \"e08b08ff-4f40-474a-a103-2c499e999942\") " pod="openshift-ingress/e2e-inf-a-e98df8-openshift-default-779d98479-r2jbq"
Oct 31 12:48:47.014867 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:47.014236    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e08b08ff-4f40-474a-a103-2c499e999942-istio-envoy\") pod \"e2e-inf-a-e98df8-openshift-default-779d98479-r2jbq\" (UID: \"e08b08ff-4f40-474a-a103-2c499e999942\") " pod="openshift-ingress/e2e-inf-a-e98df8-openshift-default-779d98479-r2jbq"
Oct 31 12:48:47.014867 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:47.014268    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5m885\" (UniqueName: \"kubernetes.io/projected/e08b08ff-4f40-474a-a103-2c499e999942-kube-api-access-5m885\") pod \"e2e-inf-a-e98df8-openshift-default-779d98479-r2jbq\" (UID: \"e08b08ff-4f40-474a-a103-2c499e999942\") " pod="openshift-ingress/e2e-inf-a-e98df8-openshift-default-779d98479-r2jbq"
Oct 31 12:48:47.014867 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:47.014331    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e08b08ff-4f40-474a-a103-2c499e999942-credential-socket\") pod \"e2e-inf-a-e98df8-openshift-default-779d98479-r2jbq\" (UID: \"e08b08ff-4f40-474a-a103-2c499e999942\") " pod="openshift-ingress/e2e-inf-a-e98df8-openshift-default-779d98479-r2jbq"
Oct 31 12:48:47.014867 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:47.014387    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e08b08ff-4f40-474a-a103-2c499e999942-istio-token\") pod \"e2e-inf-a-e98df8-openshift-default-779d98479-r2jbq\" (UID: \"e08b08ff-4f40-474a-a103-2c499e999942\") " pod="openshift-ingress/e2e-inf-a-e98df8-openshift-default-779d98479-r2jbq"
Oct 31 12:48:47.014867 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:47.014478    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e08b08ff-4f40-474a-a103-2c499e999942-istio-podinfo\") pod \"e2e-inf-a-e98df8-openshift-default-779d98479-r2jbq\" (UID: \"e08b08ff-4f40-474a-a103-2c499e999942\") " pod="openshift-ingress/e2e-inf-a-e98df8-openshift-default-779d98479-r2jbq"
Oct 31 12:48:47.014867 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:47.014531    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e08b08ff-4f40-474a-a103-2c499e999942-istiod-ca-cert\") pod \"e2e-inf-a-e98df8-openshift-default-779d98479-r2jbq\" (UID: \"e08b08ff-4f40-474a-a103-2c499e999942\") " pod="openshift-ingress/e2e-inf-a-e98df8-openshift-default-779d98479-r2jbq"
Oct 31 12:48:47.014867 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:47.014562    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e08b08ff-4f40-474a-a103-2c499e999942-istio-data\") pod \"e2e-inf-a-e98df8-openshift-default-779d98479-r2jbq\" (UID: \"e08b08ff-4f40-474a-a103-2c499e999942\") " pod="openshift-ingress/e2e-inf-a-e98df8-openshift-default-779d98479-r2jbq"
Oct 31 12:48:47.014867 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:47.014615    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e08b08ff-4f40-474a-a103-2c499e999942-workload-socket\") pod \"e2e-inf-a-e98df8-openshift-default-779d98479-r2jbq\" (UID: \"e08b08ff-4f40-474a-a103-2c499e999942\") " pod="openshift-ingress/e2e-inf-a-e98df8-openshift-default-779d98479-r2jbq"
Oct 31 12:48:47.116839 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:47.116792    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e08b08ff-4f40-474a-a103-2c499e999942-workload-certs\") pod \"e2e-inf-a-e98df8-openshift-default-779d98479-r2jbq\" (UID: \"e08b08ff-4f40-474a-a103-2c499e999942\") " pod="openshift-ingress/e2e-inf-a-e98df8-openshift-default-779d98479-r2jbq"
Oct 31 12:48:47.116839 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:47.116842    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e08b08ff-4f40-474a-a103-2c499e999942-istio-envoy\") pod \"e2e-inf-a-e98df8-openshift-default-779d98479-r2jbq\" (UID: \"e08b08ff-4f40-474a-a103-2c499e999942\") " pod="openshift-ingress/e2e-inf-a-e98df8-openshift-default-779d98479-r2jbq"
Oct 31 12:48:47.117117 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:47.116885    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5m885\" (UniqueName: \"kubernetes.io/projected/e08b08ff-4f40-474a-a103-2c499e999942-kube-api-access-5m885\") pod \"e2e-inf-a-e98df8-openshift-default-779d98479-r2jbq\" (UID: \"e08b08ff-4f40-474a-a103-2c499e999942\") " pod="openshift-ingress/e2e-inf-a-e98df8-openshift-default-779d98479-r2jbq"
Oct 31 12:48:47.117117 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:47.116945    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e08b08ff-4f40-474a-a103-2c499e999942-credential-socket\") pod \"e2e-inf-a-e98df8-openshift-default-779d98479-r2jbq\" (UID: \"e08b08ff-4f40-474a-a103-2c499e999942\") " pod="openshift-ingress/e2e-inf-a-e98df8-openshift-default-779d98479-r2jbq"
Oct 31 12:48:47.117117 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:47.117057    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e08b08ff-4f40-474a-a103-2c499e999942-istio-token\") pod \"e2e-inf-a-e98df8-openshift-default-779d98479-r2jbq\" (UID: \"e08b08ff-4f40-474a-a103-2c499e999942\") " pod="openshift-ingress/e2e-inf-a-e98df8-openshift-default-779d98479-r2jbq"
Oct 31 12:48:47.117296 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:47.117198    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e08b08ff-4f40-474a-a103-2c499e999942-istio-podinfo\") pod \"e2e-inf-a-e98df8-openshift-default-779d98479-r2jbq\" (UID: \"e08b08ff-4f40-474a-a103-2c499e999942\") " pod="openshift-ingress/e2e-inf-a-e98df8-openshift-default-779d98479-r2jbq"
Oct 31 12:48:47.117296 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:47.117249    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e08b08ff-4f40-474a-a103-2c499e999942-istiod-ca-cert\") pod \"e2e-inf-a-e98df8-openshift-default-779d98479-r2jbq\" (UID: \"e08b08ff-4f40-474a-a103-2c499e999942\") " pod="openshift-ingress/e2e-inf-a-e98df8-openshift-default-779d98479-r2jbq"
Oct 31 12:48:47.117296 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:47.117275    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e08b08ff-4f40-474a-a103-2c499e999942-istio-data\") pod \"e2e-inf-a-e98df8-openshift-default-779d98479-r2jbq\" (UID: \"e08b08ff-4f40-474a-a103-2c499e999942\") " pod="openshift-ingress/e2e-inf-a-e98df8-openshift-default-779d98479-r2jbq"
Oct 31 12:48:47.117470 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:47.117311    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e08b08ff-4f40-474a-a103-2c499e999942-credential-socket\") pod \"e2e-inf-a-e98df8-openshift-default-779d98479-r2jbq\" (UID: \"e08b08ff-4f40-474a-a103-2c499e999942\") " pod="openshift-ingress/e2e-inf-a-e98df8-openshift-default-779d98479-r2jbq"
Oct 31 12:48:47.117470 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:47.117332    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e08b08ff-4f40-474a-a103-2c499e999942-workload-socket\") pod \"e2e-inf-a-e98df8-openshift-default-779d98479-r2jbq\" (UID: \"e08b08ff-4f40-474a-a103-2c499e999942\") " pod="openshift-ingress/e2e-inf-a-e98df8-openshift-default-779d98479-r2jbq"
Oct 31 12:48:47.118047 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:47.117719    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e08b08ff-4f40-474a-a103-2c499e999942-istio-data\") pod \"e2e-inf-a-e98df8-openshift-default-779d98479-r2jbq\" (UID: \"e08b08ff-4f40-474a-a103-2c499e999942\") " pod="openshift-ingress/e2e-inf-a-e98df8-openshift-default-779d98479-r2jbq"
Oct 31 12:48:47.118047 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:47.117740    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e08b08ff-4f40-474a-a103-2c499e999942-workload-socket\") pod \"e2e-inf-a-e98df8-openshift-default-779d98479-r2jbq\" (UID: \"e08b08ff-4f40-474a-a103-2c499e999942\") " pod="openshift-ingress/e2e-inf-a-e98df8-openshift-default-779d98479-r2jbq"
Oct 31 12:48:47.118047 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:47.117966    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e08b08ff-4f40-474a-a103-2c499e999942-workload-certs\") pod \"e2e-inf-a-e98df8-openshift-default-779d98479-r2jbq\" (UID: \"e08b08ff-4f40-474a-a103-2c499e999942\") " pod="openshift-ingress/e2e-inf-a-e98df8-openshift-default-779d98479-r2jbq"
Oct 31 12:48:47.118339 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:47.118304    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e08b08ff-4f40-474a-a103-2c499e999942-istiod-ca-cert\") pod \"e2e-inf-a-e98df8-openshift-default-779d98479-r2jbq\" (UID: \"e08b08ff-4f40-474a-a103-2c499e999942\") " pod="openshift-ingress/e2e-inf-a-e98df8-openshift-default-779d98479-r2jbq"
Oct 31 12:48:47.119485 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:47.119465    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e08b08ff-4f40-474a-a103-2c499e999942-istio-envoy\") pod \"e2e-inf-a-e98df8-openshift-default-779d98479-r2jbq\" (UID: \"e08b08ff-4f40-474a-a103-2c499e999942\") " pod="openshift-ingress/e2e-inf-a-e98df8-openshift-default-779d98479-r2jbq"
Oct 31 12:48:47.120142 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:47.120121    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e08b08ff-4f40-474a-a103-2c499e999942-istio-podinfo\") pod \"e2e-inf-a-e98df8-openshift-default-779d98479-r2jbq\" (UID: \"e08b08ff-4f40-474a-a103-2c499e999942\") " pod="openshift-ingress/e2e-inf-a-e98df8-openshift-default-779d98479-r2jbq"
Oct 31 12:48:47.231522 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:47.231407    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5m885\" (UniqueName: \"kubernetes.io/projected/e08b08ff-4f40-474a-a103-2c499e999942-kube-api-access-5m885\") pod \"e2e-inf-a-e98df8-openshift-default-779d98479-r2jbq\" (UID: \"e08b08ff-4f40-474a-a103-2c499e999942\") " pod="openshift-ingress/e2e-inf-a-e98df8-openshift-default-779d98479-r2jbq"
Oct 31 12:48:47.240916 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:47.234986    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e08b08ff-4f40-474a-a103-2c499e999942-istio-token\") pod \"e2e-inf-a-e98df8-openshift-default-779d98479-r2jbq\" (UID: \"e08b08ff-4f40-474a-a103-2c499e999942\") " pod="openshift-ingress/e2e-inf-a-e98df8-openshift-default-779d98479-r2jbq"
Oct 31 12:48:47.247128 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:47.246829    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e98df8-openshift-default-779d98479-r2jbq"
Oct 31 12:48:48.120695 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:48:48.120648    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode08b08ff_4f40_474a_a103_2c499e999942.slice/crio-afbfc93954bf646eccabaeeb83fb2b76c0ecce13d4945e9b381f2c06c00d9d94 WatchSource:0}: Error finding container afbfc93954bf646eccabaeeb83fb2b76c0ecce13d4945e9b381f2c06c00d9d94: Status 404 returned error can't find the container with id afbfc93954bf646eccabaeeb83fb2b76c0ecce13d4945e9b381f2c06c00d9d94
Oct 31 12:48:48.121347 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:48.121322    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-e98df8-openshift-default-779d98479-r2jbq"]
Oct 31 12:48:48.124016 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:48.123980    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:48:48.124132 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:48.124052    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:48:48.124132 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:48.124090    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:48:48.758305 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:48.758264    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e98df8-openshift-default-779d98479-r2jbq" event={"ID":"e08b08ff-4f40-474a-a103-2c499e999942","Type":"ContainerStarted","Data":"b58adec0fddb21aa3934309a7e11d9cd4382f30d2afafd76ca744f39eef3cd01"}
Oct 31 12:48:48.759620 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:48.759591    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e98df8-openshift-default-779d98479-r2jbq" event={"ID":"e08b08ff-4f40-474a-a103-2c499e999942","Type":"ContainerStarted","Data":"afbfc93954bf646eccabaeeb83fb2b76c0ecce13d4945e9b381f2c06c00d9d94"}
Oct 31 12:48:48.793520 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:48.793437    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-e98df8-openshift-default-779d98479-r2jbq" podStartSLOduration=2.793418316 podStartE2EDuration="2.793418316s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:48:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:48:48.785846916 +0000 UTC m=+1291.888914664" watchObservedRunningTime="2026-10-31 12:48:48.793418316 +0000 UTC m=+1291.896486031"
Oct 31 12:48:49.138304 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:49.138264    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e43d8b-765bcc4ffb-g4s6x"]
Oct 31 12:48:49.139670 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:49.139596    5060 cpu_manager.go:414] "containerMap: removing container" podUID="440e8c3e-006e-4a6b-82d1-5fb80f5e26d1" containerName="istio-proxy"
Oct 31 12:48:49.139670 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:49.139621    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="440e8c3e-006e-4a6b-82d1-5fb80f5e26d1" containerName="istio-proxy"
Oct 31 12:48:49.140782 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:49.139936    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="440e8c3e-006e-4a6b-82d1-5fb80f5e26d1" containerName="istio-proxy"
Oct 31 12:48:49.214553 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:49.196878    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e43d8b-765bcc4ffb-g4s6x"
Oct 31 12:48:49.214553 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:49.209102    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-e43d8b-serving-cert\""
Oct 31 12:48:49.214553 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:49.209289    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 31 12:48:49.214553 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:49.209451    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-6jkzx\""
Oct 31 12:48:49.225582 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:49.221832    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 12:48:49.225582 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:49.221954    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 12:48:49.230729 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:49.230668    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e43d8b-765bcc4ffb-g4s6x"]
Oct 31 12:48:49.289426 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:49.287470    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-e98df8-openshift-default-779d98479-r2jbq"
Oct 31 12:48:49.312672 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:49.312622    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-e98df8-openshift-default-779d98479-r2jbq"
Oct 31 12:48:49.317552 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:49.315026    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dec1f857-9b34-458f-8954-56d2753466db-openshift-service-ca\") pod \"maas-api-e2e-inf-a-e43d8b-765bcc4ffb-g4s6x\" (UID: \"dec1f857-9b34-458f-8954-56d2753466db\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e43d8b-765bcc4ffb-g4s6x"
Oct 31 12:48:49.317552 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:49.315193    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m86mk\" (UniqueName: \"kubernetes.io/projected/dec1f857-9b34-458f-8954-56d2753466db-kube-api-access-m86mk\") pod \"maas-api-e2e-inf-a-e43d8b-765bcc4ffb-g4s6x\" (UID: \"dec1f857-9b34-458f-8954-56d2753466db\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e43d8b-765bcc4ffb-g4s6x"
Oct 31 12:48:49.317552 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:49.315252    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dec1f857-9b34-458f-8954-56d2753466db-maas-api-tls\") pod \"maas-api-e2e-inf-a-e43d8b-765bcc4ffb-g4s6x\" (UID: \"dec1f857-9b34-458f-8954-56d2753466db\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e43d8b-765bcc4ffb-g4s6x"
Oct 31 12:48:49.317552 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:49.317430    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dec1f857-9b34-458f-8954-56d2753466db-metrics-certs\") pod \"maas-api-e2e-inf-a-e43d8b-765bcc4ffb-g4s6x\" (UID: \"dec1f857-9b34-458f-8954-56d2753466db\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e43d8b-765bcc4ffb-g4s6x"
Oct 31 12:48:49.419202 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:49.419109    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m86mk\" (UniqueName: \"kubernetes.io/projected/dec1f857-9b34-458f-8954-56d2753466db-kube-api-access-m86mk\") pod \"maas-api-e2e-inf-a-e43d8b-765bcc4ffb-g4s6x\" (UID: \"dec1f857-9b34-458f-8954-56d2753466db\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e43d8b-765bcc4ffb-g4s6x"
Oct 31 12:48:49.419202 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:49.419176    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dec1f857-9b34-458f-8954-56d2753466db-maas-api-tls\") pod \"maas-api-e2e-inf-a-e43d8b-765bcc4ffb-g4s6x\" (UID: \"dec1f857-9b34-458f-8954-56d2753466db\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e43d8b-765bcc4ffb-g4s6x"
Oct 31 12:48:49.419424 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:49.419394    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dec1f857-9b34-458f-8954-56d2753466db-metrics-certs\") pod \"maas-api-e2e-inf-a-e43d8b-765bcc4ffb-g4s6x\" (UID: \"dec1f857-9b34-458f-8954-56d2753466db\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e43d8b-765bcc4ffb-g4s6x"
Oct 31 12:48:49.419672 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:49.419648    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dec1f857-9b34-458f-8954-56d2753466db-openshift-service-ca\") pod \"maas-api-e2e-inf-a-e43d8b-765bcc4ffb-g4s6x\" (UID: \"dec1f857-9b34-458f-8954-56d2753466db\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e43d8b-765bcc4ffb-g4s6x"
Oct 31 12:48:49.420470 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:49.420440    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dec1f857-9b34-458f-8954-56d2753466db-openshift-service-ca\") pod \"maas-api-e2e-inf-a-e43d8b-765bcc4ffb-g4s6x\" (UID: \"dec1f857-9b34-458f-8954-56d2753466db\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e43d8b-765bcc4ffb-g4s6x"
Oct 31 12:48:49.421870 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:49.421846    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dec1f857-9b34-458f-8954-56d2753466db-metrics-certs\") pod \"maas-api-e2e-inf-a-e43d8b-765bcc4ffb-g4s6x\" (UID: \"dec1f857-9b34-458f-8954-56d2753466db\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e43d8b-765bcc4ffb-g4s6x"
Oct 31 12:48:49.421989 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:49.421896    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dec1f857-9b34-458f-8954-56d2753466db-maas-api-tls\") pod \"maas-api-e2e-inf-a-e43d8b-765bcc4ffb-g4s6x\" (UID: \"dec1f857-9b34-458f-8954-56d2753466db\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e43d8b-765bcc4ffb-g4s6x"
Oct 31 12:48:49.437528 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:49.437459    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m86mk\" (UniqueName: \"kubernetes.io/projected/dec1f857-9b34-458f-8954-56d2753466db-kube-api-access-m86mk\") pod \"maas-api-e2e-inf-a-e43d8b-765bcc4ffb-g4s6x\" (UID: \"dec1f857-9b34-458f-8954-56d2753466db\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e43d8b-765bcc4ffb-g4s6x"
Oct 31 12:48:49.678563 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:49.672404    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e43d8b-765bcc4ffb-g4s6x"
Oct 31 12:48:49.971270 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:49.957690    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-e98df8-openshift-default-779d98479-r2jbq"
Oct 31 12:48:49.971270 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:49.958935    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-e98df8-openshift-default-779d98479-r2jbq"
Oct 31 12:48:50.601210 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:50.600863    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e43d8b-765bcc4ffb-g4s6x"]
Oct 31 12:48:50.768086 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:50.768050    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e1fc0ffe-mfq7z"]
Oct 31 12:48:50.776845 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:50.776809    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e1fc0ffe-mfq7z"]
Oct 31 12:48:50.796372 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:50.796341    5060 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 12:48:50.821586 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:50.821515    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-bb3cf5-openshift-default-5fc978478f-mw7qm"
Oct 31 12:48:50.940331 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:50.914303    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9cb64fe-d6b4-439d-bc3a-8598b7cd512e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9cb64fe-d6b4-439d-bc3a-8598b7cd512e-workload-certs\") pod \"d9cb64fe-d6b4-439d-bc3a-8598b7cd512e\" (UID: \"d9cb64fe-d6b4-439d-bc3a-8598b7cd512e\") "
Oct 31 12:48:50.940331 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:50.914379    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9cb64fe-d6b4-439d-bc3a-8598b7cd512e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9cb64fe-d6b4-439d-bc3a-8598b7cd512e-istiod-ca-cert\") pod \"d9cb64fe-d6b4-439d-bc3a-8598b7cd512e\" (UID: \"d9cb64fe-d6b4-439d-bc3a-8598b7cd512e\") "
Oct 31 12:48:50.940331 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:50.914428    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9cb64fe-d6b4-439d-bc3a-8598b7cd512e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9cb64fe-d6b4-439d-bc3a-8598b7cd512e-credential-socket\") pod \"d9cb64fe-d6b4-439d-bc3a-8598b7cd512e\" (UID: \"d9cb64fe-d6b4-439d-bc3a-8598b7cd512e\") "
Oct 31 12:48:50.940331 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:50.914629    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9cb64fe-d6b4-439d-bc3a-8598b7cd512e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9cb64fe-d6b4-439d-bc3a-8598b7cd512e-istio-envoy\") pod \"d9cb64fe-d6b4-439d-bc3a-8598b7cd512e\" (UID: \"d9cb64fe-d6b4-439d-bc3a-8598b7cd512e\") "
Oct 31 12:48:50.940331 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:50.914841    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9cb64fe-d6b4-439d-bc3a-8598b7cd512e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9cb64fe-d6b4-439d-bc3a-8598b7cd512e-workload-socket\") pod \"d9cb64fe-d6b4-439d-bc3a-8598b7cd512e\" (UID: \"d9cb64fe-d6b4-439d-bc3a-8598b7cd512e\") "
Oct 31 12:48:50.940331 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:50.915053    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9cb64fe-d6b4-439d-bc3a-8598b7cd512e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9cb64fe-d6b4-439d-bc3a-8598b7cd512e-istio-data\") pod \"d9cb64fe-d6b4-439d-bc3a-8598b7cd512e\" (UID: \"d9cb64fe-d6b4-439d-bc3a-8598b7cd512e\") "
Oct 31 12:48:50.940331 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:50.915084    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d9cb64fe-d6b4-439d-bc3a-8598b7cd512e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9cb64fe-d6b4-439d-bc3a-8598b7cd512e-istio-podinfo\") pod \"d9cb64fe-d6b4-439d-bc3a-8598b7cd512e\" (UID: \"d9cb64fe-d6b4-439d-bc3a-8598b7cd512e\") "
Oct 31 12:48:50.940331 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:50.915143    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9cb64fe-d6b4-439d-bc3a-8598b7cd512e-istio-token\" (UniqueName: \"kubernetes.io/projected/d9cb64fe-d6b4-439d-bc3a-8598b7cd512e-istio-token\") pod \"d9cb64fe-d6b4-439d-bc3a-8598b7cd512e\" (UID: \"d9cb64fe-d6b4-439d-bc3a-8598b7cd512e\") "
Oct 31 12:48:50.940331 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:50.915242    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9cb64fe-d6b4-439d-bc3a-8598b7cd512e-kube-api-access-6bdk9\" (UniqueName: \"kubernetes.io/projected/d9cb64fe-d6b4-439d-bc3a-8598b7cd512e-kube-api-access-6bdk9\") pod \"d9cb64fe-d6b4-439d-bc3a-8598b7cd512e\" (UID: \"d9cb64fe-d6b4-439d-bc3a-8598b7cd512e\") "
Oct 31 12:48:50.941945 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:50.941866    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9cb64fe-d6b4-439d-bc3a-8598b7cd512e-workload-certs" pod "d9cb64fe-d6b4-439d-bc3a-8598b7cd512e" (UID: "d9cb64fe-d6b4-439d-bc3a-8598b7cd512e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:48:50.942108 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:50.941973    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9cb64fe-d6b4-439d-bc3a-8598b7cd512e-workload-socket" pod "d9cb64fe-d6b4-439d-bc3a-8598b7cd512e" (UID: "d9cb64fe-d6b4-439d-bc3a-8598b7cd512e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:48:50.942433 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:50.942411    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9cb64fe-d6b4-439d-bc3a-8598b7cd512e-credential-socket" pod "d9cb64fe-d6b4-439d-bc3a-8598b7cd512e" (UID: "d9cb64fe-d6b4-439d-bc3a-8598b7cd512e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:48:50.944914 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:50.944883    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9cb64fe-d6b4-439d-bc3a-8598b7cd512e-istiod-ca-cert" pod "d9cb64fe-d6b4-439d-bc3a-8598b7cd512e" (UID: "d9cb64fe-d6b4-439d-bc3a-8598b7cd512e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:48:50.966185 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:50.966126    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9cb64fe-d6b4-439d-bc3a-8598b7cd512e-istio-data" pod "d9cb64fe-d6b4-439d-bc3a-8598b7cd512e" (UID: "d9cb64fe-d6b4-439d-bc3a-8598b7cd512e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:48:51.019280 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:51.019244    5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9cb64fe-d6b4-439d-bc3a-8598b7cd512e-workload-socket\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:48:51.038310 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:51.038273    5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9cb64fe-d6b4-439d-bc3a-8598b7cd512e-istio-data\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:48:51.038310 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:51.038312    5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9cb64fe-d6b4-439d-bc3a-8598b7cd512e-workload-certs\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:48:51.038519 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:51.038327    5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9cb64fe-d6b4-439d-bc3a-8598b7cd512e-istiod-ca-cert\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:48:51.038519 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:51.038342    5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9cb64fe-d6b4-439d-bc3a-8598b7cd512e-credential-socket\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:48:51.038519 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:51.033866    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-bb3cf5-openshift-default-5fc978478f-mw7qm"
Oct 31 12:48:51.038781 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:51.033781    5060 generic.go:356] "Generic (PLEG): container finished" podID="d9cb64fe-d6b4-439d-bc3a-8598b7cd512e" containerID="d4a8f0047375c3091a7328bd485e99dc745049f6dbe9a643546d3273f279c468" exitCode=0
Oct 31 12:48:51.039227 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:51.033807    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-bb3cf5-openshift-default-5fc978478f-mw7qm" event={"ID":"d9cb64fe-d6b4-439d-bc3a-8598b7cd512e","Type":"ContainerDied","Data":"d4a8f0047375c3091a7328bd485e99dc745049f6dbe9a643546d3273f279c468"}
Oct 31 12:48:51.039227 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:51.039093    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-bb3cf5-openshift-default-5fc978478f-mw7qm" event={"ID":"d9cb64fe-d6b4-439d-bc3a-8598b7cd512e","Type":"ContainerDied","Data":"3dcf1e815720a92211f45f85fd88f04a245b38d399fa98bed0e0a5dcff3d1d5b"}
Oct 31 12:48:51.039227 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:51.039123    5060 scope.go:122] "RemoveContainer" containerID="d4a8f0047375c3091a7328bd485e99dc745049f6dbe9a643546d3273f279c468"
Oct 31 12:48:51.041714 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:51.039893    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9cb64fe-d6b4-439d-bc3a-8598b7cd512e-istio-envoy" pod "d9cb64fe-d6b4-439d-bc3a-8598b7cd512e" (UID: "d9cb64fe-d6b4-439d-bc3a-8598b7cd512e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:48:51.041714 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:51.041676    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9cb64fe-d6b4-439d-bc3a-8598b7cd512e-kube-api-access-6bdk9" pod "d9cb64fe-d6b4-439d-bc3a-8598b7cd512e" (UID: "d9cb64fe-d6b4-439d-bc3a-8598b7cd512e"). InnerVolumeSpecName "kube-api-access-6bdk9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:48:51.044754 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:51.044714    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d9cb64fe-d6b4-439d-bc3a-8598b7cd512e-istio-podinfo" pod "d9cb64fe-d6b4-439d-bc3a-8598b7cd512e" (UID: "d9cb64fe-d6b4-439d-bc3a-8598b7cd512e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 12:48:51.044754 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:51.044737    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9cb64fe-d6b4-439d-bc3a-8598b7cd512e-istio-token" pod "d9cb64fe-d6b4-439d-bc3a-8598b7cd512e" (UID: "d9cb64fe-d6b4-439d-bc3a-8598b7cd512e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:48:51.053285 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:51.053257    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7110c247-4202-4b54-bc1b-3d4cf7ae05d1" path="/var/lib/kubelet/pods/7110c247-4202-4b54-bc1b-3d4cf7ae05d1/volumes"
Oct 31 12:48:51.053907 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:51.053886    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e43d8b-765bcc4ffb-g4s6x" event={"ID":"dec1f857-9b34-458f-8954-56d2753466db","Type":"ContainerStarted","Data":"6ed65c0b483f070f3f7aa452be1cab42bbb3e9d9363fa146ba884b44f66d70d4"}
Oct 31 12:48:51.058501 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:51.058461    5060 scope.go:122] "RemoveContainer" containerID="15e608e4b881d6dc2804ab6cfc24e27e38e7e9798db2a22b07b5bdd67fffaf78"
Oct 31 12:48:51.081388 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:51.081363    5060 scope.go:122] "RemoveContainer" containerID="d4a8f0047375c3091a7328bd485e99dc745049f6dbe9a643546d3273f279c468"
Oct 31 12:48:51.081817 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:48:51.081793    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4a8f0047375c3091a7328bd485e99dc745049f6dbe9a643546d3273f279c468\": container with ID starting with d4a8f0047375c3091a7328bd485e99dc745049f6dbe9a643546d3273f279c468 not found: ID does not exist" containerID="d4a8f0047375c3091a7328bd485e99dc745049f6dbe9a643546d3273f279c468"
Oct 31 12:48:51.081906 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:51.081825    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4a8f0047375c3091a7328bd485e99dc745049f6dbe9a643546d3273f279c468"} err="failed to get container status \"d4a8f0047375c3091a7328bd485e99dc745049f6dbe9a643546d3273f279c468\": rpc error: code = NotFound desc = could not find container \"d4a8f0047375c3091a7328bd485e99dc745049f6dbe9a643546d3273f279c468\": container with ID starting with d4a8f0047375c3091a7328bd485e99dc745049f6dbe9a643546d3273f279c468 not found: ID does not exist"
Oct 31 12:48:51.081906 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:51.081844    5060 scope.go:122] "RemoveContainer" containerID="15e608e4b881d6dc2804ab6cfc24e27e38e7e9798db2a22b07b5bdd67fffaf78"
Oct 31 12:48:51.082122 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:48:51.082103    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15e608e4b881d6dc2804ab6cfc24e27e38e7e9798db2a22b07b5bdd67fffaf78\": container with ID starting with 15e608e4b881d6dc2804ab6cfc24e27e38e7e9798db2a22b07b5bdd67fffaf78 not found: ID does not exist" containerID="15e608e4b881d6dc2804ab6cfc24e27e38e7e9798db2a22b07b5bdd67fffaf78"
Oct 31 12:48:51.082189 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:51.082130    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15e608e4b881d6dc2804ab6cfc24e27e38e7e9798db2a22b07b5bdd67fffaf78"} err="failed to get container status \"15e608e4b881d6dc2804ab6cfc24e27e38e7e9798db2a22b07b5bdd67fffaf78\": rpc error: code = NotFound desc = could not find container \"15e608e4b881d6dc2804ab6cfc24e27e38e7e9798db2a22b07b5bdd67fffaf78\": container with ID starting with 15e608e4b881d6dc2804ab6cfc24e27e38e7e9798db2a22b07b5bdd67fffaf78 not found: ID does not exist"
Oct 31 12:48:51.143040 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:51.142931    5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9cb64fe-d6b4-439d-bc3a-8598b7cd512e-istio-envoy\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:48:51.143040 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:51.142993    5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9cb64fe-d6b4-439d-bc3a-8598b7cd512e-istio-podinfo\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:48:51.143040 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:51.143011    5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9cb64fe-d6b4-439d-bc3a-8598b7cd512e-istio-token\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:48:51.143726 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:51.143052    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6bdk9\" (UniqueName: \"kubernetes.io/projected/d9cb64fe-d6b4-439d-bc3a-8598b7cd512e-kube-api-access-6bdk9\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:48:51.372184 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:51.372097    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-bb3cf5-openshift-default-5fc978478f-mw7qm"]
Oct 31 12:48:51.379257 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:51.379222    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-bb3cf5-openshift-default-5fc978478f-mw7qm"]
Oct 31 12:48:53.130970 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:53.130933    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9cb64fe-d6b4-439d-bc3a-8598b7cd512e" path="/var/lib/kubelet/pods/d9cb64fe-d6b4-439d-bc3a-8598b7cd512e/volumes"
Oct 31 12:48:53.777502 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:53.777452    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-dm6jq"]
Oct 31 12:48:53.778847 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:53.778819    5060 cpu_manager.go:414] "containerMap: removing container" podUID="d9cb64fe-d6b4-439d-bc3a-8598b7cd512e" containerName="istio-proxy"
Oct 31 12:48:53.778847 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:53.778848    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9cb64fe-d6b4-439d-bc3a-8598b7cd512e" containerName="istio-proxy"
Oct 31 12:48:53.779736 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:53.778870    5060 cpu_manager.go:414] "containerMap: removing container" podUID="d9cb64fe-d6b4-439d-bc3a-8598b7cd512e" containerName="istio-proxy"
Oct 31 12:48:53.779736 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:53.778879    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9cb64fe-d6b4-439d-bc3a-8598b7cd512e" containerName="istio-proxy"
Oct 31 12:48:53.779736 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:53.779095    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9cb64fe-d6b4-439d-bc3a-8598b7cd512e" containerName="istio-proxy"
Oct 31 12:48:53.779736 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:53.779115    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9cb64fe-d6b4-439d-bc3a-8598b7cd512e" containerName="istio-proxy"
Oct 31 12:48:53.784045 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:53.784024    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-dm6jq"
Oct 31 12:48:53.816319 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:53.816285    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-dm6jq"]
Oct 31 12:48:53.863545 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:53.863508    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6750d8d2-c91b-4d08-9553-2fe15afb061a-istio-envoy\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-dm6jq\" (UID: \"6750d8d2-c91b-4d08-9553-2fe15afb061a\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-dm6jq"
Oct 31 12:48:53.863749 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:53.863619    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6750d8d2-c91b-4d08-9553-2fe15afb061a-credential-socket\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-dm6jq\" (UID: \"6750d8d2-c91b-4d08-9553-2fe15afb061a\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-dm6jq"
Oct 31 12:48:53.863749 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:53.863671    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zjq9\" (UniqueName: \"kubernetes.io/projected/6750d8d2-c91b-4d08-9553-2fe15afb061a-kube-api-access-8zjq9\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-dm6jq\" (UID: \"6750d8d2-c91b-4d08-9553-2fe15afb061a\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-dm6jq"
Oct 31 12:48:53.863873 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:53.863759    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6750d8d2-c91b-4d08-9553-2fe15afb061a-workload-socket\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-dm6jq\" (UID: \"6750d8d2-c91b-4d08-9553-2fe15afb061a\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-dm6jq"
Oct 31 12:48:53.863873 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:53.863783    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6750d8d2-c91b-4d08-9553-2fe15afb061a-istio-token\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-dm6jq\" (UID: \"6750d8d2-c91b-4d08-9553-2fe15afb061a\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-dm6jq"
Oct 31 12:48:53.863873 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:53.863842    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6750d8d2-c91b-4d08-9553-2fe15afb061a-istiod-ca-cert\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-dm6jq\" (UID: \"6750d8d2-c91b-4d08-9553-2fe15afb061a\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-dm6jq"
Oct 31 12:48:53.864039 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:53.863919    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6750d8d2-c91b-4d08-9553-2fe15afb061a-istio-data\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-dm6jq\" (UID: \"6750d8d2-c91b-4d08-9553-2fe15afb061a\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-dm6jq"
Oct 31 12:48:53.864039 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:53.863979    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6750d8d2-c91b-4d08-9553-2fe15afb061a-istio-podinfo\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-dm6jq\" (UID: \"6750d8d2-c91b-4d08-9553-2fe15afb061a\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-dm6jq"
Oct 31 12:48:53.864039 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:53.864015    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6750d8d2-c91b-4d08-9553-2fe15afb061a-workload-certs\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-dm6jq\" (UID: \"6750d8d2-c91b-4d08-9553-2fe15afb061a\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-dm6jq"
Oct 31 12:48:53.969283 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:53.969176    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6750d8d2-c91b-4d08-9553-2fe15afb061a-credential-socket\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-dm6jq\" (UID: \"6750d8d2-c91b-4d08-9553-2fe15afb061a\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-dm6jq"
Oct 31 12:48:53.969283 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:53.969279    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zjq9\" (UniqueName: \"kubernetes.io/projected/6750d8d2-c91b-4d08-9553-2fe15afb061a-kube-api-access-8zjq9\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-dm6jq\" (UID: \"6750d8d2-c91b-4d08-9553-2fe15afb061a\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-dm6jq"
Oct 31 12:48:53.969567 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:53.969371    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6750d8d2-c91b-4d08-9553-2fe15afb061a-workload-socket\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-dm6jq\" (UID: \"6750d8d2-c91b-4d08-9553-2fe15afb061a\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-dm6jq"
Oct 31 12:48:53.969567 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:53.969408    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6750d8d2-c91b-4d08-9553-2fe15afb061a-istio-token\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-dm6jq\" (UID: \"6750d8d2-c91b-4d08-9553-2fe15afb061a\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-dm6jq"
Oct 31 12:48:53.969567 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:53.969466    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6750d8d2-c91b-4d08-9553-2fe15afb061a-istiod-ca-cert\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-dm6jq\" (UID: \"6750d8d2-c91b-4d08-9553-2fe15afb061a\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-dm6jq"
Oct 31 12:48:53.969734 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:53.969583    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6750d8d2-c91b-4d08-9553-2fe15afb061a-istio-data\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-dm6jq\" (UID: \"6750d8d2-c91b-4d08-9553-2fe15afb061a\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-dm6jq"
Oct 31 12:48:53.969734 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:53.969658    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6750d8d2-c91b-4d08-9553-2fe15afb061a-istio-podinfo\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-dm6jq\" (UID: \"6750d8d2-c91b-4d08-9553-2fe15afb061a\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-dm6jq"
Oct 31 12:48:53.969734 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:53.969721    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6750d8d2-c91b-4d08-9553-2fe15afb061a-workload-certs\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-dm6jq\" (UID: \"6750d8d2-c91b-4d08-9553-2fe15afb061a\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-dm6jq"
Oct 31 12:48:53.969930 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:53.969907    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6750d8d2-c91b-4d08-9553-2fe15afb061a-istio-envoy\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-dm6jq\" (UID: \"6750d8d2-c91b-4d08-9553-2fe15afb061a\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-dm6jq"
Oct 31 12:48:53.974968 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:53.970775    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6750d8d2-c91b-4d08-9553-2fe15afb061a-credential-socket\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-dm6jq\" (UID: \"6750d8d2-c91b-4d08-9553-2fe15afb061a\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-dm6jq"
Oct 31 12:48:53.975136 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:53.971003    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6750d8d2-c91b-4d08-9553-2fe15afb061a-istio-data\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-dm6jq\" (UID: \"6750d8d2-c91b-4d08-9553-2fe15afb061a\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-dm6jq"
Oct 31 12:48:53.975136 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:53.971616    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6750d8d2-c91b-4d08-9553-2fe15afb061a-istiod-ca-cert\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-dm6jq\" (UID: \"6750d8d2-c91b-4d08-9553-2fe15afb061a\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-dm6jq"
Oct 31 12:48:53.975136 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:53.972980    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6750d8d2-c91b-4d08-9553-2fe15afb061a-workload-socket\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-dm6jq\" (UID: \"6750d8d2-c91b-4d08-9553-2fe15afb061a\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-dm6jq"
Oct 31 12:48:53.975136 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:53.973232    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6750d8d2-c91b-4d08-9553-2fe15afb061a-workload-certs\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-dm6jq\" (UID: \"6750d8d2-c91b-4d08-9553-2fe15afb061a\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-dm6jq"
Oct 31 12:48:53.996798 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:53.996708    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6750d8d2-c91b-4d08-9553-2fe15afb061a-istio-envoy\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-dm6jq\" (UID: \"6750d8d2-c91b-4d08-9553-2fe15afb061a\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-dm6jq"
Oct 31 12:48:53.997603 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:53.997577    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6750d8d2-c91b-4d08-9553-2fe15afb061a-istio-podinfo\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-dm6jq\" (UID: \"6750d8d2-c91b-4d08-9553-2fe15afb061a\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-dm6jq"
Oct 31 12:48:54.010574 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:54.010463    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6750d8d2-c91b-4d08-9553-2fe15afb061a-istio-token\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-dm6jq\" (UID: \"6750d8d2-c91b-4d08-9553-2fe15afb061a\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-dm6jq"
Oct 31 12:48:54.023132 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:54.023071    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zjq9\" (UniqueName: \"kubernetes.io/projected/6750d8d2-c91b-4d08-9553-2fe15afb061a-kube-api-access-8zjq9\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-dm6jq\" (UID: \"6750d8d2-c91b-4d08-9553-2fe15afb061a\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-dm6jq"
Oct 31 12:48:54.116148 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:54.116113    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-dm6jq"
Oct 31 12:48:54.177546 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:54.175233    5060 generic.go:356] "Generic (PLEG): container finished" podID="b4e57789-6b1a-4d17-bdaf-925a3d3d0c43" containerID="a4d096354cb95f71298f5e7a682f3e560e1fa6bb66078dbb6336fb4812ff473d" exitCode=0
Oct 31 12:48:54.177546 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:54.175533    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-787316-openshift-default-58b9b668c5-qpf4t" event={"ID":"b4e57789-6b1a-4d17-bdaf-925a3d3d0c43","Type":"ContainerDied","Data":"a4d096354cb95f71298f5e7a682f3e560e1fa6bb66078dbb6336fb4812ff473d"}
Oct 31 12:48:54.177546 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:54.176117    5060 scope.go:122] "RemoveContainer" containerID="a4d096354cb95f71298f5e7a682f3e560e1fa6bb66078dbb6336fb4812ff473d"
Oct 31 12:48:54.181053 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:54.180120    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:48:54.181053 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:54.180242    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:48:54.181053 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:54.180289    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:48:54.201206 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:54.200778    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e43d8b-765bcc4ffb-g4s6x" event={"ID":"dec1f857-9b34-458f-8954-56d2753466db","Type":"ContainerStarted","Data":"899bb28709d691b64688ef03645611a5e6dc2d0ef12a447bd5722fa22622a316"}
Oct 31 12:48:54.202145 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:54.202087    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e43d8b-765bcc4ffb-g4s6x"
Oct 31 12:48:54.325325 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:54.325279    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e43d8b-765bcc4ffb-g4s6x" podStartSLOduration=2.444105753 podStartE2EDuration="5.32526058s" totalImagesPullingTime="2.881154827s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:48:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:48:54.322760329 +0000 UTC m=+1297.425828043" watchObservedRunningTime="2026-10-31 12:48:54.32526058 +0000 UTC m=+1297.428328293"
Oct 31 12:48:54.446807 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:54.446770    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-dm6jq"]
Oct 31 12:48:54.498982 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:54.482368    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:48:54.498982 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:54.485629    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:48:54.498982 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:54.485682    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:48:55.214695 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:55.214656    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-dm6jq" event={"ID":"6750d8d2-c91b-4d08-9553-2fe15afb061a","Type":"ContainerStarted","Data":"4a4fa34a0917cc911ea4ce7ecf3411bc3464d8a28cbdc5a67b9fc4e5b2ac6fa9"}
Oct 31 12:48:55.215105 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:55.214725    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-dm6jq" event={"ID":"6750d8d2-c91b-4d08-9553-2fe15afb061a","Type":"ContainerStarted","Data":"61dcff1427466a44995a9e918eb77dce6ce41bbb1a96ae00c2ddc9246fbd0119"}
Oct 31 12:48:55.217090 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:55.217058    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-787316-openshift-default-58b9b668c5-qpf4t" event={"ID":"b4e57789-6b1a-4d17-bdaf-925a3d3d0c43","Type":"ContainerStarted","Data":"09f8ef560f674ab0efd43c8d325d8b659416abfbfb18f08407c0923d4a0e71d6"}
Oct 31 12:48:55.241758 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:55.241706    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-dm6jq" podStartSLOduration=2.24169228 podStartE2EDuration="2.24169228s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:48:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:48:55.237902926 +0000 UTC m=+1298.340970637" watchObservedRunningTime="2026-10-31 12:48:55.24169228 +0000 UTC m=+1298.344760041"
Oct 31 12:48:56.120229 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:56.120193    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-787316-openshift-default-58b9b668c5-qpf4t"
Oct 31 12:48:56.120619 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:56.120595    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-dm6jq"
Oct 31 12:48:56.122624 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:56.122597    5060 patch_prober.go:29] interesting pod/e2e-api-b-787316-openshift-default-58b9b668c5-qpf4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" start-of-body=
Oct 31 12:48:56.122813 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:56.122786    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-787316-openshift-default-58b9b668c5-qpf4t" podUID="b4e57789-6b1a-4d17-bdaf-925a3d3d0c43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused"
Oct 31 12:48:56.123190 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:56.123145    5060 patch_prober.go:29] interesting pod/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-dm6jq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" start-of-body=
Oct 31 12:48:56.123296 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:56.123205    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-dm6jq" podUID="6750d8d2-c91b-4d08-9553-2fe15afb061a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused"
Oct 31 12:48:57.071367 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:57.071329    5060 patch_prober.go:29] interesting pod/e2e-api-b-787316-openshift-default-58b9b668c5-qpf4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" start-of-body=
Oct 31 12:48:57.071990 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:57.071950    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-787316-openshift-default-58b9b668c5-qpf4t" podUID="b4e57789-6b1a-4d17-bdaf-925a3d3d0c43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused"
Oct 31 12:48:57.122511 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:57.116895    5060 patch_prober.go:29] interesting pod/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-dm6jq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" start-of-body=
Oct 31 12:48:57.122511 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:57.116957    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-dm6jq" podUID="6750d8d2-c91b-4d08-9553-2fe15afb061a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused"
Oct 31 12:48:57.629952 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:57.629916    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5c5815-cbc4t"]
Oct 31 12:48:57.751521 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:57.723697    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5c5815-cbc4t"]
Oct 31 12:48:58.071372 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:58.071332    5060 patch_prober.go:29] interesting pod/e2e-api-b-787316-openshift-default-58b9b668c5-qpf4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" start-of-body=
Oct 31 12:48:58.071996 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:58.071963    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-787316-openshift-default-58b9b668c5-qpf4t" podUID="b4e57789-6b1a-4d17-bdaf-925a3d3d0c43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused"
Oct 31 12:48:58.229834 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:58.229675    5060 patch_prober.go:29] interesting pod/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-dm6jq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" start-of-body=
Oct 31 12:48:58.231229 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:58.230852    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-dm6jq" podUID="6750d8d2-c91b-4d08-9553-2fe15afb061a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused"
Oct 31 12:48:59.051710 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:59.051670    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e39c020a-e0e5-4a0c-9c52-3d63241cc13c" path="/var/lib/kubelet/pods/e39c020a-e0e5-4a0c-9c52-3d63241cc13c/volumes"
Oct 31 12:48:59.073923 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:59.073881    5060 patch_prober.go:29] interesting pod/e2e-api-b-787316-openshift-default-58b9b668c5-qpf4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" start-of-body=
Oct 31 12:48:59.074475 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:59.074442    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-787316-openshift-default-58b9b668c5-qpf4t" podUID="b4e57789-6b1a-4d17-bdaf-925a3d3d0c43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused"
Oct 31 12:48:59.118516 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:59.117898    5060 patch_prober.go:29] interesting pod/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-dm6jq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" start-of-body=
Oct 31 12:48:59.118516 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:59.117959    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-dm6jq" podUID="6750d8d2-c91b-4d08-9553-2fe15afb061a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused"
Oct 31 12:48:59.283515 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:48:59.267586    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-787316-openshift-default-58b9b668c5-qpf4t"
Oct 31 12:49:00.084521 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:00.076930    5060 patch_prober.go:29] interesting pod/e2e-api-b-787316-openshift-default-58b9b668c5-qpf4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" start-of-body=
Oct 31 12:49:00.084521 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:00.076989    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-787316-openshift-default-58b9b668c5-qpf4t" podUID="b4e57789-6b1a-4d17-bdaf-925a3d3d0c43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused"
Oct 31 12:49:00.118513 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:00.117953    5060 patch_prober.go:29] interesting pod/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-dm6jq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" start-of-body=
Oct 31 12:49:00.118513 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:00.118019    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-dm6jq" podUID="6750d8d2-c91b-4d08-9553-2fe15afb061a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused"
Oct 31 12:49:00.131811 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:49:00.131775    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7phb" containerName="extractor"
Oct 31 12:49:01.083520 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:01.079891    5060 patch_prober.go:29] interesting pod/e2e-api-b-787316-openshift-default-58b9b668c5-qpf4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" start-of-body=
Oct 31 12:49:01.083520 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:01.079954    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-787316-openshift-default-58b9b668c5-qpf4t" podUID="b4e57789-6b1a-4d17-bdaf-925a3d3d0c43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused"
Oct 31 12:49:01.119526 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:01.118907    5060 patch_prober.go:29] interesting pod/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-dm6jq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" start-of-body=
Oct 31 12:49:01.119526 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:01.118966    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-dm6jq" podUID="6750d8d2-c91b-4d08-9553-2fe15afb061a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused"
Oct 31 12:49:01.451567 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:01.439678    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e43d8b-765bcc4ffb-g4s6x"
Oct 31 12:49:02.140521 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:02.140078    5060 patch_prober.go:29] interesting pod/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-dm6jq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" start-of-body=
Oct 31 12:49:02.140521 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:02.140176    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-dm6jq" podUID="6750d8d2-c91b-4d08-9553-2fe15afb061a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused"
Oct 31 12:49:02.171518 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:02.168814    5060 patch_prober.go:29] interesting pod/e2e-api-b-787316-openshift-default-58b9b668c5-qpf4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" start-of-body=
Oct 31 12:49:02.171518 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:02.168910    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-787316-openshift-default-58b9b668c5-qpf4t" podUID="b4e57789-6b1a-4d17-bdaf-925a3d3d0c43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused"
Oct 31 12:49:03.151515 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:03.144042    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-787316-openshift-default-58b9b668c5-qpf4t"
Oct 31 12:49:03.169826 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:03.169630    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-787316-openshift-default-58b9b668c5-qpf4t"
Oct 31 12:49:03.214695 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:03.214640    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-dm6jq"
Oct 31 12:49:03.215364 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:03.215335    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-dm6jq"
Oct 31 12:49:03.218222 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:03.218190    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-dm6jq"
Oct 31 12:49:07.579278 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:07.579245    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-787316-hvr6k"]
Oct 31 12:49:07.585937 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:07.585911    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-787316-hvr6k"]
Oct 31 12:49:07.586106 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:07.586076    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-787316-hvr6k"
Oct 31 12:49:07.591751 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:07.591726    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 12:49:07.591997 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:07.591975    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7wsmb\""
Oct 31 12:49:07.762799 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:07.762761    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7j58\" (UniqueName: \"kubernetes.io/projected/ccb0f2b8-e593-491b-8aff-d0463a95c5b1-kube-api-access-s7j58\") pod \"maas-api-revoke-keys-e2e-api-b-787316-hvr6k\" (UID: \"ccb0f2b8-e593-491b-8aff-d0463a95c5b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-787316-hvr6k"
Oct 31 12:49:07.762974 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:07.762885    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccb0f2b8-e593-491b-8aff-d0463a95c5b1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-787316-hvr6k\" (UID: \"ccb0f2b8-e593-491b-8aff-d0463a95c5b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-787316-hvr6k"
Oct 31 12:49:07.763024 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:07.762992    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccb0f2b8-e593-491b-8aff-d0463a95c5b1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-787316-hvr6k\" (UID: \"ccb0f2b8-e593-491b-8aff-d0463a95c5b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-787316-hvr6k"
Oct 31 12:49:07.865421 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:07.865318    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7j58\" (UniqueName: \"kubernetes.io/projected/ccb0f2b8-e593-491b-8aff-d0463a95c5b1-kube-api-access-s7j58\") pod \"maas-api-revoke-keys-e2e-api-b-787316-hvr6k\" (UID: \"ccb0f2b8-e593-491b-8aff-d0463a95c5b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-787316-hvr6k"
Oct 31 12:49:07.865651 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:07.865453    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccb0f2b8-e593-491b-8aff-d0463a95c5b1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-787316-hvr6k\" (UID: \"ccb0f2b8-e593-491b-8aff-d0463a95c5b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-787316-hvr6k"
Oct 31 12:49:07.865651 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:07.865563    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccb0f2b8-e593-491b-8aff-d0463a95c5b1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-787316-hvr6k\" (UID: \"ccb0f2b8-e593-491b-8aff-d0463a95c5b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-787316-hvr6k"
Oct 31 12:49:07.866224 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:07.866180    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccb0f2b8-e593-491b-8aff-d0463a95c5b1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-787316-hvr6k\" (UID: \"ccb0f2b8-e593-491b-8aff-d0463a95c5b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-787316-hvr6k"
Oct 31 12:49:07.868543 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:07.868469    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccb0f2b8-e593-491b-8aff-d0463a95c5b1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-787316-hvr6k\" (UID: \"ccb0f2b8-e593-491b-8aff-d0463a95c5b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-787316-hvr6k"
Oct 31 12:49:07.884058 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:07.884030    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7j58\" (UniqueName: \"kubernetes.io/projected/ccb0f2b8-e593-491b-8aff-d0463a95c5b1-kube-api-access-s7j58\") pod \"maas-api-revoke-keys-e2e-api-b-787316-hvr6k\" (UID: \"ccb0f2b8-e593-491b-8aff-d0463a95c5b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-787316-hvr6k"
Oct 31 12:49:07.902669 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:07.902644    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-787316-hvr6k"
Oct 31 12:49:08.224961 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:49:08.224918    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccb0f2b8_e593_491b_8aff_d0463a95c5b1.slice/crio-2a00530d84d01859e9fe12ab58f885ec9db85d40199ec7c0dea7850c28dccea8 WatchSource:0}: Error finding container 2a00530d84d01859e9fe12ab58f885ec9db85d40199ec7c0dea7850c28dccea8: Status 404 returned error can't find the container with id 2a00530d84d01859e9fe12ab58f885ec9db85d40199ec7c0dea7850c28dccea8
Oct 31 12:49:08.232178 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:08.232033    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-787316-hvr6k"]
Oct 31 12:49:08.590557 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:08.590521    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-787316-hvr6k" event={"ID":"ccb0f2b8-e593-491b-8aff-d0463a95c5b1","Type":"ContainerStarted","Data":"2a00530d84d01859e9fe12ab58f885ec9db85d40199ec7c0dea7850c28dccea8"}
Oct 31 12:49:09.134106 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:09.130634    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-rr2hg"]
Oct 31 12:49:09.151512 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:09.134765    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-rr2hg" podUID="15a47e5b-2ade-4ca3-b0d0-6e4c09917588" containerName="istio-proxy" containerID="cri-o://2bb4f5e3c1b84da3d002959b676fa2bd230119ff2186a6007a5553bcc7cc4af6" gracePeriod=30
Oct 31 12:49:09.602364 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:09.602329    5060 generic.go:356] "Generic (PLEG): container finished" podID="ccb0f2b8-e593-491b-8aff-d0463a95c5b1" containerID="241445e5ee669bb9b3c7e968e26e95300e51aa8e9808e0228a5643c1d68276ac" exitCode=0
Oct 31 12:49:09.602837 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:09.602408    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-787316-hvr6k" event={"ID":"ccb0f2b8-e593-491b-8aff-d0463a95c5b1","Type":"ContainerDied","Data":"241445e5ee669bb9b3c7e968e26e95300e51aa8e9808e0228a5643c1d68276ac"}
Oct 31 12:49:10.029403 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:10.029365    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3bf1e1f-lbw5g"]
Oct 31 12:49:10.034380 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:10.034353    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3bf1e1f-lbw5g"]
Oct 31 12:49:11.651290 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:11.651252    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc3210d5-cef9-47d8-b4d1-986c62846911" path="/var/lib/kubelet/pods/bc3210d5-cef9-47d8-b4d1-986c62846911/volumes"
Oct 31 12:49:11.691622 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:11.691595    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-787316-hvr6k"
Oct 31 12:49:11.852830 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:11.852800    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccb0f2b8-e593-491b-8aff-d0463a95c5b1-kube-api-access-s7j58\" (UniqueName: \"kubernetes.io/projected/ccb0f2b8-e593-491b-8aff-d0463a95c5b1-kube-api-access-s7j58\") pod \"ccb0f2b8-e593-491b-8aff-d0463a95c5b1\" (UID: \"ccb0f2b8-e593-491b-8aff-d0463a95c5b1\") "
Oct 31 12:49:11.853025 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:11.853011    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ccb0f2b8-e593-491b-8aff-d0463a95c5b1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccb0f2b8-e593-491b-8aff-d0463a95c5b1-maas-api-xks-ca\") pod \"ccb0f2b8-e593-491b-8aff-d0463a95c5b1\" (UID: \"ccb0f2b8-e593-491b-8aff-d0463a95c5b1\") "
Oct 31 12:49:11.853076 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:11.853040    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ccb0f2b8-e593-491b-8aff-d0463a95c5b1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccb0f2b8-e593-491b-8aff-d0463a95c5b1-maas-api-openshift-ca\") pod \"ccb0f2b8-e593-491b-8aff-d0463a95c5b1\" (UID: \"ccb0f2b8-e593-491b-8aff-d0463a95c5b1\") "
Oct 31 12:49:11.853444 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:11.853415    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccb0f2b8-e593-491b-8aff-d0463a95c5b1-maas-api-openshift-ca" pod "ccb0f2b8-e593-491b-8aff-d0463a95c5b1" (UID: "ccb0f2b8-e593-491b-8aff-d0463a95c5b1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:49:11.853821 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:11.853799    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccb0f2b8-e593-491b-8aff-d0463a95c5b1-maas-api-openshift-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:49:11.855117 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:11.855095    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ccb0f2b8-e593-491b-8aff-d0463a95c5b1-maas-api-xks-ca" pod "ccb0f2b8-e593-491b-8aff-d0463a95c5b1" (UID: "ccb0f2b8-e593-491b-8aff-d0463a95c5b1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:49:11.855883 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:11.855856    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccb0f2b8-e593-491b-8aff-d0463a95c5b1-kube-api-access-s7j58" pod "ccb0f2b8-e593-491b-8aff-d0463a95c5b1" (UID: "ccb0f2b8-e593-491b-8aff-d0463a95c5b1"). InnerVolumeSpecName "kube-api-access-s7j58". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:49:11.958561 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:11.958454    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccb0f2b8-e593-491b-8aff-d0463a95c5b1-maas-api-xks-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:49:11.958561 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:11.958512    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7j58\" (UniqueName: \"kubernetes.io/projected/ccb0f2b8-e593-491b-8aff-d0463a95c5b1-kube-api-access-s7j58\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:49:12.673727 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:12.673687    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-787316-hvr6k" event={"ID":"ccb0f2b8-e593-491b-8aff-d0463a95c5b1","Type":"ContainerDied","Data":"2a00530d84d01859e9fe12ab58f885ec9db85d40199ec7c0dea7850c28dccea8"}
Oct 31 12:49:12.674228 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:12.673754    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2a00530d84d01859e9fe12ab58f885ec9db85d40199ec7c0dea7850c28dccea8"
Oct 31 12:49:12.674228 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:12.673694    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-787316-hvr6k"
Oct 31 12:49:14.698604 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:14.698552    5060 generic.go:356] "Generic (PLEG): container finished" podID="15a47e5b-2ade-4ca3-b0d0-6e4c09917588" containerID="2bb4f5e3c1b84da3d002959b676fa2bd230119ff2186a6007a5553bcc7cc4af6" exitCode=0
Oct 31 12:49:14.699100 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:14.698625    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-rr2hg" event={"ID":"15a47e5b-2ade-4ca3-b0d0-6e4c09917588","Type":"ContainerDied","Data":"2bb4f5e3c1b84da3d002959b676fa2bd230119ff2186a6007a5553bcc7cc4af6"}
Oct 31 12:49:14.894417 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:14.894386    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-rr2hg"
Oct 31 12:49:14.932468 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:14.932437    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-xmr62"]
Oct 31 12:49:14.933245 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:14.933228    5060 cpu_manager.go:414] "containerMap: removing container" podUID="ccb0f2b8-e593-491b-8aff-d0463a95c5b1" containerName="revoke-keys"
Oct 31 12:49:14.933303 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:14.933246    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ccb0f2b8-e593-491b-8aff-d0463a95c5b1" containerName="revoke-keys"
Oct 31 12:49:14.933303 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:14.933277    5060 cpu_manager.go:414] "containerMap: removing container" podUID="15a47e5b-2ade-4ca3-b0d0-6e4c09917588" containerName="istio-proxy"
Oct 31 12:49:14.933303 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:14.933282    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15a47e5b-2ade-4ca3-b0d0-6e4c09917588" containerName="istio-proxy"
Oct 31 12:49:14.933884 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:14.933400    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="15a47e5b-2ade-4ca3-b0d0-6e4c09917588" containerName="istio-proxy"
Oct 31 12:49:14.933884 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:14.933417    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="ccb0f2b8-e593-491b-8aff-d0463a95c5b1" containerName="revoke-keys"
Oct 31 12:49:14.939566 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:14.938159    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-xmr62"
Oct 31 12:49:14.967176 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:14.967137    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-xmr62"]
Oct 31 12:49:15.105389 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:15.102889    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15a47e5b-2ade-4ca3-b0d0-6e4c09917588-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15a47e5b-2ade-4ca3-b0d0-6e4c09917588-workload-certs\") pod \"15a47e5b-2ade-4ca3-b0d0-6e4c09917588\" (UID: \"15a47e5b-2ade-4ca3-b0d0-6e4c09917588\") "
Oct 31 12:49:15.105389 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:15.103008    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15a47e5b-2ade-4ca3-b0d0-6e4c09917588-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15a47e5b-2ade-4ca3-b0d0-6e4c09917588-istio-envoy\") pod \"15a47e5b-2ade-4ca3-b0d0-6e4c09917588\" (UID: \"15a47e5b-2ade-4ca3-b0d0-6e4c09917588\") "
Oct 31 12:49:15.105389 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:15.103952    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15a47e5b-2ade-4ca3-b0d0-6e4c09917588-workload-certs" pod "15a47e5b-2ade-4ca3-b0d0-6e4c09917588" (UID: "15a47e5b-2ade-4ca3-b0d0-6e4c09917588"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:49:15.124590 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:15.121180    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15a47e5b-2ade-4ca3-b0d0-6e4c09917588-istio-data" pod "15a47e5b-2ade-4ca3-b0d0-6e4c09917588" (UID: "15a47e5b-2ade-4ca3-b0d0-6e4c09917588"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:49:15.124590 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:15.123808    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15a47e5b-2ade-4ca3-b0d0-6e4c09917588-istio-envoy" pod "15a47e5b-2ade-4ca3-b0d0-6e4c09917588" (UID: "15a47e5b-2ade-4ca3-b0d0-6e4c09917588"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:49:15.133549 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:15.129657    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15a47e5b-2ade-4ca3-b0d0-6e4c09917588-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15a47e5b-2ade-4ca3-b0d0-6e4c09917588-istio-data\") pod \"15a47e5b-2ade-4ca3-b0d0-6e4c09917588\" (UID: \"15a47e5b-2ade-4ca3-b0d0-6e4c09917588\") "
Oct 31 12:49:15.133549 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:15.129770    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/15a47e5b-2ade-4ca3-b0d0-6e4c09917588-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15a47e5b-2ade-4ca3-b0d0-6e4c09917588-istio-podinfo\") pod \"15a47e5b-2ade-4ca3-b0d0-6e4c09917588\" (UID: \"15a47e5b-2ade-4ca3-b0d0-6e4c09917588\") "
Oct 31 12:49:15.133549 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:15.129949    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15a47e5b-2ade-4ca3-b0d0-6e4c09917588-kube-api-access-t4wlk\" (UniqueName: \"kubernetes.io/projected/15a47e5b-2ade-4ca3-b0d0-6e4c09917588-kube-api-access-t4wlk\") pod \"15a47e5b-2ade-4ca3-b0d0-6e4c09917588\" (UID: \"15a47e5b-2ade-4ca3-b0d0-6e4c09917588\") "
Oct 31 12:49:15.133549 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:15.129979    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15a47e5b-2ade-4ca3-b0d0-6e4c09917588-istio-token\" (UniqueName: \"kubernetes.io/projected/15a47e5b-2ade-4ca3-b0d0-6e4c09917588-istio-token\") pod \"15a47e5b-2ade-4ca3-b0d0-6e4c09917588\" (UID: \"15a47e5b-2ade-4ca3-b0d0-6e4c09917588\") "
Oct 31 12:49:15.133549 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:15.130025    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15a47e5b-2ade-4ca3-b0d0-6e4c09917588-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15a47e5b-2ade-4ca3-b0d0-6e4c09917588-istiod-ca-cert\") pod \"15a47e5b-2ade-4ca3-b0d0-6e4c09917588\" (UID: \"15a47e5b-2ade-4ca3-b0d0-6e4c09917588\") "
Oct 31 12:49:15.133549 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:15.130091    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15a47e5b-2ade-4ca3-b0d0-6e4c09917588-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15a47e5b-2ade-4ca3-b0d0-6e4c09917588-workload-socket\") pod \"15a47e5b-2ade-4ca3-b0d0-6e4c09917588\" (UID: \"15a47e5b-2ade-4ca3-b0d0-6e4c09917588\") "
Oct 31 12:49:15.133549 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:15.130189    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15a47e5b-2ade-4ca3-b0d0-6e4c09917588-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15a47e5b-2ade-4ca3-b0d0-6e4c09917588-credential-socket\") pod \"15a47e5b-2ade-4ca3-b0d0-6e4c09917588\" (UID: \"15a47e5b-2ade-4ca3-b0d0-6e4c09917588\") "
Oct 31 12:49:15.144524 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:15.142061    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15a47e5b-2ade-4ca3-b0d0-6e4c09917588-istiod-ca-cert" pod "15a47e5b-2ade-4ca3-b0d0-6e4c09917588" (UID: "15a47e5b-2ade-4ca3-b0d0-6e4c09917588"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:49:15.167523 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:15.148538    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15a47e5b-2ade-4ca3-b0d0-6e4c09917588-workload-socket" pod "15a47e5b-2ade-4ca3-b0d0-6e4c09917588" (UID: "15a47e5b-2ade-4ca3-b0d0-6e4c09917588"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:49:15.167523 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:15.148915    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15a47e5b-2ade-4ca3-b0d0-6e4c09917588-credential-socket" pod "15a47e5b-2ade-4ca3-b0d0-6e4c09917588" (UID: "15a47e5b-2ade-4ca3-b0d0-6e4c09917588"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:49:15.167523 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:15.151361    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1e6c7f5-d872-4549-b950-1e751b6374ab-istio-token\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-xmr62\" (UID: \"e1e6c7f5-d872-4549-b950-1e751b6374ab\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-xmr62"
Oct 31 12:49:15.167523 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:15.152775    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1e6c7f5-d872-4549-b950-1e751b6374ab-workload-certs\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-xmr62\" (UID: \"e1e6c7f5-d872-4549-b950-1e751b6374ab\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-xmr62"
Oct 31 12:49:15.167523 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:15.161586    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1e6c7f5-d872-4549-b950-1e751b6374ab-workload-socket\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-xmr62\" (UID: \"e1e6c7f5-d872-4549-b950-1e751b6374ab\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-xmr62"
Oct 31 12:49:15.167523 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:15.162079    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1e6c7f5-d872-4549-b950-1e751b6374ab-istio-envoy\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-xmr62\" (UID: \"e1e6c7f5-d872-4549-b950-1e751b6374ab\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-xmr62"
Oct 31 12:49:15.167523 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:15.162331    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1e6c7f5-d872-4549-b950-1e751b6374ab-istio-data\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-xmr62\" (UID: \"e1e6c7f5-d872-4549-b950-1e751b6374ab\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-xmr62"
Oct 31 12:49:15.170573 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:15.168854    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1e6c7f5-d872-4549-b950-1e751b6374ab-credential-socket\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-xmr62\" (UID: \"e1e6c7f5-d872-4549-b950-1e751b6374ab\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-xmr62"
Oct 31 12:49:15.170573 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:15.168941    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1e6c7f5-d872-4549-b950-1e751b6374ab-istiod-ca-cert\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-xmr62\" (UID: \"e1e6c7f5-d872-4549-b950-1e751b6374ab\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-xmr62"
Oct 31 12:49:15.170573 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:15.169032    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1e6c7f5-d872-4549-b950-1e751b6374ab-istio-podinfo\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-xmr62\" (UID: \"e1e6c7f5-d872-4549-b950-1e751b6374ab\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-xmr62"
Oct 31 12:49:15.170573 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:15.169732    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kc5gq\" (UniqueName: \"kubernetes.io/projected/e1e6c7f5-d872-4549-b950-1e751b6374ab-kube-api-access-kc5gq\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-xmr62\" (UID: \"e1e6c7f5-d872-4549-b950-1e751b6374ab\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-xmr62"
Oct 31 12:49:15.170573 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:15.169920    5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15a47e5b-2ade-4ca3-b0d0-6e4c09917588-istio-envoy\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:49:15.170573 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:15.170163    5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15a47e5b-2ade-4ca3-b0d0-6e4c09917588-istio-data\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:49:15.170573 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:15.170184    5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15a47e5b-2ade-4ca3-b0d0-6e4c09917588-istiod-ca-cert\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:49:15.170573 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:15.170199    5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15a47e5b-2ade-4ca3-b0d0-6e4c09917588-workload-socket\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:49:15.170573 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:15.170213    5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15a47e5b-2ade-4ca3-b0d0-6e4c09917588-credential-socket\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:49:15.170573 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:15.170227    5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15a47e5b-2ade-4ca3-b0d0-6e4c09917588-workload-certs\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:49:15.180519 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:15.180454    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/15a47e5b-2ade-4ca3-b0d0-6e4c09917588-istio-podinfo" pod "15a47e5b-2ade-4ca3-b0d0-6e4c09917588" (UID: "15a47e5b-2ade-4ca3-b0d0-6e4c09917588"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 12:49:15.180752 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:15.180548    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15a47e5b-2ade-4ca3-b0d0-6e4c09917588-istio-token" pod "15a47e5b-2ade-4ca3-b0d0-6e4c09917588" (UID: "15a47e5b-2ade-4ca3-b0d0-6e4c09917588"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:49:15.182562 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:15.182532    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15a47e5b-2ade-4ca3-b0d0-6e4c09917588-kube-api-access-t4wlk" pod "15a47e5b-2ade-4ca3-b0d0-6e4c09917588" (UID: "15a47e5b-2ade-4ca3-b0d0-6e4c09917588"). InnerVolumeSpecName "kube-api-access-t4wlk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:49:15.271362 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:15.271278    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1e6c7f5-d872-4549-b950-1e751b6374ab-credential-socket\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-xmr62\" (UID: \"e1e6c7f5-d872-4549-b950-1e751b6374ab\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-xmr62"
Oct 31 12:49:15.271362 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:15.271316    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1e6c7f5-d872-4549-b950-1e751b6374ab-istiod-ca-cert\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-xmr62\" (UID: \"e1e6c7f5-d872-4549-b950-1e751b6374ab\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-xmr62"
Oct 31 12:49:15.271636 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:15.271464    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1e6c7f5-d872-4549-b950-1e751b6374ab-istio-podinfo\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-xmr62\" (UID: \"e1e6c7f5-d872-4549-b950-1e751b6374ab\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-xmr62"
Oct 31 12:49:15.271703 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:15.271642    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kc5gq\" (UniqueName: \"kubernetes.io/projected/e1e6c7f5-d872-4549-b950-1e751b6374ab-kube-api-access-kc5gq\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-xmr62\" (UID: \"e1e6c7f5-d872-4549-b950-1e751b6374ab\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-xmr62"
Oct 31 12:49:15.271703 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:15.271690    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1e6c7f5-d872-4549-b950-1e751b6374ab-istio-token\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-xmr62\" (UID: \"e1e6c7f5-d872-4549-b950-1e751b6374ab\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-xmr62"
Oct 31 12:49:15.271807 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:15.271741    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1e6c7f5-d872-4549-b950-1e751b6374ab-credential-socket\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-xmr62\" (UID: \"e1e6c7f5-d872-4549-b950-1e751b6374ab\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-xmr62"
Oct 31 12:49:15.271902 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:15.271860    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1e6c7f5-d872-4549-b950-1e751b6374ab-workload-certs\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-xmr62\" (UID: \"e1e6c7f5-d872-4549-b950-1e751b6374ab\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-xmr62"
Oct 31 12:49:15.271966 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:15.271910    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1e6c7f5-d872-4549-b950-1e751b6374ab-workload-socket\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-xmr62\" (UID: \"e1e6c7f5-d872-4549-b950-1e751b6374ab\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-xmr62"
Oct 31 12:49:15.271966 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:15.271942    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1e6c7f5-d872-4549-b950-1e751b6374ab-istio-envoy\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-xmr62\" (UID: \"e1e6c7f5-d872-4549-b950-1e751b6374ab\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-xmr62"
Oct 31 12:49:15.272104 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:15.271983    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1e6c7f5-d872-4549-b950-1e751b6374ab-istio-data\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-xmr62\" (UID: \"e1e6c7f5-d872-4549-b950-1e751b6374ab\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-xmr62"
Oct 31 12:49:15.272104 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:15.272087    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1e6c7f5-d872-4549-b950-1e751b6374ab-istiod-ca-cert\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-xmr62\" (UID: \"e1e6c7f5-d872-4549-b950-1e751b6374ab\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-xmr62"
Oct 31 12:49:15.272232 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:15.272197    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1e6c7f5-d872-4549-b950-1e751b6374ab-workload-certs\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-xmr62\" (UID: \"e1e6c7f5-d872-4549-b950-1e751b6374ab\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-xmr62"
Oct 31 12:49:15.272287 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:15.272238    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1e6c7f5-d872-4549-b950-1e751b6374ab-workload-socket\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-xmr62\" (UID: \"e1e6c7f5-d872-4549-b950-1e751b6374ab\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-xmr62"
Oct 31 12:49:15.272287 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:15.272267    5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15a47e5b-2ade-4ca3-b0d0-6e4c09917588-istio-podinfo\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:49:15.272368 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:15.272289    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4wlk\" (UniqueName: \"kubernetes.io/projected/15a47e5b-2ade-4ca3-b0d0-6e4c09917588-kube-api-access-t4wlk\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:49:15.272368 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:15.272307    5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15a47e5b-2ade-4ca3-b0d0-6e4c09917588-istio-token\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:49:15.272368 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:15.272357    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1e6c7f5-d872-4549-b950-1e751b6374ab-istio-data\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-xmr62\" (UID: \"e1e6c7f5-d872-4549-b950-1e751b6374ab\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-xmr62"
Oct 31 12:49:15.274327 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:15.274298    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1e6c7f5-d872-4549-b950-1e751b6374ab-istio-envoy\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-xmr62\" (UID: \"e1e6c7f5-d872-4549-b950-1e751b6374ab\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-xmr62"
Oct 31 12:49:15.274461 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:15.274442    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1e6c7f5-d872-4549-b950-1e751b6374ab-istio-podinfo\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-xmr62\" (UID: \"e1e6c7f5-d872-4549-b950-1e751b6374ab\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-xmr62"
Oct 31 12:49:15.284451 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:15.284406    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1e6c7f5-d872-4549-b950-1e751b6374ab-istio-token\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-xmr62\" (UID: \"e1e6c7f5-d872-4549-b950-1e751b6374ab\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-xmr62"
Oct 31 12:49:15.284660 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:15.284641    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kc5gq\" (UniqueName: \"kubernetes.io/projected/e1e6c7f5-d872-4549-b950-1e751b6374ab-kube-api-access-kc5gq\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-xmr62\" (UID: \"e1e6c7f5-d872-4549-b950-1e751b6374ab\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-xmr62"
Oct 31 12:49:15.574208 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:15.574174    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-xmr62"
Oct 31 12:49:15.716927 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:15.716889    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-rr2hg" event={"ID":"15a47e5b-2ade-4ca3-b0d0-6e4c09917588","Type":"ContainerDied","Data":"ef5a4210fb1b20db23e6f2a9f7c16813f879a1544bf1caa39387291627fc7172"}
Oct 31 12:49:15.717398 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:15.716960    5060 scope.go:122] "RemoveContainer" containerID="2bb4f5e3c1b84da3d002959b676fa2bd230119ff2186a6007a5553bcc7cc4af6"
Oct 31 12:49:15.717398 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:15.717119    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-rr2hg"
Oct 31 12:49:15.794158 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:15.793675    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-rr2hg"]
Oct 31 12:49:15.801772 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:15.801741    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-rr2hg"]
Oct 31 12:49:15.876885 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:49:15.876853    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1e6c7f5_d872_4549_b950_1e751b6374ab.slice/crio-c488f4c2abb95ac89cdde2cb4c444ba6f592255dcea31e1d4eb8f9cebcf29959 WatchSource:0}: Error finding container c488f4c2abb95ac89cdde2cb4c444ba6f592255dcea31e1d4eb8f9cebcf29959: Status 404 returned error can't find the container with id c488f4c2abb95ac89cdde2cb4c444ba6f592255dcea31e1d4eb8f9cebcf29959
Oct 31 12:49:15.877031 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:15.876946    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-xmr62"]
Oct 31 12:49:15.880657 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:15.880579    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:49:15.880971 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:15.880947    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:49:15.881074 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:15.881005    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:49:16.735076 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:16.735039    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-xmr62" event={"ID":"e1e6c7f5-d872-4549-b950-1e751b6374ab","Type":"ContainerStarted","Data":"15eb93212f666238582935bcfb7fb66c4776d563bb22273bc4eb80596e90659b"}
Oct 31 12:49:16.735647 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:16.735098    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-xmr62" event={"ID":"e1e6c7f5-d872-4549-b950-1e751b6374ab","Type":"ContainerStarted","Data":"c488f4c2abb95ac89cdde2cb4c444ba6f592255dcea31e1d4eb8f9cebcf29959"}
Oct 31 12:49:16.777707 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:16.777646    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-xmr62" podStartSLOduration=8.777631125 podStartE2EDuration="8.777631125s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:49:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:49:16.76386483 +0000 UTC m=+1319.866932543" watchObservedRunningTime="2026-10-31 12:49:16.777631125 +0000 UTC m=+1319.880698836"
Oct 31 12:49:17.051329 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:17.051299    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15a47e5b-2ade-4ca3-b0d0-6e4c09917588" path="/var/lib/kubelet/pods/15a47e5b-2ade-4ca3-b0d0-6e4c09917588/volumes"
Oct 31 12:49:17.594732 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:17.589925    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-xmr62"
Oct 31 12:49:17.749525 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:17.736559    5060 patch_prober.go:29] interesting pod/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-xmr62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body=
Oct 31 12:49:17.749525 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:17.736641    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-xmr62" podUID="e1e6c7f5-d872-4549-b950-1e751b6374ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused"
Oct 31 12:49:18.591513 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:18.581837    5060 patch_prober.go:29] interesting pod/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-xmr62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body=
Oct 31 12:49:18.591513 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:18.581904    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-xmr62" podUID="e1e6c7f5-d872-4549-b950-1e751b6374ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused"
Oct 31 12:49:19.577515 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:19.575826    5060 patch_prober.go:29] interesting pod/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-xmr62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body=
Oct 31 12:49:19.577515 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:19.575890    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-xmr62" podUID="e1e6c7f5-d872-4549-b950-1e751b6374ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused"
Oct 31 12:49:20.575509 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:20.574880    5060 patch_prober.go:29] interesting pod/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-xmr62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body=
Oct 31 12:49:20.575509 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:20.574944    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-xmr62" podUID="e1e6c7f5-d872-4549-b950-1e751b6374ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused"
Oct 31 12:49:21.574861 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:21.574820    5060 patch_prober.go:29] interesting pod/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-xmr62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body=
Oct 31 12:49:21.575428 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:21.575394    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-xmr62" podUID="e1e6c7f5-d872-4549-b950-1e751b6374ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused"
Oct 31 12:49:22.576852 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:22.576813    5060 patch_prober.go:29] interesting pod/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-xmr62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body=
Oct 31 12:49:22.577453 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:22.577416    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-xmr62" podUID="e1e6c7f5-d872-4549-b950-1e751b6374ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused"
Oct 31 12:49:23.029222 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:23.029179    5060 scope.go:122] "RemoveContainer" containerID="4030fb42cfe547701ce89efbfd321769c56427be03b55a00a1861a6fe5e0368c"
Oct 31 12:49:23.045541 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:23.045516    5060 scope.go:122] "RemoveContainer" containerID="b48ef5839b9faff8f6c6de6f61b9c9421b1605aa356e1fe3fa9982a5cfb308fb"
Oct 31 12:49:23.108875 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:23.108848    5060 scope.go:122] "RemoveContainer" containerID="95b47a2c3acfe251a1d9c24318203c56441584e4f340befe96c8d9550a184157"
Oct 31 12:49:23.128933 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:23.128903    5060 scope.go:122] "RemoveContainer" containerID="06ced8366fdcd41e638098e6810a00b993bbbc8d565213ecd4cbe8caf635334a"
Oct 31 12:49:23.140739 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:23.140721    5060 scope.go:122] "RemoveContainer" containerID="c36b807976c8918d4b6f4efede835f597e3d6bcbf6da477aedbbd666c4aa48ed"
Oct 31 12:49:23.151602 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:23.151580    5060 scope.go:122] "RemoveContainer" containerID="259072df86166fd266c2bbe36f55175413c641c7825cd14d4aad99fb22ca7b5e"
Oct 31 12:49:23.197838 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:23.197811    5060 scope.go:122] "RemoveContainer" containerID="4f49adcc5a41287c8dc8ab5079f906b7d61a1ca0199d915da1d31e3c4424108e"
Oct 31 12:49:23.211670 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:23.211648    5060 scope.go:122] "RemoveContainer" containerID="aac66dda09e0dcdfc5516de8930a983fc1ccbf16cf7d2218e2bd210bc99a204d"
Oct 31 12:49:23.222626 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:23.222597    5060 scope.go:122] "RemoveContainer" containerID="8f78780b79621f1fbfd5d8a445fe54669d3176a2cebcb190cbd516f66ef034e9"
Oct 31 12:49:23.237168 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:23.237145    5060 scope.go:122] "RemoveContainer" containerID="28fb4ff8de3ac2703156a593eaff912df36502c48a7a3dae70c37cad22b9f927"
Oct 31 12:49:23.248956 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:23.248934    5060 scope.go:122] "RemoveContainer" containerID="fb9347a728d418c82cd8782dca3ac902277750b8975f91f859a27dff89dac0b7"
Oct 31 12:49:23.575212 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:23.575132    5060 patch_prober.go:29] interesting pod/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-xmr62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body=
Oct 31 12:49:23.575387 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:23.575208    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-xmr62" podUID="e1e6c7f5-d872-4549-b950-1e751b6374ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused"
Oct 31 12:49:24.584628 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:24.583287    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-xmr62"
Oct 31 12:49:24.584628 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:24.584225    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-xmr62"
Oct 31 12:49:24.587056 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:24.587030    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-xmr62"
Oct 31 12:49:30.057417 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:49:30.057370    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7phb" containerName="extractor"
Oct 31 12:49:35.455526 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:35.453586    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-2-w5-b25d3b-kserve-76f798d59f-vvzhg" podUID="394ed76c-84ac-4612-9acb-a7e36cb68d89" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:49:35.475532 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:35.473606    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/simulator-w4-c67de2-kserve-868679cf9b-gqjbx" podUID="e104d7da-1fb2-4c99-a480-b4ae84c4f273" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded"
Oct 31 12:49:35.475532 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:35.474211    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/premium-w2-e6c79d-kserve-6468bf46d4-h66f8" podUID="cd6a266f-8d12-4089-83ee-c75c716bf26c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Oct 31 12:49:35.567541 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:35.561432    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x2qzv" podUID="c247ed2a-aa90-424b-9b7e-4f2936d67e3f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:49:36.484625 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:36.484576    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/premium-w2-e6c79d-kserve-6468bf46d4-h66f8" podUID="cd6a266f-8d12-4089-83ee-c75c716bf26c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:49:36.517552 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:36.501706    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e6c79d/premium-w2-e6c79d-kserve-6468bf46d4-h66f8"
Oct 31 12:49:36.579962 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:36.579918    5060 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-c67de2/simulator-w4-c67de2-kserve-868679cf9b-gqjbx" podUID="e104d7da-1fb2-4c99-a480-b4ae84c4f273" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:49:36.624851 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:36.624798    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x2qzv" podUID="c247ed2a-aa90-424b-9b7e-4f2936d67e3f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:49:36.636796 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:36.636760    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x2qzv"
Oct 31 12:49:36.877622 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:36.877590    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e6c79d/premium-w2-e6c79d-kserve-6468bf46d4-h66f8"
Oct 31 12:49:36.927026 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:36.926993    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x2qzv"
Oct 31 12:49:51.003689 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:51.003636    5060 patch_prober.go:29] interesting pod/e2e-inf-a-e98df8-openshift-default-779d98479-r2jbq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.203:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 12:49:51.004640 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:51.003715    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-e98df8-openshift-default-779d98479-r2jbq" podUID="e08b08ff-4f40-474a-a103-2c499e999942" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.203:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:49:52.601299 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:52.601250    5060 patch_prober.go:29] interesting pod/e2e-inf-a-4e8eb9-openshift-default-768d574d85-vxzjg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 12:49:52.601895 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:52.601346    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-vxzjg" podUID="20fa8b33-c9e6-4fb0-9f3c-c8254dece2f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:49:53.851025 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:53.850967    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-b9brm"]
Oct 31 12:49:53.852117 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:53.852072    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-b9brm" podUID="83855e68-24ce-41e1-97be-79df68bc9505" containerName="istio-proxy" containerID="cri-o://936c8c9326eed056654b98a9642430bc812b1b735c87dc6e38b8a4dc66023643" gracePeriod=30
Oct 31 12:49:54.455681 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:54.455593    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-787316-hvr6k"]
Oct 31 12:49:54.464656 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:54.464144    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-787316-hvr6k"]
Oct 31 12:49:55.112597 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:49:55.112554    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ccb0f2b8-e593-491b-8aff-d0463a95c5b1" path="/var/lib/kubelet/pods/ccb0f2b8-e593-491b-8aff-d0463a95c5b1/volumes"
Oct 31 12:49:59.925368 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:49:59.925289    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7phb" containerName="extractor"
Oct 31 12:50:00.340847 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:00.340812    5060 generic.go:356] "Generic (PLEG): container finished" podID="83855e68-24ce-41e1-97be-79df68bc9505" containerID="936c8c9326eed056654b98a9642430bc812b1b735c87dc6e38b8a4dc66023643" exitCode=0
Oct 31 12:50:00.341162 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:00.341141    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-b9brm" event={"ID":"83855e68-24ce-41e1-97be-79df68bc9505","Type":"ContainerDied","Data":"936c8c9326eed056654b98a9642430bc812b1b735c87dc6e38b8a4dc66023643"}
Oct 31 12:50:00.965928 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:00.965890    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-b9brm"
Oct 31 12:50:01.016137 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:01.016060    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83855e68-24ce-41e1-97be-79df68bc9505-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83855e68-24ce-41e1-97be-79df68bc9505-istiod-ca-cert\") pod \"83855e68-24ce-41e1-97be-79df68bc9505\" (UID: \"83855e68-24ce-41e1-97be-79df68bc9505\") "
Oct 31 12:50:01.016379 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:01.016167    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83855e68-24ce-41e1-97be-79df68bc9505-kube-api-access-p6r5r\" (UniqueName: \"kubernetes.io/projected/83855e68-24ce-41e1-97be-79df68bc9505-kube-api-access-p6r5r\") pod \"83855e68-24ce-41e1-97be-79df68bc9505\" (UID: \"83855e68-24ce-41e1-97be-79df68bc9505\") "
Oct 31 12:50:01.017224 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:01.016741    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83855e68-24ce-41e1-97be-79df68bc9505-istio-token\" (UniqueName: \"kubernetes.io/projected/83855e68-24ce-41e1-97be-79df68bc9505-istio-token\") pod \"83855e68-24ce-41e1-97be-79df68bc9505\" (UID: \"83855e68-24ce-41e1-97be-79df68bc9505\") "
Oct 31 12:50:01.017224 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:01.017207    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/83855e68-24ce-41e1-97be-79df68bc9505-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83855e68-24ce-41e1-97be-79df68bc9505-istio-podinfo\") pod \"83855e68-24ce-41e1-97be-79df68bc9505\" (UID: \"83855e68-24ce-41e1-97be-79df68bc9505\") "
Oct 31 12:50:01.017415 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:01.017268    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83855e68-24ce-41e1-97be-79df68bc9505-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83855e68-24ce-41e1-97be-79df68bc9505-istio-envoy\") pod \"83855e68-24ce-41e1-97be-79df68bc9505\" (UID: \"83855e68-24ce-41e1-97be-79df68bc9505\") "
Oct 31 12:50:01.017415 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:01.017305    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83855e68-24ce-41e1-97be-79df68bc9505-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83855e68-24ce-41e1-97be-79df68bc9505-workload-certs\") pod \"83855e68-24ce-41e1-97be-79df68bc9505\" (UID: \"83855e68-24ce-41e1-97be-79df68bc9505\") "
Oct 31 12:50:01.017415 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:01.017356    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83855e68-24ce-41e1-97be-79df68bc9505-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83855e68-24ce-41e1-97be-79df68bc9505-credential-socket\") pod \"83855e68-24ce-41e1-97be-79df68bc9505\" (UID: \"83855e68-24ce-41e1-97be-79df68bc9505\") "
Oct 31 12:50:01.017812 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:01.017419    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83855e68-24ce-41e1-97be-79df68bc9505-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83855e68-24ce-41e1-97be-79df68bc9505-istio-data\") pod \"83855e68-24ce-41e1-97be-79df68bc9505\" (UID: \"83855e68-24ce-41e1-97be-79df68bc9505\") "
Oct 31 12:50:01.017812 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:01.017452    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83855e68-24ce-41e1-97be-79df68bc9505-istiod-ca-cert" pod "83855e68-24ce-41e1-97be-79df68bc9505" (UID: "83855e68-24ce-41e1-97be-79df68bc9505"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:50:01.018429 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:01.018066    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83855e68-24ce-41e1-97be-79df68bc9505-credential-socket" pod "83855e68-24ce-41e1-97be-79df68bc9505" (UID: "83855e68-24ce-41e1-97be-79df68bc9505"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:50:01.018429 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:01.018143    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83855e68-24ce-41e1-97be-79df68bc9505-workload-certs" pod "83855e68-24ce-41e1-97be-79df68bc9505" (UID: "83855e68-24ce-41e1-97be-79df68bc9505"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:50:01.018631 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:01.018469    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83855e68-24ce-41e1-97be-79df68bc9505-istio-data" pod "83855e68-24ce-41e1-97be-79df68bc9505" (UID: "83855e68-24ce-41e1-97be-79df68bc9505"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:50:01.018631 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:01.018602    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83855e68-24ce-41e1-97be-79df68bc9505-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83855e68-24ce-41e1-97be-79df68bc9505-workload-socket\") pod \"83855e68-24ce-41e1-97be-79df68bc9505\" (UID: \"83855e68-24ce-41e1-97be-79df68bc9505\") "
Oct 31 12:50:01.019842 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:01.019799    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83855e68-24ce-41e1-97be-79df68bc9505-workload-socket" pod "83855e68-24ce-41e1-97be-79df68bc9505" (UID: "83855e68-24ce-41e1-97be-79df68bc9505"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:50:01.023405 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:01.023374    5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83855e68-24ce-41e1-97be-79df68bc9505-istiod-ca-cert\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:50:01.023582 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:01.023414    5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83855e68-24ce-41e1-97be-79df68bc9505-workload-certs\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:50:01.023582 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:01.023430    5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83855e68-24ce-41e1-97be-79df68bc9505-credential-socket\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:50:01.023582 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:01.023447    5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83855e68-24ce-41e1-97be-79df68bc9505-istio-data\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:50:01.023582 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:01.023462    5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83855e68-24ce-41e1-97be-79df68bc9505-workload-socket\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:50:01.027451 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:01.027385    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83855e68-24ce-41e1-97be-79df68bc9505-istio-envoy" pod "83855e68-24ce-41e1-97be-79df68bc9505" (UID: "83855e68-24ce-41e1-97be-79df68bc9505"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:50:01.032065 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:01.032016    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83855e68-24ce-41e1-97be-79df68bc9505-istio-token" pod "83855e68-24ce-41e1-97be-79df68bc9505" (UID: "83855e68-24ce-41e1-97be-79df68bc9505"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:50:01.032787 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:01.032083    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83855e68-24ce-41e1-97be-79df68bc9505-kube-api-access-p6r5r" pod "83855e68-24ce-41e1-97be-79df68bc9505" (UID: "83855e68-24ce-41e1-97be-79df68bc9505"). InnerVolumeSpecName "kube-api-access-p6r5r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:50:01.034272 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:01.034232    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/83855e68-24ce-41e1-97be-79df68bc9505-istio-podinfo" pod "83855e68-24ce-41e1-97be-79df68bc9505" (UID: "83855e68-24ce-41e1-97be-79df68bc9505"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 12:50:01.068408 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:01.067478    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-2a0ea4-584584b785-hpjz7"]
Oct 31 12:50:01.069021 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:01.068995    5060 cpu_manager.go:414] "containerMap: removing container" podUID="83855e68-24ce-41e1-97be-79df68bc9505" containerName="istio-proxy"
Oct 31 12:50:01.069185 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:01.069168    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83855e68-24ce-41e1-97be-79df68bc9505" containerName="istio-proxy"
Oct 31 12:50:01.069586 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:01.069565    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="83855e68-24ce-41e1-97be-79df68bc9505" containerName="istio-proxy"
Oct 31 12:50:01.120763 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:01.120726    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-1a7d0e-69d4d6fb9f-7r75d"]
Oct 31 12:50:01.121043 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:01.120852    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-2a0ea4-584584b785-hpjz7"
Oct 31 12:50:01.127903 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:01.127873    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6r5r\" (UniqueName: \"kubernetes.io/projected/83855e68-24ce-41e1-97be-79df68bc9505-kube-api-access-p6r5r\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:50:01.128107 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:01.128088    5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83855e68-24ce-41e1-97be-79df68bc9505-istio-token\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:50:01.128233 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:01.128218    5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83855e68-24ce-41e1-97be-79df68bc9505-istio-podinfo\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:50:01.128346 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:01.128333    5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83855e68-24ce-41e1-97be-79df68bc9505-istio-envoy\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:50:01.130837 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:01.130815    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-2a0ea4\""
Oct 31 12:50:01.131185 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:01.131166    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-2a0ea4-dockercfg-fhtv8\""
Oct 31 12:50:01.176438 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:01.176380    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-1a7d0e-69d4d6fb9f-7r75d"]
Oct 31 12:50:01.176655 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:01.176449    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-2a0ea4-584584b785-hpjz7"]
Oct 31 12:50:01.176655 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:01.176582    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-1a7d0e-69d4d6fb9f-7r75d"
Oct 31 12:50:01.231584 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:01.231045    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nd58k\" (UniqueName: \"kubernetes.io/projected/218f45e9-a5c5-4a5d-b4a6-365fa52b66ba-kube-api-access-nd58k\") pod \"payload-processing-e2e-inf-b-1a7d0e-69d4d6fb9f-7r75d\" (UID: \"218f45e9-a5c5-4a5d-b4a6-365fa52b66ba\") " pod="openshift-ingress/payload-processing-e2e-inf-b-1a7d0e-69d4d6fb9f-7r75d"
Oct 31 12:50:01.231584 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:01.231108    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcbzm\" (UniqueName: \"kubernetes.io/projected/ecb5bb94-76fd-43a2-a49e-b1b0c8320cb2-kube-api-access-wcbzm\") pod \"payload-processing-e2e-inf-b-2a0ea4-584584b785-hpjz7\" (UID: \"ecb5bb94-76fd-43a2-a49e-b1b0c8320cb2\") " pod="openshift-ingress/payload-processing-e2e-inf-b-2a0ea4-584584b785-hpjz7"
Oct 31 12:50:01.231584 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:01.231235    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecb5bb94-76fd-43a2-a49e-b1b0c8320cb2-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-2a0ea4-584584b785-hpjz7\" (UID: \"ecb5bb94-76fd-43a2-a49e-b1b0c8320cb2\") " pod="openshift-ingress/payload-processing-e2e-inf-b-2a0ea4-584584b785-hpjz7"
Oct 31 12:50:01.231584 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:01.231275    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/218f45e9-a5c5-4a5d-b4a6-365fa52b66ba-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-1a7d0e-69d4d6fb9f-7r75d\" (UID: \"218f45e9-a5c5-4a5d-b4a6-365fa52b66ba\") " pod="openshift-ingress/payload-processing-e2e-inf-b-1a7d0e-69d4d6fb9f-7r75d"
Oct 31 12:50:01.235107 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:01.235060    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-1a7d0e-dockercfg-jtss9\""
Oct 31 12:50:01.235903 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:01.235883    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-1a7d0e\""
Oct 31 12:50:01.332755 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:01.332669    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcbzm\" (UniqueName: \"kubernetes.io/projected/ecb5bb94-76fd-43a2-a49e-b1b0c8320cb2-kube-api-access-wcbzm\") pod \"payload-processing-e2e-inf-b-2a0ea4-584584b785-hpjz7\" (UID: \"ecb5bb94-76fd-43a2-a49e-b1b0c8320cb2\") " pod="openshift-ingress/payload-processing-e2e-inf-b-2a0ea4-584584b785-hpjz7"
Oct 31 12:50:01.332961 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:01.332803    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecb5bb94-76fd-43a2-a49e-b1b0c8320cb2-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-2a0ea4-584584b785-hpjz7\" (UID: \"ecb5bb94-76fd-43a2-a49e-b1b0c8320cb2\") " pod="openshift-ingress/payload-processing-e2e-inf-b-2a0ea4-584584b785-hpjz7"
Oct 31 12:50:01.332961 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:01.332835    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/218f45e9-a5c5-4a5d-b4a6-365fa52b66ba-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-1a7d0e-69d4d6fb9f-7r75d\" (UID: \"218f45e9-a5c5-4a5d-b4a6-365fa52b66ba\") " pod="openshift-ingress/payload-processing-e2e-inf-b-1a7d0e-69d4d6fb9f-7r75d"
Oct 31 12:50:01.333194 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:01.333171    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nd58k\" (UniqueName: \"kubernetes.io/projected/218f45e9-a5c5-4a5d-b4a6-365fa52b66ba-kube-api-access-nd58k\") pod \"payload-processing-e2e-inf-b-1a7d0e-69d4d6fb9f-7r75d\" (UID: \"218f45e9-a5c5-4a5d-b4a6-365fa52b66ba\") " pod="openshift-ingress/payload-processing-e2e-inf-b-1a7d0e-69d4d6fb9f-7r75d"
Oct 31 12:50:01.333732 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:01.333711    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecb5bb94-76fd-43a2-a49e-b1b0c8320cb2-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-2a0ea4-584584b785-hpjz7\" (UID: \"ecb5bb94-76fd-43a2-a49e-b1b0c8320cb2\") " pod="openshift-ingress/payload-processing-e2e-inf-b-2a0ea4-584584b785-hpjz7"
Oct 31 12:50:01.333838 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:01.333742    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/218f45e9-a5c5-4a5d-b4a6-365fa52b66ba-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-1a7d0e-69d4d6fb9f-7r75d\" (UID: \"218f45e9-a5c5-4a5d-b4a6-365fa52b66ba\") " pod="openshift-ingress/payload-processing-e2e-inf-b-1a7d0e-69d4d6fb9f-7r75d"
Oct 31 12:50:01.362952 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:01.362917    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcbzm\" (UniqueName: \"kubernetes.io/projected/ecb5bb94-76fd-43a2-a49e-b1b0c8320cb2-kube-api-access-wcbzm\") pod \"payload-processing-e2e-inf-b-2a0ea4-584584b785-hpjz7\" (UID: \"ecb5bb94-76fd-43a2-a49e-b1b0c8320cb2\") " pod="openshift-ingress/payload-processing-e2e-inf-b-2a0ea4-584584b785-hpjz7"
Oct 31 12:50:01.365401 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:01.365368    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-b9brm" event={"ID":"83855e68-24ce-41e1-97be-79df68bc9505","Type":"ContainerDied","Data":"3a1859c37b15e081f389254ee8ac15727489369144b780c33abfef157c003fa8"}
Oct 31 12:50:01.365587 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:01.365439    5060 scope.go:122] "RemoveContainer" containerID="936c8c9326eed056654b98a9642430bc812b1b735c87dc6e38b8a4dc66023643"
Oct 31 12:50:01.365680 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:01.365665    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-b9brm"
Oct 31 12:50:01.370655 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:01.370630    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nd58k\" (UniqueName: \"kubernetes.io/projected/218f45e9-a5c5-4a5d-b4a6-365fa52b66ba-kube-api-access-nd58k\") pod \"payload-processing-e2e-inf-b-1a7d0e-69d4d6fb9f-7r75d\" (UID: \"218f45e9-a5c5-4a5d-b4a6-365fa52b66ba\") " pod="openshift-ingress/payload-processing-e2e-inf-b-1a7d0e-69d4d6fb9f-7r75d"
Oct 31 12:50:01.399400 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:01.399360    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-b9brm"]
Oct 31 12:50:01.407799 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:01.407767    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-b9brm"]
Oct 31 12:50:01.500517 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:01.486626    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-2a0ea4-584584b785-hpjz7"
Oct 31 12:50:01.565293 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:01.561885    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-1a7d0e-69d4d6fb9f-7r75d"
Oct 31 12:50:02.430257 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:02.430227    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-1a7d0e-69d4d6fb9f-7r75d"]
Oct 31 12:50:02.484121 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:02.484090    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-2a0ea4-584584b785-hpjz7"]
Oct 31 12:50:03.051544 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:03.051510    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83855e68-24ce-41e1-97be-79df68bc9505" path="/var/lib/kubelet/pods/83855e68-24ce-41e1-97be-79df68bc9505/volumes"
Oct 31 12:50:03.417154 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:03.417041    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-1a7d0e-69d4d6fb9f-7r75d" event={"ID":"218f45e9-a5c5-4a5d-b4a6-365fa52b66ba","Type":"ContainerStarted","Data":"a0b2c52b819b82b53933d0bdfacb36ab0288b4dad1019578aa73aec6059c43d5"}
Oct 31 12:50:03.417154 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:03.417115    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-1a7d0e-69d4d6fb9f-7r75d" event={"ID":"218f45e9-a5c5-4a5d-b4a6-365fa52b66ba","Type":"ContainerStarted","Data":"0f3624adfa8763c2d916e97c2e8a9fcf91a533ed13e9cd6e0a8cd3dbf0b85b84"}
Oct 31 12:50:03.417445 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:03.417176    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-1a7d0e-69d4d6fb9f-7r75d"
Oct 31 12:50:03.419120 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:03.419092    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-2a0ea4-584584b785-hpjz7" event={"ID":"ecb5bb94-76fd-43a2-a49e-b1b0c8320cb2","Type":"ContainerStarted","Data":"c5eb8feef072ffdf4535618d202840acdaa52297d180229a81cf5a4c19036d8f"}
Oct 31 12:50:03.419235 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:03.419146    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-2a0ea4-584584b785-hpjz7"
Oct 31 12:50:03.419235 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:03.419173    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-2a0ea4-584584b785-hpjz7" event={"ID":"ecb5bb94-76fd-43a2-a49e-b1b0c8320cb2","Type":"ContainerStarted","Data":"c120f5b32341ad44668c6b16e5cb27515a24bcf1541a4d1b1ed591840e76b2fd"}
Oct 31 12:50:03.446299 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:03.446236    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-1a7d0e-69d4d6fb9f-7r75d" podStartSLOduration=14.446218472 podStartE2EDuration="14.446218472s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:49:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:50:03.439848549 +0000 UTC m=+1366.542916267" watchObservedRunningTime="2026-10-31 12:50:03.446218472 +0000 UTC m=+1366.549286186"
Oct 31 12:50:03.465979 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:03.465920    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-2a0ea4-584584b785-hpjz7" podStartSLOduration=7.465903059 podStartE2EDuration="7.465903059s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:49:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:50:03.464885697 +0000 UTC m=+1366.567953408" watchObservedRunningTime="2026-10-31 12:50:03.465903059 +0000 UTC m=+1366.568970772"
Oct 31 12:50:05.328511 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:05.327798    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-787316-openshift-default-58b9b668c5-qpf4t"]
Oct 31 12:50:05.328511 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:05.328065    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-787316-openshift-default-58b9b668c5-qpf4t" podUID="b4e57789-6b1a-4d17-bdaf-925a3d3d0c43" containerName="istio-proxy" containerID="cri-o://09f8ef560f674ab0efd43c8d325d8b659416abfbfb18f08407c0923d4a0e71d6" gracePeriod=30
Oct 31 12:50:06.003585 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:06.001631    5060 patch_prober.go:29] interesting pod/e2e-inf-a-e98df8-openshift-default-779d98479-r2jbq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.203:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 12:50:06.003585 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:06.001695    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-e98df8-openshift-default-779d98479-r2jbq" podUID="e08b08ff-4f40-474a-a103-2c499e999942" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.203:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:50:07.604666 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:07.604619    5060 patch_prober.go:29] interesting pod/e2e-inf-a-4e8eb9-openshift-default-768d574d85-vxzjg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 12:50:07.605088 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:07.604705    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-vxzjg" podUID="20fa8b33-c9e6-4fb0-9f3c-c8254dece2f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:50:09.690741 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:09.690706    5060 generic.go:356] "Generic (PLEG): container finished" podID="20fa8b33-c9e6-4fb0-9f3c-c8254dece2f6" containerID="cec778106ab298c2c59e82c8abfe4bb666ce3948bf26f0b090cf4e8426f6b03a" exitCode=0
Oct 31 12:50:09.691349 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:09.690778    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-vxzjg" event={"ID":"20fa8b33-c9e6-4fb0-9f3c-c8254dece2f6","Type":"ContainerDied","Data":"cec778106ab298c2c59e82c8abfe4bb666ce3948bf26f0b090cf4e8426f6b03a"}
Oct 31 12:50:09.691433 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:09.691364    5060 scope.go:122] "RemoveContainer" containerID="cec778106ab298c2c59e82c8abfe4bb666ce3948bf26f0b090cf4e8426f6b03a"
Oct 31 12:50:09.692856 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:09.692834    5060 generic.go:356] "Generic (PLEG): container finished" podID="e08b08ff-4f40-474a-a103-2c499e999942" containerID="b58adec0fddb21aa3934309a7e11d9cd4382f30d2afafd76ca744f39eef3cd01" exitCode=0
Oct 31 12:50:09.692976 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:09.692906    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e98df8-openshift-default-779d98479-r2jbq" event={"ID":"e08b08ff-4f40-474a-a103-2c499e999942","Type":"ContainerDied","Data":"b58adec0fddb21aa3934309a7e11d9cd4382f30d2afafd76ca744f39eef3cd01"}
Oct 31 12:50:09.693410 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:09.693393    5060 scope.go:122] "RemoveContainer" containerID="b58adec0fddb21aa3934309a7e11d9cd4382f30d2afafd76ca744f39eef3cd01"
Oct 31 12:50:09.693735 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:09.693703    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:50:09.693809 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:09.693792    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:50:09.693860 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:09.693845    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:50:09.695373 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:09.695345    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:50:09.695474 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:09.695404    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:50:09.695474 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:09.695435    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:50:10.847198 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:10.847151    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-vxzjg" event={"ID":"20fa8b33-c9e6-4fb0-9f3c-c8254dece2f6","Type":"ContainerStarted","Data":"6c60c3bdad21a71ead4fe8545b4a414aae5df9b36a2afad4d6cc8b5c09170636"}
Oct 31 12:50:10.850138 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:10.850106    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e98df8-openshift-default-779d98479-r2jbq" event={"ID":"e08b08ff-4f40-474a-a103-2c499e999942","Type":"ContainerStarted","Data":"69e50444a6492290d3c0f1f8e259cce1808ee453214910de09e68331867cc777"}
Oct 31 12:50:10.853101 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:10.853074    5060 generic.go:356] "Generic (PLEG): container finished" podID="b4e57789-6b1a-4d17-bdaf-925a3d3d0c43" containerID="09f8ef560f674ab0efd43c8d325d8b659416abfbfb18f08407c0923d4a0e71d6" exitCode=0
Oct 31 12:50:10.853246 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:10.853173    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-787316-openshift-default-58b9b668c5-qpf4t" event={"ID":"b4e57789-6b1a-4d17-bdaf-925a3d3d0c43","Type":"ContainerDied","Data":"09f8ef560f674ab0efd43c8d325d8b659416abfbfb18f08407c0923d4a0e71d6"}
Oct 31 12:50:10.853246 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:10.853218    5060 scope.go:122] "RemoveContainer" containerID="a4d096354cb95f71298f5e7a682f3e560e1fa6bb66078dbb6336fb4812ff473d"
Oct 31 12:50:11.248588 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:11.248560    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-e98df8-openshift-default-779d98479-r2jbq"
Oct 31 12:50:11.248798 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:11.248774    5060 patch_prober.go:29] interesting pod/e2e-inf-a-e98df8-openshift-default-779d98479-r2jbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" start-of-body=
Oct 31 12:50:11.248872 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:11.248848    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e98df8-openshift-default-779d98479-r2jbq" podUID="e08b08ff-4f40-474a-a103-2c499e999942" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused"
Oct 31 12:50:11.325441 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:11.325408    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-787316-openshift-default-58b9b668c5-qpf4t"
Oct 31 12:50:11.410515 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:11.400092    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-vxzjg"
Oct 31 12:50:11.410515 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:11.400209    5060 patch_prober.go:29] interesting pod/e2e-inf-a-4e8eb9-openshift-default-768d574d85-vxzjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" start-of-body=
Oct 31 12:50:11.410515 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:11.400251    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-vxzjg" podUID="20fa8b33-c9e6-4fb0-9f3c-c8254dece2f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused"
Oct 31 12:50:11.444098 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:11.442152    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4e57789-6b1a-4d17-bdaf-925a3d3d0c43-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4e57789-6b1a-4d17-bdaf-925a3d3d0c43-workload-socket\") pod \"b4e57789-6b1a-4d17-bdaf-925a3d3d0c43\" (UID: \"b4e57789-6b1a-4d17-bdaf-925a3d3d0c43\") "
Oct 31 12:50:11.444098 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:11.442508    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4e57789-6b1a-4d17-bdaf-925a3d3d0c43-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4e57789-6b1a-4d17-bdaf-925a3d3d0c43-istiod-ca-cert\") pod \"b4e57789-6b1a-4d17-bdaf-925a3d3d0c43\" (UID: \"b4e57789-6b1a-4d17-bdaf-925a3d3d0c43\") "
Oct 31 12:50:11.444098 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:11.442562    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4e57789-6b1a-4d17-bdaf-925a3d3d0c43-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4e57789-6b1a-4d17-bdaf-925a3d3d0c43-credential-socket\") pod \"b4e57789-6b1a-4d17-bdaf-925a3d3d0c43\" (UID: \"b4e57789-6b1a-4d17-bdaf-925a3d3d0c43\") "
Oct 31 12:50:11.444098 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:11.442589    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4e57789-6b1a-4d17-bdaf-925a3d3d0c43-istio-token\" (UniqueName: \"kubernetes.io/projected/b4e57789-6b1a-4d17-bdaf-925a3d3d0c43-istio-token\") pod \"b4e57789-6b1a-4d17-bdaf-925a3d3d0c43\" (UID: \"b4e57789-6b1a-4d17-bdaf-925a3d3d0c43\") "
Oct 31 12:50:11.444098 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:11.442679    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4e57789-6b1a-4d17-bdaf-925a3d3d0c43-kube-api-access-5z4hd\" (UniqueName: \"kubernetes.io/projected/b4e57789-6b1a-4d17-bdaf-925a3d3d0c43-kube-api-access-5z4hd\") pod \"b4e57789-6b1a-4d17-bdaf-925a3d3d0c43\" (UID: \"b4e57789-6b1a-4d17-bdaf-925a3d3d0c43\") "
Oct 31 12:50:11.444098 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:11.442711    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4e57789-6b1a-4d17-bdaf-925a3d3d0c43-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4e57789-6b1a-4d17-bdaf-925a3d3d0c43-istio-data\") pod \"b4e57789-6b1a-4d17-bdaf-925a3d3d0c43\" (UID: \"b4e57789-6b1a-4d17-bdaf-925a3d3d0c43\") "
Oct 31 12:50:11.444098 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:11.442799    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b4e57789-6b1a-4d17-bdaf-925a3d3d0c43-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4e57789-6b1a-4d17-bdaf-925a3d3d0c43-istio-podinfo\") pod \"b4e57789-6b1a-4d17-bdaf-925a3d3d0c43\" (UID: \"b4e57789-6b1a-4d17-bdaf-925a3d3d0c43\") "
Oct 31 12:50:11.444098 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:11.442867    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4e57789-6b1a-4d17-bdaf-925a3d3d0c43-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4e57789-6b1a-4d17-bdaf-925a3d3d0c43-istio-envoy\") pod \"b4e57789-6b1a-4d17-bdaf-925a3d3d0c43\" (UID: \"b4e57789-6b1a-4d17-bdaf-925a3d3d0c43\") "
Oct 31 12:50:11.444098 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:11.442905    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4e57789-6b1a-4d17-bdaf-925a3d3d0c43-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4e57789-6b1a-4d17-bdaf-925a3d3d0c43-workload-certs\") pod \"b4e57789-6b1a-4d17-bdaf-925a3d3d0c43\" (UID: \"b4e57789-6b1a-4d17-bdaf-925a3d3d0c43\") "
Oct 31 12:50:11.465509 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:11.460905    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4e57789-6b1a-4d17-bdaf-925a3d3d0c43-workload-socket" pod "b4e57789-6b1a-4d17-bdaf-925a3d3d0c43" (UID: "b4e57789-6b1a-4d17-bdaf-925a3d3d0c43"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:50:11.465509 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:11.461404    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4e57789-6b1a-4d17-bdaf-925a3d3d0c43-istiod-ca-cert" pod "b4e57789-6b1a-4d17-bdaf-925a3d3d0c43" (UID: "b4e57789-6b1a-4d17-bdaf-925a3d3d0c43"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:50:11.465509 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:11.464639    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4e57789-6b1a-4d17-bdaf-925a3d3d0c43-credential-socket" pod "b4e57789-6b1a-4d17-bdaf-925a3d3d0c43" (UID: "b4e57789-6b1a-4d17-bdaf-925a3d3d0c43"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:50:11.504564 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:11.504204    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b4e57789-6b1a-4d17-bdaf-925a3d3d0c43-istio-podinfo" pod "b4e57789-6b1a-4d17-bdaf-925a3d3d0c43" (UID: "b4e57789-6b1a-4d17-bdaf-925a3d3d0c43"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 12:50:11.527751 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:11.511634    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4e57789-6b1a-4d17-bdaf-925a3d3d0c43-istio-data" pod "b4e57789-6b1a-4d17-bdaf-925a3d3d0c43" (UID: "b4e57789-6b1a-4d17-bdaf-925a3d3d0c43"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:50:11.543004 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:11.529978    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4e57789-6b1a-4d17-bdaf-925a3d3d0c43-workload-certs" pod "b4e57789-6b1a-4d17-bdaf-925a3d3d0c43" (UID: "b4e57789-6b1a-4d17-bdaf-925a3d3d0c43"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:50:11.603512 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:11.584722    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4e57789-6b1a-4d17-bdaf-925a3d3d0c43-istio-token" pod "b4e57789-6b1a-4d17-bdaf-925a3d3d0c43" (UID: "b4e57789-6b1a-4d17-bdaf-925a3d3d0c43"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:50:11.603512 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:11.586736    5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4e57789-6b1a-4d17-bdaf-925a3d3d0c43-workload-socket\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:50:11.603512 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:11.586767    5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4e57789-6b1a-4d17-bdaf-925a3d3d0c43-istiod-ca-cert\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:50:11.603512 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:11.586784    5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4e57789-6b1a-4d17-bdaf-925a3d3d0c43-credential-socket\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:50:11.603512 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:11.586798    5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4e57789-6b1a-4d17-bdaf-925a3d3d0c43-istio-token\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:50:11.603512 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:11.586812    5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4e57789-6b1a-4d17-bdaf-925a3d3d0c43-istio-data\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:50:11.603512 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:11.586826    5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4e57789-6b1a-4d17-bdaf-925a3d3d0c43-istio-podinfo\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:50:11.603512 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:11.586840    5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4e57789-6b1a-4d17-bdaf-925a3d3d0c43-workload-certs\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:50:11.603512 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:11.587424    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4e57789-6b1a-4d17-bdaf-925a3d3d0c43-istio-envoy" pod "b4e57789-6b1a-4d17-bdaf-925a3d3d0c43" (UID: "b4e57789-6b1a-4d17-bdaf-925a3d3d0c43"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:50:11.625301 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:11.624750    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-015701-openshift-default-86f6dfd4b7-5lsm7"]
Oct 31 12:50:11.627528 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:11.626453    5060 cpu_manager.go:414] "containerMap: removing container" podUID="b4e57789-6b1a-4d17-bdaf-925a3d3d0c43" containerName="istio-proxy"
Oct 31 12:50:11.627528 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:11.626482    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4e57789-6b1a-4d17-bdaf-925a3d3d0c43" containerName="istio-proxy"
Oct 31 12:50:11.627528 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:11.626574    5060 cpu_manager.go:414] "containerMap: removing container" podUID="b4e57789-6b1a-4d17-bdaf-925a3d3d0c43" containerName="istio-proxy"
Oct 31 12:50:11.627528 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:11.626584    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4e57789-6b1a-4d17-bdaf-925a3d3d0c43" containerName="istio-proxy"
Oct 31 12:50:11.627528 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:11.626769    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4e57789-6b1a-4d17-bdaf-925a3d3d0c43" containerName="istio-proxy"
Oct 31 12:50:11.632365 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:11.632343    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-015701-openshift-default-86f6dfd4b7-5lsm7"
Oct 31 12:50:11.640698 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:11.639966    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-015701-openshift-default-dockercfg-prb4x\""
Oct 31 12:50:11.694126 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:11.694091    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-015701-openshift-default-86f6dfd4b7-5lsm7"]
Oct 31 12:50:11.694731 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:11.694695    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce0f9360-f1f6-462c-b4b1-1e0a2232fa7d-istio-podinfo\") pod \"e2e-inf-a-015701-openshift-default-86f6dfd4b7-5lsm7\" (UID: \"ce0f9360-f1f6-462c-b4b1-1e0a2232fa7d\") " pod="openshift-ingress/e2e-inf-a-015701-openshift-default-86f6dfd4b7-5lsm7"
Oct 31 12:50:11.694861 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:11.694742    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce0f9360-f1f6-462c-b4b1-1e0a2232fa7d-istiod-ca-cert\") pod \"e2e-inf-a-015701-openshift-default-86f6dfd4b7-5lsm7\" (UID: \"ce0f9360-f1f6-462c-b4b1-1e0a2232fa7d\") " pod="openshift-ingress/e2e-inf-a-015701-openshift-default-86f6dfd4b7-5lsm7"
Oct 31 12:50:11.694923 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:11.694872    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2w6b\" (UniqueName: \"kubernetes.io/projected/ce0f9360-f1f6-462c-b4b1-1e0a2232fa7d-kube-api-access-s2w6b\") pod \"e2e-inf-a-015701-openshift-default-86f6dfd4b7-5lsm7\" (UID: \"ce0f9360-f1f6-462c-b4b1-1e0a2232fa7d\") " pod="openshift-ingress/e2e-inf-a-015701-openshift-default-86f6dfd4b7-5lsm7"
Oct 31 12:50:11.694981 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:11.694939    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce0f9360-f1f6-462c-b4b1-1e0a2232fa7d-workload-socket\") pod \"e2e-inf-a-015701-openshift-default-86f6dfd4b7-5lsm7\" (UID: \"ce0f9360-f1f6-462c-b4b1-1e0a2232fa7d\") " pod="openshift-ingress/e2e-inf-a-015701-openshift-default-86f6dfd4b7-5lsm7"
Oct 31 12:50:11.695038 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:11.694995    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce0f9360-f1f6-462c-b4b1-1e0a2232fa7d-credential-socket\") pod \"e2e-inf-a-015701-openshift-default-86f6dfd4b7-5lsm7\" (UID: \"ce0f9360-f1f6-462c-b4b1-1e0a2232fa7d\") " pod="openshift-ingress/e2e-inf-a-015701-openshift-default-86f6dfd4b7-5lsm7"
Oct 31 12:50:11.695173 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:11.695155    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce0f9360-f1f6-462c-b4b1-1e0a2232fa7d-istio-token\") pod \"e2e-inf-a-015701-openshift-default-86f6dfd4b7-5lsm7\" (UID: \"ce0f9360-f1f6-462c-b4b1-1e0a2232fa7d\") " pod="openshift-ingress/e2e-inf-a-015701-openshift-default-86f6dfd4b7-5lsm7"
Oct 31 12:50:11.695500 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:11.695460    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce0f9360-f1f6-462c-b4b1-1e0a2232fa7d-istio-data\") pod \"e2e-inf-a-015701-openshift-default-86f6dfd4b7-5lsm7\" (UID: \"ce0f9360-f1f6-462c-b4b1-1e0a2232fa7d\") " pod="openshift-ingress/e2e-inf-a-015701-openshift-default-86f6dfd4b7-5lsm7"
Oct 31 12:50:11.695732 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:11.695694    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce0f9360-f1f6-462c-b4b1-1e0a2232fa7d-workload-certs\") pod \"e2e-inf-a-015701-openshift-default-86f6dfd4b7-5lsm7\" (UID: \"ce0f9360-f1f6-462c-b4b1-1e0a2232fa7d\") " pod="openshift-ingress/e2e-inf-a-015701-openshift-default-86f6dfd4b7-5lsm7"
Oct 31 12:50:11.695904 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:11.695882    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce0f9360-f1f6-462c-b4b1-1e0a2232fa7d-istio-envoy\") pod \"e2e-inf-a-015701-openshift-default-86f6dfd4b7-5lsm7\" (UID: \"ce0f9360-f1f6-462c-b4b1-1e0a2232fa7d\") " pod="openshift-ingress/e2e-inf-a-015701-openshift-default-86f6dfd4b7-5lsm7"
Oct 31 12:50:11.696116 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:11.696097    5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4e57789-6b1a-4d17-bdaf-925a3d3d0c43-istio-envoy\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:50:11.715511 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:11.711894    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4e57789-6b1a-4d17-bdaf-925a3d3d0c43-kube-api-access-5z4hd" pod "b4e57789-6b1a-4d17-bdaf-925a3d3d0c43" (UID: "b4e57789-6b1a-4d17-bdaf-925a3d3d0c43"). InnerVolumeSpecName "kube-api-access-5z4hd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:50:11.812537 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:11.805180    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce0f9360-f1f6-462c-b4b1-1e0a2232fa7d-istio-token\") pod \"e2e-inf-a-015701-openshift-default-86f6dfd4b7-5lsm7\" (UID: \"ce0f9360-f1f6-462c-b4b1-1e0a2232fa7d\") " pod="openshift-ingress/e2e-inf-a-015701-openshift-default-86f6dfd4b7-5lsm7"
Oct 31 12:50:11.812791 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:11.812746    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce0f9360-f1f6-462c-b4b1-1e0a2232fa7d-istio-data\") pod \"e2e-inf-a-015701-openshift-default-86f6dfd4b7-5lsm7\" (UID: \"ce0f9360-f1f6-462c-b4b1-1e0a2232fa7d\") " pod="openshift-ingress/e2e-inf-a-015701-openshift-default-86f6dfd4b7-5lsm7"
Oct 31 12:50:11.812878 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:11.812861    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce0f9360-f1f6-462c-b4b1-1e0a2232fa7d-workload-certs\") pod \"e2e-inf-a-015701-openshift-default-86f6dfd4b7-5lsm7\" (UID: \"ce0f9360-f1f6-462c-b4b1-1e0a2232fa7d\") " pod="openshift-ingress/e2e-inf-a-015701-openshift-default-86f6dfd4b7-5lsm7"
Oct 31 12:50:11.812942 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:11.812919    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce0f9360-f1f6-462c-b4b1-1e0a2232fa7d-istio-envoy\") pod \"e2e-inf-a-015701-openshift-default-86f6dfd4b7-5lsm7\" (UID: \"ce0f9360-f1f6-462c-b4b1-1e0a2232fa7d\") " pod="openshift-ingress/e2e-inf-a-015701-openshift-default-86f6dfd4b7-5lsm7"
Oct 31 12:50:11.812991 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:11.812963    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce0f9360-f1f6-462c-b4b1-1e0a2232fa7d-istio-podinfo\") pod \"e2e-inf-a-015701-openshift-default-86f6dfd4b7-5lsm7\" (UID: \"ce0f9360-f1f6-462c-b4b1-1e0a2232fa7d\") " pod="openshift-ingress/e2e-inf-a-015701-openshift-default-86f6dfd4b7-5lsm7"
Oct 31 12:50:11.813041 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:11.812991    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce0f9360-f1f6-462c-b4b1-1e0a2232fa7d-istiod-ca-cert\") pod \"e2e-inf-a-015701-openshift-default-86f6dfd4b7-5lsm7\" (UID: \"ce0f9360-f1f6-462c-b4b1-1e0a2232fa7d\") " pod="openshift-ingress/e2e-inf-a-015701-openshift-default-86f6dfd4b7-5lsm7"
Oct 31 12:50:11.813153 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:11.813136    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2w6b\" (UniqueName: \"kubernetes.io/projected/ce0f9360-f1f6-462c-b4b1-1e0a2232fa7d-kube-api-access-s2w6b\") pod \"e2e-inf-a-015701-openshift-default-86f6dfd4b7-5lsm7\" (UID: \"ce0f9360-f1f6-462c-b4b1-1e0a2232fa7d\") " pod="openshift-ingress/e2e-inf-a-015701-openshift-default-86f6dfd4b7-5lsm7"
Oct 31 12:50:11.813223 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:11.813206    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce0f9360-f1f6-462c-b4b1-1e0a2232fa7d-workload-socket\") pod \"e2e-inf-a-015701-openshift-default-86f6dfd4b7-5lsm7\" (UID: \"ce0f9360-f1f6-462c-b4b1-1e0a2232fa7d\") " pod="openshift-ingress/e2e-inf-a-015701-openshift-default-86f6dfd4b7-5lsm7"
Oct 31 12:50:11.813290 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:11.813272    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce0f9360-f1f6-462c-b4b1-1e0a2232fa7d-credential-socket\") pod \"e2e-inf-a-015701-openshift-default-86f6dfd4b7-5lsm7\" (UID: \"ce0f9360-f1f6-462c-b4b1-1e0a2232fa7d\") " pod="openshift-ingress/e2e-inf-a-015701-openshift-default-86f6dfd4b7-5lsm7"
Oct 31 12:50:11.813430 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:11.813414    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5z4hd\" (UniqueName: \"kubernetes.io/projected/b4e57789-6b1a-4d17-bdaf-925a3d3d0c43-kube-api-access-5z4hd\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:50:11.813545 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:11.813461    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce0f9360-f1f6-462c-b4b1-1e0a2232fa7d-workload-certs\") pod \"e2e-inf-a-015701-openshift-default-86f6dfd4b7-5lsm7\" (UID: \"ce0f9360-f1f6-462c-b4b1-1e0a2232fa7d\") " pod="openshift-ingress/e2e-inf-a-015701-openshift-default-86f6dfd4b7-5lsm7"
Oct 31 12:50:11.813803 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:11.813780    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce0f9360-f1f6-462c-b4b1-1e0a2232fa7d-credential-socket\") pod \"e2e-inf-a-015701-openshift-default-86f6dfd4b7-5lsm7\" (UID: \"ce0f9360-f1f6-462c-b4b1-1e0a2232fa7d\") " pod="openshift-ingress/e2e-inf-a-015701-openshift-default-86f6dfd4b7-5lsm7"
Oct 31 12:50:11.814274 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:11.814250    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce0f9360-f1f6-462c-b4b1-1e0a2232fa7d-istiod-ca-cert\") pod \"e2e-inf-a-015701-openshift-default-86f6dfd4b7-5lsm7\" (UID: \"ce0f9360-f1f6-462c-b4b1-1e0a2232fa7d\") " pod="openshift-ingress/e2e-inf-a-015701-openshift-default-86f6dfd4b7-5lsm7"
Oct 31 12:50:11.815071 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:11.815045    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce0f9360-f1f6-462c-b4b1-1e0a2232fa7d-workload-socket\") pod \"e2e-inf-a-015701-openshift-default-86f6dfd4b7-5lsm7\" (UID: \"ce0f9360-f1f6-462c-b4b1-1e0a2232fa7d\") " pod="openshift-ingress/e2e-inf-a-015701-openshift-default-86f6dfd4b7-5lsm7"
Oct 31 12:50:11.815226 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:11.815069    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce0f9360-f1f6-462c-b4b1-1e0a2232fa7d-istio-data\") pod \"e2e-inf-a-015701-openshift-default-86f6dfd4b7-5lsm7\" (UID: \"ce0f9360-f1f6-462c-b4b1-1e0a2232fa7d\") " pod="openshift-ingress/e2e-inf-a-015701-openshift-default-86f6dfd4b7-5lsm7"
Oct 31 12:50:11.818033 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:11.817997    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce0f9360-f1f6-462c-b4b1-1e0a2232fa7d-istio-podinfo\") pod \"e2e-inf-a-015701-openshift-default-86f6dfd4b7-5lsm7\" (UID: \"ce0f9360-f1f6-462c-b4b1-1e0a2232fa7d\") " pod="openshift-ingress/e2e-inf-a-015701-openshift-default-86f6dfd4b7-5lsm7"
Oct 31 12:50:11.818131 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:11.818116    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce0f9360-f1f6-462c-b4b1-1e0a2232fa7d-istio-envoy\") pod \"e2e-inf-a-015701-openshift-default-86f6dfd4b7-5lsm7\" (UID: \"ce0f9360-f1f6-462c-b4b1-1e0a2232fa7d\") " pod="openshift-ingress/e2e-inf-a-015701-openshift-default-86f6dfd4b7-5lsm7"
Oct 31 12:50:11.831136 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:11.831107    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce0f9360-f1f6-462c-b4b1-1e0a2232fa7d-istio-token\") pod \"e2e-inf-a-015701-openshift-default-86f6dfd4b7-5lsm7\" (UID: \"ce0f9360-f1f6-462c-b4b1-1e0a2232fa7d\") " pod="openshift-ingress/e2e-inf-a-015701-openshift-default-86f6dfd4b7-5lsm7"
Oct 31 12:50:11.843051 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:11.843025    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2w6b\" (UniqueName: \"kubernetes.io/projected/ce0f9360-f1f6-462c-b4b1-1e0a2232fa7d-kube-api-access-s2w6b\") pod \"e2e-inf-a-015701-openshift-default-86f6dfd4b7-5lsm7\" (UID: \"ce0f9360-f1f6-462c-b4b1-1e0a2232fa7d\") " pod="openshift-ingress/e2e-inf-a-015701-openshift-default-86f6dfd4b7-5lsm7"
Oct 31 12:50:11.925765 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:11.925719    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-787316-openshift-default-58b9b668c5-qpf4t" event={"ID":"b4e57789-6b1a-4d17-bdaf-925a3d3d0c43","Type":"ContainerDied","Data":"825945ecadf2fb01b08ecd079c5842bc1db0628fea8426e4dd4a234a5eb66a4c"}
Oct 31 12:50:11.926154 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:11.925786    5060 scope.go:122] "RemoveContainer" containerID="09f8ef560f674ab0efd43c8d325d8b659416abfbfb18f08407c0923d4a0e71d6"
Oct 31 12:50:11.926154 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:11.925893    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-787316-openshift-default-58b9b668c5-qpf4t"
Oct 31 12:50:12.002276 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:11.999813    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-015701-openshift-default-86f6dfd4b7-5lsm7"
Oct 31 12:50:12.025466 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:12.025419    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-787316-openshift-default-58b9b668c5-qpf4t"]
Oct 31 12:50:12.043993 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:12.043966    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-787316-openshift-default-58b9b668c5-qpf4t"]
Oct 31 12:50:12.249085 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:12.249058    5060 patch_prober.go:29] interesting pod/e2e-inf-a-e98df8-openshift-default-779d98479-r2jbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" start-of-body=
Oct 31 12:50:12.249314 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:12.249294    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e98df8-openshift-default-779d98479-r2jbq" podUID="e08b08ff-4f40-474a-a103-2c499e999942" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused"
Oct 31 12:50:12.418321 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:12.418085    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-015701-openshift-default-86f6dfd4b7-5lsm7"]
Oct 31 12:50:12.419006 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:12.418977    5060 patch_prober.go:29] interesting pod/e2e-inf-a-4e8eb9-openshift-default-768d574d85-vxzjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" start-of-body=
Oct 31 12:50:12.419353 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:12.419183    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-vxzjg" podUID="20fa8b33-c9e6-4fb0-9f3c-c8254dece2f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused"
Oct 31 12:50:12.423175 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:50:12.423142    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce0f9360_f1f6_462c_b4b1_1e0a2232fa7d.slice/crio-454e7423ea7590145bc4e0ebc6d8da13fb4b9e1e7b00ea85a683808ff2d7eb42 WatchSource:0}: Error finding container 454e7423ea7590145bc4e0ebc6d8da13fb4b9e1e7b00ea85a683808ff2d7eb42: Status 404 returned error can't find the container with id 454e7423ea7590145bc4e0ebc6d8da13fb4b9e1e7b00ea85a683808ff2d7eb42
Oct 31 12:50:12.426396 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:12.426362    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:50:12.426602 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:12.426574    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:50:12.426722 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:12.426709    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:50:12.951623 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:12.951484    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-015701-openshift-default-86f6dfd4b7-5lsm7" event={"ID":"ce0f9360-f1f6-462c-b4b1-1e0a2232fa7d","Type":"ContainerStarted","Data":"b4fafc52af107f28606032b647220c133808eda2b08c36ce98b6cdf4487a3fe1"}
Oct 31 12:50:12.952098 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:12.951634    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-015701-openshift-default-86f6dfd4b7-5lsm7" event={"ID":"ce0f9360-f1f6-462c-b4b1-1e0a2232fa7d","Type":"ContainerStarted","Data":"454e7423ea7590145bc4e0ebc6d8da13fb4b9e1e7b00ea85a683808ff2d7eb42"}
Oct 31 12:50:13.012842 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:13.012752    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-015701-openshift-default-86f6dfd4b7-5lsm7"
Oct 31 12:50:13.014314 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:13.014283    5060 patch_prober.go:29] interesting pod/e2e-inf-a-015701-openshift-default-86f6dfd4b7-5lsm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body=
Oct 31 12:50:13.014595 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:13.014563    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-015701-openshift-default-86f6dfd4b7-5lsm7" podUID="ce0f9360-f1f6-462c-b4b1-1e0a2232fa7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused"
Oct 31 12:50:13.026058 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:13.025996    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-015701-openshift-default-86f6dfd4b7-5lsm7" podStartSLOduration=13.025978555 podStartE2EDuration="13.025978555s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:50:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:50:12.990372666 +0000 UTC m=+1376.093440389" watchObservedRunningTime="2026-10-31 12:50:13.025978555 +0000 UTC m=+1376.129046363"
Oct 31 12:50:13.068519 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:13.055143    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4e57789-6b1a-4d17-bdaf-925a3d3d0c43" path="/var/lib/kubelet/pods/b4e57789-6b1a-4d17-bdaf-925a3d3d0c43/volumes"
Oct 31 12:50:13.248648 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:13.248567    5060 patch_prober.go:29] interesting pod/e2e-inf-a-e98df8-openshift-default-779d98479-r2jbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" start-of-body=
Oct 31 12:50:13.248648 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:13.248628    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e98df8-openshift-default-779d98479-r2jbq" podUID="e08b08ff-4f40-474a-a103-2c499e999942" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused"
Oct 31 12:50:13.398414 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:13.398376    5060 patch_prober.go:29] interesting pod/e2e-inf-a-4e8eb9-openshift-default-768d574d85-vxzjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" start-of-body=
Oct 31 12:50:13.399425 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:13.398793    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-vxzjg" podUID="20fa8b33-c9e6-4fb0-9f3c-c8254dece2f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused"
Oct 31 12:50:14.041514 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:14.033831    5060 patch_prober.go:29] interesting pod/e2e-inf-a-015701-openshift-default-86f6dfd4b7-5lsm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body=
Oct 31 12:50:14.041514 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:14.033892    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-015701-openshift-default-86f6dfd4b7-5lsm7" podUID="ce0f9360-f1f6-462c-b4b1-1e0a2232fa7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused"
Oct 31 12:50:14.252522 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:14.252273    5060 patch_prober.go:29] interesting pod/e2e-inf-a-e98df8-openshift-default-779d98479-r2jbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" start-of-body=
Oct 31 12:50:14.252522 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:14.252338    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e98df8-openshift-default-779d98479-r2jbq" podUID="e08b08ff-4f40-474a-a103-2c499e999942" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused"
Oct 31 12:50:14.415577 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:14.414610    5060 patch_prober.go:29] interesting pod/e2e-inf-a-4e8eb9-openshift-default-768d574d85-vxzjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" start-of-body=
Oct 31 12:50:14.415577 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:14.414681    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-vxzjg" podUID="20fa8b33-c9e6-4fb0-9f3c-c8254dece2f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused"
Oct 31 12:50:14.468517 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:14.466622    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-1a7d0e-69d4d6fb9f-7r75d"
Oct 31 12:50:14.501518 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:14.484709    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-2a0ea4-584584b785-hpjz7"
Oct 31 12:50:15.015515 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:15.014615    5060 patch_prober.go:29] interesting pod/e2e-inf-a-015701-openshift-default-86f6dfd4b7-5lsm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body=
Oct 31 12:50:15.015515 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:15.014685    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-015701-openshift-default-86f6dfd4b7-5lsm7" podUID="ce0f9360-f1f6-462c-b4b1-1e0a2232fa7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused"
Oct 31 12:50:15.259514 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:15.257929    5060 patch_prober.go:29] interesting pod/e2e-inf-a-e98df8-openshift-default-779d98479-r2jbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" start-of-body=
Oct 31 12:50:15.259514 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:15.257994    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e98df8-openshift-default-779d98479-r2jbq" podUID="e08b08ff-4f40-474a-a103-2c499e999942" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused"
Oct 31 12:50:15.427518 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:15.427436    5060 patch_prober.go:29] interesting pod/e2e-inf-a-4e8eb9-openshift-default-768d574d85-vxzjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" start-of-body=
Oct 31 12:50:15.427738 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:15.427516    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-vxzjg" podUID="20fa8b33-c9e6-4fb0-9f3c-c8254dece2f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused"
Oct 31 12:50:15.468518 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:15.467555    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-2-w5-b25d3b-kserve-76f798d59f-vvzhg" podUID="394ed76c-84ac-4612-9acb-a7e36cb68d89" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:50:16.016526 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:16.015509    5060 patch_prober.go:29] interesting pod/e2e-inf-a-015701-openshift-default-86f6dfd4b7-5lsm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body=
Oct 31 12:50:16.016526 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:16.015577    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-015701-openshift-default-86f6dfd4b7-5lsm7" podUID="ce0f9360-f1f6-462c-b4b1-1e0a2232fa7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused"
Oct 31 12:50:16.157513 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:16.157332    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b8ceeeb-9sbgk"]
Oct 31 12:50:16.284601 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:16.259251    5060 patch_prober.go:29] interesting pod/e2e-inf-a-e98df8-openshift-default-779d98479-r2jbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" start-of-body=
Oct 31 12:50:16.284601 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:16.259311    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e98df8-openshift-default-779d98479-r2jbq" podUID="e08b08ff-4f40-474a-a103-2c499e999942" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused"
Oct 31 12:50:16.325734 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:16.287397    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b8ceeeb-9sbgk"]
Oct 31 12:50:16.440520 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:16.424232    5060 patch_prober.go:29] interesting pod/e2e-inf-a-4e8eb9-openshift-default-768d574d85-vxzjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" start-of-body=
Oct 31 12:50:16.440520 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:16.425395    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-vxzjg" podUID="20fa8b33-c9e6-4fb0-9f3c-c8254dece2f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused"
Oct 31 12:50:16.482689 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:16.482633    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-2-w5-b25d3b-kserve-76f798d59f-vvzhg" podUID="394ed76c-84ac-4612-9acb-a7e36cb68d89" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:50:16.483531 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:16.483480    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-2-w5-b25d3b-kserve-76f798d59f-vvzhg"
Oct 31 12:50:16.517657 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:16.517616    5060 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-2-w5-b25d3b-kserve-76f798d59f-vvzhg" podUID="394ed76c-84ac-4612-9acb-a7e36cb68d89" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:50:16.541367 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:16.541283    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-2-w5-b25d3b-kserve-76f798d59f-vvzhg"
Oct 31 12:50:17.043607 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:17.029472    5060 patch_prober.go:29] interesting pod/e2e-inf-a-015701-openshift-default-86f6dfd4b7-5lsm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body=
Oct 31 12:50:17.043607 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:17.029718    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-015701-openshift-default-86f6dfd4b7-5lsm7" podUID="ce0f9360-f1f6-462c-b4b1-1e0a2232fa7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused"
Oct 31 12:50:17.103879 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:17.103828    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd073675-8da6-4d49-bf2d-bf6fd1340b51" path="/var/lib/kubelet/pods/cd073675-8da6-4d49-bf2d-bf6fd1340b51/volumes"
Oct 31 12:50:17.249732 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:17.249694    5060 patch_prober.go:29] interesting pod/e2e-inf-a-e98df8-openshift-default-779d98479-r2jbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" start-of-body=
Oct 31 12:50:17.249935 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:17.249771    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e98df8-openshift-default-779d98479-r2jbq" podUID="e08b08ff-4f40-474a-a103-2c499e999942" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused"
Oct 31 12:50:17.622086 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:17.622051    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-vxzjg"
Oct 31 12:50:17.623303 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:17.623277    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-vxzjg"
Oct 31 12:50:17.624320 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:17.624296    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-vxzjg"
Oct 31 12:50:18.002748 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:18.002657    5060 patch_prober.go:29] interesting pod/e2e-inf-a-015701-openshift-default-86f6dfd4b7-5lsm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body=
Oct 31 12:50:18.002748 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:18.002718    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-015701-openshift-default-86f6dfd4b7-5lsm7" podUID="ce0f9360-f1f6-462c-b4b1-1e0a2232fa7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused"
Oct 31 12:50:18.303277 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:18.303248    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-e98df8-openshift-default-779d98479-r2jbq"
Oct 31 12:50:18.304974 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:18.304859    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-e98df8-openshift-default-779d98479-r2jbq"
Oct 31 12:50:18.305799 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:18.305776    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-e98df8-openshift-default-779d98479-r2jbq"
Oct 31 12:50:19.010993 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:19.010949    5060 patch_prober.go:29] interesting pod/e2e-inf-a-015701-openshift-default-86f6dfd4b7-5lsm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body=
Oct 31 12:50:19.011646 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:19.011610    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-015701-openshift-default-86f6dfd4b7-5lsm7" podUID="ce0f9360-f1f6-462c-b4b1-1e0a2232fa7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused"
Oct 31 12:50:20.007516 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:20.006460    5060 patch_prober.go:29] interesting pod/e2e-inf-a-015701-openshift-default-86f6dfd4b7-5lsm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body=
Oct 31 12:50:20.007516 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:20.006542    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-015701-openshift-default-86f6dfd4b7-5lsm7" podUID="ce0f9360-f1f6-462c-b4b1-1e0a2232fa7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused"
Oct 31 12:50:21.007834 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:21.007802    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-015701-openshift-default-86f6dfd4b7-5lsm7"
Oct 31 12:50:21.008343 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:21.008309    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-015701-openshift-default-86f6dfd4b7-5lsm7"
Oct 31 12:50:21.008994 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:21.008973    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-015701-openshift-default-86f6dfd4b7-5lsm7"
Oct 31 12:50:23.536995 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:23.536960    5060 scope.go:122] "RemoveContainer" containerID="41c3aca603a4f0b29ce259ea8da92927ca9c6c13f57317d6f6dead9e8721479b"
Oct 31 12:50:23.549692 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:23.549669    5060 scope.go:122] "RemoveContainer" containerID="713e0be1c0ac6a852d5adac9b6e465a1a779e5d5953887093396407ffb0a8215"
Oct 31 12:50:23.561504 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:23.561444    5060 scope.go:122] "RemoveContainer" containerID="8cb07de612b2bc92d1780daf393195acb445a1c1eda2cea0a134626b8b26ba58"
Oct 31 12:50:23.572872 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:23.572850    5060 scope.go:122] "RemoveContainer" containerID="5e6f756002afe25183885ebe3601efc4d4cf83bf2f6d0a0f523c7bf4669a04d9"
Oct 31 12:50:23.585556 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:23.585486    5060 scope.go:122] "RemoveContainer" containerID="be123b8add8c5d7494dee8a9a60179d44dd325fb208e2e70fdf6aa466f992175"
Oct 31 12:50:23.596692 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:23.596661    5060 scope.go:122] "RemoveContainer" containerID="4a8e2162d8ca6fce13c30b1b0cfb178283457dbd528c4e20d4f901a2a7edaa66"
Oct 31 12:50:29.922931 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:50:29.922849    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7phb" containerName="extractor"
Oct 31 12:50:37.037132 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:37.037091    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4a6b547-bnbgj"]
Oct 31 12:50:37.052735 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:50:37.052694    5060 status_manager.go:1045] "Failed to get status for pod" err="pods \"maas-api-revoke-sub-a4a6b547-bnbgj\" is forbidden: User \"system:node:ip-10-0-130-190.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"odh-ai-gateway-infra\": no relationship found between node 'ip-10-0-130-190.us-east-2.compute.internal' and this object" podUID="c26a0a34-fc47-4c98-9eaa-2e36700c645d" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4a6b547-bnbgj"
Oct 31 12:50:37.055340 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:37.055315    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4a6b547-bnbgj"]
Oct 31 12:50:37.943578 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:37.943530    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8300aa74-49c5l"]
Oct 31 12:50:37.944527 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:37.944510    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4e57789-6b1a-4d17-bdaf-925a3d3d0c43" containerName="istio-proxy"
Oct 31 12:50:37.948773 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:37.948752    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8300aa74-49c5l"
Oct 31 12:50:37.966870 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:37.966842    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 12:50:37.967063 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:37.966925    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7wsmb\""
Oct 31 12:50:37.968375 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:37.968351    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8300aa74-49c5l"]
Oct 31 12:50:37.993245 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:37.993204    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d15391ff-6f37-40b8-bdfa-42410ca1074a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8300aa74-49c5l\" (UID: \"d15391ff-6f37-40b8-bdfa-42410ca1074a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8300aa74-49c5l"
Oct 31 12:50:37.993413 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:37.993290    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8h4rn\" (UniqueName: \"kubernetes.io/projected/d15391ff-6f37-40b8-bdfa-42410ca1074a-kube-api-access-8h4rn\") pod \"maas-api-revoke-sub-8300aa74-49c5l\" (UID: \"d15391ff-6f37-40b8-bdfa-42410ca1074a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8300aa74-49c5l"
Oct 31 12:50:37.993413 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:37.993390    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d15391ff-6f37-40b8-bdfa-42410ca1074a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8300aa74-49c5l\" (UID: \"d15391ff-6f37-40b8-bdfa-42410ca1074a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8300aa74-49c5l"
Oct 31 12:50:38.094939 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:38.094903    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8h4rn\" (UniqueName: \"kubernetes.io/projected/d15391ff-6f37-40b8-bdfa-42410ca1074a-kube-api-access-8h4rn\") pod \"maas-api-revoke-sub-8300aa74-49c5l\" (UID: \"d15391ff-6f37-40b8-bdfa-42410ca1074a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8300aa74-49c5l"
Oct 31 12:50:38.095377 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:38.094975    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d15391ff-6f37-40b8-bdfa-42410ca1074a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8300aa74-49c5l\" (UID: \"d15391ff-6f37-40b8-bdfa-42410ca1074a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8300aa74-49c5l"
Oct 31 12:50:38.095454 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:38.095375    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d15391ff-6f37-40b8-bdfa-42410ca1074a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8300aa74-49c5l\" (UID: \"d15391ff-6f37-40b8-bdfa-42410ca1074a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8300aa74-49c5l"
Oct 31 12:50:38.095872 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:38.095839    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d15391ff-6f37-40b8-bdfa-42410ca1074a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8300aa74-49c5l\" (UID: \"d15391ff-6f37-40b8-bdfa-42410ca1074a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8300aa74-49c5l"
Oct 31 12:50:38.097854 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:38.097830    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d15391ff-6f37-40b8-bdfa-42410ca1074a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8300aa74-49c5l\" (UID: \"d15391ff-6f37-40b8-bdfa-42410ca1074a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8300aa74-49c5l"
Oct 31 12:50:38.108977 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:38.108952    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8h4rn\" (UniqueName: \"kubernetes.io/projected/d15391ff-6f37-40b8-bdfa-42410ca1074a-kube-api-access-8h4rn\") pod \"maas-api-revoke-sub-8300aa74-49c5l\" (UID: \"d15391ff-6f37-40b8-bdfa-42410ca1074a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8300aa74-49c5l"
Oct 31 12:50:38.266550 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:38.266450    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8300aa74-49c5l"
Oct 31 12:50:38.578829 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:50:38.578797    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd15391ff_6f37_40b8_bdfa_42410ca1074a.slice/crio-4e8c896893452471ad19f5d4fb68a0c1ae193d5a9c853ab782cbc65bd41e2821 WatchSource:0}: Error finding container 4e8c896893452471ad19f5d4fb68a0c1ae193d5a9c853ab782cbc65bd41e2821: Status 404 returned error can't find the container with id 4e8c896893452471ad19f5d4fb68a0c1ae193d5a9c853ab782cbc65bd41e2821
Oct 31 12:50:38.582010 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:38.581722    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8300aa74-49c5l"]
Oct 31 12:50:39.051813 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:39.051775    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c26a0a34-fc47-4c98-9eaa-2e36700c645d" path="/var/lib/kubelet/pods/c26a0a34-fc47-4c98-9eaa-2e36700c645d/volumes"
Oct 31 12:50:39.368384 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:39.368294    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8300aa74-49c5l" event={"ID":"d15391ff-6f37-40b8-bdfa-42410ca1074a","Type":"ContainerStarted","Data":"ccfc755f8b94bbd46d89db1fe24611084b3cfd202ff00c49da51659de1e13deb"}
Oct 31 12:50:39.368384 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:39.368358    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8300aa74-49c5l" event={"ID":"d15391ff-6f37-40b8-bdfa-42410ca1074a","Type":"ContainerStarted","Data":"4e8c896893452471ad19f5d4fb68a0c1ae193d5a9c853ab782cbc65bd41e2821"}
Oct 31 12:50:39.391721 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:39.391669    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8300aa74-49c5l" podStartSLOduration=2.391656203 podStartE2EDuration="2.391656203s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:50:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:50:39.389257843 +0000 UTC m=+1402.492325555" watchObservedRunningTime="2026-10-31 12:50:39.391656203 +0000 UTC m=+1402.494723915"
Oct 31 12:50:40.382651 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:40.382601    5060 generic.go:356] "Generic (PLEG): container finished" podID="d15391ff-6f37-40b8-bdfa-42410ca1074a" containerID="ccfc755f8b94bbd46d89db1fe24611084b3cfd202ff00c49da51659de1e13deb" exitCode=0
Oct 31 12:50:40.383374 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:40.382838    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8300aa74-49c5l" event={"ID":"d15391ff-6f37-40b8-bdfa-42410ca1074a","Type":"ContainerDied","Data":"ccfc755f8b94bbd46d89db1fe24611084b3cfd202ff00c49da51659de1e13deb"}
Oct 31 12:50:41.623213 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:41.623189    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8300aa74-49c5l"
Oct 31 12:50:41.677973 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:41.677885    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d15391ff-6f37-40b8-bdfa-42410ca1074a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d15391ff-6f37-40b8-bdfa-42410ca1074a-maas-api-xks-ca\") pod \"d15391ff-6f37-40b8-bdfa-42410ca1074a\" (UID: \"d15391ff-6f37-40b8-bdfa-42410ca1074a\") "
Oct 31 12:50:41.678125 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:41.678060    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d15391ff-6f37-40b8-bdfa-42410ca1074a-kube-api-access-8h4rn\" (UniqueName: \"kubernetes.io/projected/d15391ff-6f37-40b8-bdfa-42410ca1074a-kube-api-access-8h4rn\") pod \"d15391ff-6f37-40b8-bdfa-42410ca1074a\" (UID: \"d15391ff-6f37-40b8-bdfa-42410ca1074a\") "
Oct 31 12:50:41.678251 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:41.678231    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d15391ff-6f37-40b8-bdfa-42410ca1074a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d15391ff-6f37-40b8-bdfa-42410ca1074a-maas-api-openshift-ca\") pod \"d15391ff-6f37-40b8-bdfa-42410ca1074a\" (UID: \"d15391ff-6f37-40b8-bdfa-42410ca1074a\") "
Oct 31 12:50:41.678646 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:41.678616    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d15391ff-6f37-40b8-bdfa-42410ca1074a-maas-api-openshift-ca" pod "d15391ff-6f37-40b8-bdfa-42410ca1074a" (UID: "d15391ff-6f37-40b8-bdfa-42410ca1074a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:50:41.679576 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:41.679546    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d15391ff-6f37-40b8-bdfa-42410ca1074a-maas-api-openshift-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:50:41.680401 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:41.680374    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d15391ff-6f37-40b8-bdfa-42410ca1074a-kube-api-access-8h4rn" pod "d15391ff-6f37-40b8-bdfa-42410ca1074a" (UID: "d15391ff-6f37-40b8-bdfa-42410ca1074a"). InnerVolumeSpecName "kube-api-access-8h4rn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:50:41.680572 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:41.680463    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d15391ff-6f37-40b8-bdfa-42410ca1074a-maas-api-xks-ca" pod "d15391ff-6f37-40b8-bdfa-42410ca1074a" (UID: "d15391ff-6f37-40b8-bdfa-42410ca1074a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:50:41.781274 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:41.781234    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8h4rn\" (UniqueName: \"kubernetes.io/projected/d15391ff-6f37-40b8-bdfa-42410ca1074a-kube-api-access-8h4rn\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:50:41.781274 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:41.781275    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d15391ff-6f37-40b8-bdfa-42410ca1074a-maas-api-xks-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:50:42.415929 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:42.415899    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8300aa74-49c5l"
Oct 31 12:50:42.416119 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:42.415924    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8300aa74-49c5l" event={"ID":"d15391ff-6f37-40b8-bdfa-42410ca1074a","Type":"ContainerDied","Data":"4e8c896893452471ad19f5d4fb68a0c1ae193d5a9c853ab782cbc65bd41e2821"}
Oct 31 12:50:42.416119 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:42.415979    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4e8c896893452471ad19f5d4fb68a0c1ae193d5a9c853ab782cbc65bd41e2821"
Oct 31 12:50:55.034460 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:55.034425    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d9c635e-6g2s2"]
Oct 31 12:50:55.113857 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:55.113812    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d9c635e-6g2s2"]
Oct 31 12:50:57.050950 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:50:57.050920    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d3acdbc-4d7e-4381-9d28-58426f9b3ab1" path="/var/lib/kubelet/pods/6d3acdbc-4d7e-4381-9d28-58426f9b3ab1/volumes"
Oct 31 12:50:59.923359 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:50:59.923321    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7phb" containerName="extractor"
Oct 31 12:51:23.683157 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:51:23.683115    5060 scope.go:122] "RemoveContainer" containerID="d96e44f8637e694f466bd30c2363a2268c34547106ca907b518782800d362ca0"
Oct 31 12:51:30.031088 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:51:30.031044    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7phb" containerName="extractor"
Oct 31 12:51:59.923406 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:51:59.923320    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7phb" containerName="extractor"
Oct 31 12:52:17.717071 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:52:17.717039    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pcvvm_f032909d-bc39-4c9e-99b7-f3d800842119/console-operator/2.log"
Oct 31 12:52:17.719183 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:52:17.719142    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pcvvm_f032909d-bc39-4c9e-99b7-f3d800842119/console-operator/2.log"
Oct 31 12:52:18.029596 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:52:18.029563    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3a5f961-mzrdk"]
Oct 31 12:52:18.034641 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:52:18.034615    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3a5f961-mzrdk"]
Oct 31 12:52:19.142720 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:52:19.142462    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c370a0b-7c18-46a6-8748-92c92b46dca0" path="/var/lib/kubelet/pods/8c370a0b-7c18-46a6-8748-92c92b46dca0/volumes"
Oct 31 12:52:23.722410 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:52:23.722376    5060 scope.go:122] "RemoveContainer" containerID="c447daaf1be3e2ecf74dbc99fd58d71d16f56325999e75a7bbbf6fb6300625dc"
Oct 31 12:52:23.735115 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:52:23.735086    5060 scope.go:122] "RemoveContainer" containerID="91656a6e403cac0b8e7054ee9ad5f7e3746f95bf700620e2d08f90191d058f40"
Oct 31 12:52:23.746637 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:52:23.746612    5060 scope.go:122] "RemoveContainer" containerID="20a88c3241dd7e4aee428d848660c36f9ebe7d091db2450674a045d0c27d7eab"
Oct 31 12:52:29.928138 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:52:29.928096    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7phb" containerName="extractor"
Oct 31 12:52:34.136658 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:52:34.136613    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c59c634d-jprkg"]
Oct 31 12:52:34.147885 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:52:34.147846    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c59c634d-jprkg"]
Oct 31 12:52:35.053008 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:52:35.052965    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3099e3c-9650-4241-93fd-a5236dbddd5c" path="/var/lib/kubelet/pods/e3099e3c-9650-4241-93fd-a5236dbddd5c/volumes"
Oct 31 12:52:47.248452 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:52:47.248417    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e43d8b-vf8xd"]
Oct 31 12:52:47.249556 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:52:47.249338    5060 cpu_manager.go:414] "containerMap: removing container" podUID="d15391ff-6f37-40b8-bdfa-42410ca1074a" containerName="revoke-keys"
Oct 31 12:52:47.249556 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:52:47.249357    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d15391ff-6f37-40b8-bdfa-42410ca1074a" containerName="revoke-keys"
Oct 31 12:52:47.249556 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:52:47.249517    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="d15391ff-6f37-40b8-bdfa-42410ca1074a" containerName="revoke-keys"
Oct 31 12:52:47.255428 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:52:47.255397    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e43d8b-vf8xd"
Oct 31 12:52:47.260466 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:52:47.260441    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 12:52:47.261226 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:52:47.260701    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7wsmb\""
Oct 31 12:52:47.265011 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:52:47.264985    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e43d8b-vf8xd"]
Oct 31 12:52:47.349355 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:52:47.349316    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlk8r\" (UniqueName: \"kubernetes.io/projected/571793e3-cfd3-47d9-b32a-452e2433c40d-kube-api-access-qlk8r\") pod \"maas-api-revoke-keys-e2e-inf-a-e43d8b-vf8xd\" (UID: \"571793e3-cfd3-47d9-b32a-452e2433c40d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e43d8b-vf8xd"
Oct 31 12:52:47.349581 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:52:47.349381    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/571793e3-cfd3-47d9-b32a-452e2433c40d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-e43d8b-vf8xd\" (UID: \"571793e3-cfd3-47d9-b32a-452e2433c40d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e43d8b-vf8xd"
Oct 31 12:52:47.349714 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:52:47.349689    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/571793e3-cfd3-47d9-b32a-452e2433c40d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-e43d8b-vf8xd\" (UID: \"571793e3-cfd3-47d9-b32a-452e2433c40d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e43d8b-vf8xd"
Oct 31 12:52:47.463785 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:52:47.455622    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlk8r\" (UniqueName: \"kubernetes.io/projected/571793e3-cfd3-47d9-b32a-452e2433c40d-kube-api-access-qlk8r\") pod \"maas-api-revoke-keys-e2e-inf-a-e43d8b-vf8xd\" (UID: \"571793e3-cfd3-47d9-b32a-452e2433c40d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e43d8b-vf8xd"
Oct 31 12:52:47.476511 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:52:47.464251    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/571793e3-cfd3-47d9-b32a-452e2433c40d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-e43d8b-vf8xd\" (UID: \"571793e3-cfd3-47d9-b32a-452e2433c40d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e43d8b-vf8xd"
Oct 31 12:52:47.477046 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:52:47.477016    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/571793e3-cfd3-47d9-b32a-452e2433c40d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-e43d8b-vf8xd\" (UID: \"571793e3-cfd3-47d9-b32a-452e2433c40d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e43d8b-vf8xd"
Oct 31 12:52:47.479028 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:52:47.470690    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/571793e3-cfd3-47d9-b32a-452e2433c40d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-e43d8b-vf8xd\" (UID: \"571793e3-cfd3-47d9-b32a-452e2433c40d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e43d8b-vf8xd"
Oct 31 12:52:47.510290 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:52:47.510175    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/571793e3-cfd3-47d9-b32a-452e2433c40d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-e43d8b-vf8xd\" (UID: \"571793e3-cfd3-47d9-b32a-452e2433c40d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e43d8b-vf8xd"
Oct 31 12:52:47.512323 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:52:47.512294    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlk8r\" (UniqueName: \"kubernetes.io/projected/571793e3-cfd3-47d9-b32a-452e2433c40d-kube-api-access-qlk8r\") pod \"maas-api-revoke-keys-e2e-inf-a-e43d8b-vf8xd\" (UID: \"571793e3-cfd3-47d9-b32a-452e2433c40d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e43d8b-vf8xd"
Oct 31 12:52:47.582676 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:52:47.582640    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e43d8b-vf8xd"
Oct 31 12:52:47.961978 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:52:47.961920    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod571793e3_cfd3_47d9_b32a_452e2433c40d.slice/crio-1105674885d3d62d7700eee9e7b35aa4ff0b627605a9c1314f5efeb42fd01d10 WatchSource:0}: Error finding container 1105674885d3d62d7700eee9e7b35aa4ff0b627605a9c1314f5efeb42fd01d10: Status 404 returned error can't find the container with id 1105674885d3d62d7700eee9e7b35aa4ff0b627605a9c1314f5efeb42fd01d10
Oct 31 12:52:47.962735 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:52:47.962700    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e43d8b-vf8xd"]
Oct 31 12:52:48.858415 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:52:48.858377    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e43d8b-vf8xd" event={"ID":"571793e3-cfd3-47d9-b32a-452e2433c40d","Type":"ContainerStarted","Data":"028ec2c00286bccbe1187c8fdba35c324f29c29447cf5bc88797c227060c454f"}
Oct 31 12:52:48.858844 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:52:48.858441    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e43d8b-vf8xd" event={"ID":"571793e3-cfd3-47d9-b32a-452e2433c40d","Type":"ContainerStarted","Data":"1105674885d3d62d7700eee9e7b35aa4ff0b627605a9c1314f5efeb42fd01d10"}
Oct 31 12:52:48.877378 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:52:48.877317    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e43d8b-vf8xd" podStartSLOduration=1.877304576 podStartE2EDuration="1.877304576s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:52:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:52:48.877193306 +0000 UTC m=+1531.980261030" watchObservedRunningTime="2026-10-31 12:52:48.877304576 +0000 UTC m=+1531.980372281"
Oct 31 12:52:49.952912 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:52:49.952868    5060 generic.go:356] "Generic (PLEG): container finished" podID="571793e3-cfd3-47d9-b32a-452e2433c40d" containerID="028ec2c00286bccbe1187c8fdba35c324f29c29447cf5bc88797c227060c454f" exitCode=0
Oct 31 12:52:49.953402 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:52:49.952926    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e43d8b-vf8xd" event={"ID":"571793e3-cfd3-47d9-b32a-452e2433c40d","Type":"ContainerDied","Data":"028ec2c00286bccbe1187c8fdba35c324f29c29447cf5bc88797c227060c454f"}
Oct 31 12:52:51.207565 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:52:51.207526    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e43d8b-vf8xd"
Oct 31 12:52:51.278846 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:52:51.278816    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/571793e3-cfd3-47d9-b32a-452e2433c40d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/571793e3-cfd3-47d9-b32a-452e2433c40d-maas-api-xks-ca\") pod \"571793e3-cfd3-47d9-b32a-452e2433c40d\" (UID: \"571793e3-cfd3-47d9-b32a-452e2433c40d\") "
Oct 31 12:52:51.279204 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:52:51.279181    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/571793e3-cfd3-47d9-b32a-452e2433c40d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/571793e3-cfd3-47d9-b32a-452e2433c40d-maas-api-openshift-ca\") pod \"571793e3-cfd3-47d9-b32a-452e2433c40d\" (UID: \"571793e3-cfd3-47d9-b32a-452e2433c40d\") "
Oct 31 12:52:51.279312 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:52:51.279238    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/571793e3-cfd3-47d9-b32a-452e2433c40d-kube-api-access-qlk8r\" (UniqueName: \"kubernetes.io/projected/571793e3-cfd3-47d9-b32a-452e2433c40d-kube-api-access-qlk8r\") pod \"571793e3-cfd3-47d9-b32a-452e2433c40d\" (UID: \"571793e3-cfd3-47d9-b32a-452e2433c40d\") "
Oct 31 12:52:51.279730 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:52:51.279634    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/571793e3-cfd3-47d9-b32a-452e2433c40d-maas-api-openshift-ca" pod "571793e3-cfd3-47d9-b32a-452e2433c40d" (UID: "571793e3-cfd3-47d9-b32a-452e2433c40d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:52:51.280595 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:52:51.280041    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/571793e3-cfd3-47d9-b32a-452e2433c40d-maas-api-openshift-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:52:51.282103 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:52:51.282061    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/571793e3-cfd3-47d9-b32a-452e2433c40d-maas-api-xks-ca" pod "571793e3-cfd3-47d9-b32a-452e2433c40d" (UID: "571793e3-cfd3-47d9-b32a-452e2433c40d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:52:51.282445 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:52:51.282414    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/571793e3-cfd3-47d9-b32a-452e2433c40d-kube-api-access-qlk8r" pod "571793e3-cfd3-47d9-b32a-452e2433c40d" (UID: "571793e3-cfd3-47d9-b32a-452e2433c40d"). InnerVolumeSpecName "kube-api-access-qlk8r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:52:51.381985 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:52:51.381950    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qlk8r\" (UniqueName: \"kubernetes.io/projected/571793e3-cfd3-47d9-b32a-452e2433c40d-kube-api-access-qlk8r\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:52:51.381985 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:52:51.381982    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/571793e3-cfd3-47d9-b32a-452e2433c40d-maas-api-xks-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:52:51.985765 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:52:51.985732    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e43d8b-vf8xd"
Oct 31 12:52:51.985765 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:52:51.985749    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e43d8b-vf8xd" event={"ID":"571793e3-cfd3-47d9-b32a-452e2433c40d","Type":"ContainerDied","Data":"1105674885d3d62d7700eee9e7b35aa4ff0b627605a9c1314f5efeb42fd01d10"}
Oct 31 12:52:51.985992 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:52:51.985799    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1105674885d3d62d7700eee9e7b35aa4ff0b627605a9c1314f5efeb42fd01d10"
Oct 31 12:52:53.410639 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:52:53.410603    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4e8eb9-hxhgd"]
Oct 31 12:52:53.411465 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:52:53.411447    5060 cpu_manager.go:414] "containerMap: removing container" podUID="571793e3-cfd3-47d9-b32a-452e2433c40d" containerName="revoke-keys"
Oct 31 12:52:53.411528 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:52:53.411467    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="571793e3-cfd3-47d9-b32a-452e2433c40d" containerName="revoke-keys"
Oct 31 12:52:53.411632 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:52:53.411621    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="571793e3-cfd3-47d9-b32a-452e2433c40d" containerName="revoke-keys"
Oct 31 12:52:53.415666 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:52:53.415646    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4e8eb9-hxhgd"
Oct 31 12:52:53.428125 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:52:53.428090    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7wsmb\""
Oct 31 12:52:53.428861 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:52:53.428837    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 12:52:53.446060 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:52:53.446027    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4e8eb9-hxhgd"]
Oct 31 12:52:53.531255 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:52:53.531210    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fg94x\" (UniqueName: \"kubernetes.io/projected/01233c4a-9626-42cc-b7fb-e58e8f1c0278-kube-api-access-fg94x\") pod \"maas-api-revoke-keys-e2e-inf-a-4e8eb9-hxhgd\" (UID: \"01233c4a-9626-42cc-b7fb-e58e8f1c0278\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4e8eb9-hxhgd"
Oct 31 12:52:53.531478 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:52:53.531308    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01233c4a-9626-42cc-b7fb-e58e8f1c0278-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-4e8eb9-hxhgd\" (UID: \"01233c4a-9626-42cc-b7fb-e58e8f1c0278\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4e8eb9-hxhgd"
Oct 31 12:52:53.531478 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:52:53.531429    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01233c4a-9626-42cc-b7fb-e58e8f1c0278-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-4e8eb9-hxhgd\" (UID: \"01233c4a-9626-42cc-b7fb-e58e8f1c0278\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4e8eb9-hxhgd"
Oct 31 12:52:53.633477 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:52:53.633445    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fg94x\" (UniqueName: \"kubernetes.io/projected/01233c4a-9626-42cc-b7fb-e58e8f1c0278-kube-api-access-fg94x\") pod \"maas-api-revoke-keys-e2e-inf-a-4e8eb9-hxhgd\" (UID: \"01233c4a-9626-42cc-b7fb-e58e8f1c0278\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4e8eb9-hxhgd"
Oct 31 12:52:53.633685 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:52:53.633553    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01233c4a-9626-42cc-b7fb-e58e8f1c0278-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-4e8eb9-hxhgd\" (UID: \"01233c4a-9626-42cc-b7fb-e58e8f1c0278\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4e8eb9-hxhgd"
Oct 31 12:52:53.633685 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:52:53.633649    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01233c4a-9626-42cc-b7fb-e58e8f1c0278-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-4e8eb9-hxhgd\" (UID: \"01233c4a-9626-42cc-b7fb-e58e8f1c0278\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4e8eb9-hxhgd"
Oct 31 12:52:53.634326 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:52:53.634303    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01233c4a-9626-42cc-b7fb-e58e8f1c0278-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-4e8eb9-hxhgd\" (UID: \"01233c4a-9626-42cc-b7fb-e58e8f1c0278\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4e8eb9-hxhgd"
Oct 31 12:52:53.635905 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:52:53.635883    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01233c4a-9626-42cc-b7fb-e58e8f1c0278-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-4e8eb9-hxhgd\" (UID: \"01233c4a-9626-42cc-b7fb-e58e8f1c0278\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4e8eb9-hxhgd"
Oct 31 12:52:53.655209 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:52:53.655183    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fg94x\" (UniqueName: \"kubernetes.io/projected/01233c4a-9626-42cc-b7fb-e58e8f1c0278-kube-api-access-fg94x\") pod \"maas-api-revoke-keys-e2e-inf-a-4e8eb9-hxhgd\" (UID: \"01233c4a-9626-42cc-b7fb-e58e8f1c0278\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4e8eb9-hxhgd"
Oct 31 12:52:53.753715 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:52:53.753634    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4e8eb9-hxhgd"
Oct 31 12:52:54.055135 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:52:54.055097    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01233c4a_9626_42cc_b7fb_e58e8f1c0278.slice/crio-d9d1c17156a644f072274ca3858f6834d7d4267692b9580ec356ba7b9b5f650d WatchSource:0}: Error finding container d9d1c17156a644f072274ca3858f6834d7d4267692b9580ec356ba7b9b5f650d: Status 404 returned error can't find the container with id d9d1c17156a644f072274ca3858f6834d7d4267692b9580ec356ba7b9b5f650d
Oct 31 12:52:54.057685 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:52:54.057657    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4e8eb9-hxhgd"]
Oct 31 12:52:54.207827 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:52:54.207784    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-njl9z"]
Oct 31 12:52:54.208109 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:52:54.208074    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-njl9z" podUID="29b176fa-229b-433e-928f-41daf33d1f37" containerName="istio-proxy" containerID="cri-o://6fafb77a1324904f28fa906f139fc8ee088f838268da7825463b6b19073c703d" gracePeriod=30
Oct 31 12:52:54.214569 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:52:54.214540    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-mb4nn"]
Oct 31 12:52:54.214764 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:52:54.214726    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-mb4nn" podUID="0ade94c0-4cba-4ccd-87c0-cdd0cc2aa72d" containerName="istio-proxy" containerID="cri-o://b1792e1e8e3a369422a6939d613bdc1025975b26261bb7666cbc4972dcae2ebd" gracePeriod=30
Oct 31 12:52:54.278514 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:52:54.274963    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-jf8bm"]
Oct 31 12:52:54.278514 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:52:54.275155    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-jf8bm" podUID="02360407-0dfa-45b7-b276-6487dcfb02f0" containerName="istio-proxy" containerID="cri-o://7c91c181e45356241d7139f6aeedfaae2dd5b0a8dd32e6953ea1f26c79face5f" gracePeriod=30
Oct 31 12:52:54.348762 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:52:54.346377    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-gxs26"]
Oct 31 12:52:54.348762 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:52:54.346632    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-gxs26" podUID="86c84864-127f-4507-8f60-2dbb684c687a" containerName="istio-proxy" containerID="cri-o://b0e2771f19a497662c194c4899a869898da0bab9a01207f1478d9e573f31a211" gracePeriod=30
Oct 31 12:52:54.360134 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:52:54.357057    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-sv4w2"]
Oct 31 12:52:54.360134 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:52:54.357282    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-sv4w2" podUID="38bdca08-eb2b-43c5-8c07-7059e5c9e199" containerName="istio-proxy" containerID="cri-o://4c7d3ca2a9c8a503e740a602271d240f30dea66d2fed340dac15ec3cb71094ef" gracePeriod=30
Oct 31 12:52:54.798869 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:52:54.798833    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-sc6ff"]
Oct 31 12:52:54.799284 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:52:54.799046    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-sc6ff" podUID="360843c7-795f-4f22-84cd-76317addf9c2" containerName="istio-proxy" containerID="cri-o://b163f7ffd10c6999491edd6aadba751bcc983950890ceebea3672e7540b9ec48" gracePeriod=30
Oct 31 12:52:55.069988 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:52:55.069900    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4e8eb9-hxhgd" event={"ID":"01233c4a-9626-42cc-b7fb-e58e8f1c0278","Type":"ContainerStarted","Data":"5639e6818bba3be7fa72253bfb4ca3ed52dc79c5726b90a8bbf52c4520c10e44"}
Oct 31 12:52:55.069988 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:52:55.069966    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4e8eb9-hxhgd" event={"ID":"01233c4a-9626-42cc-b7fb-e58e8f1c0278","Type":"ContainerStarted","Data":"d9d1c17156a644f072274ca3858f6834d7d4267692b9580ec356ba7b9b5f650d"}
Oct 31 12:52:55.116404 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:52:55.116353    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4e8eb9-hxhgd" podStartSLOduration=2.116337762 podStartE2EDuration="2.116337762s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:52:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:52:55.111270105 +0000 UTC m=+1538.214337819" watchObservedRunningTime="2026-10-31 12:52:55.116337762 +0000 UTC m=+1538.219405475"
Oct 31 12:52:56.072618 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:52:56.072582    5060 generic.go:356] "Generic (PLEG): container finished" podID="01233c4a-9626-42cc-b7fb-e58e8f1c0278" containerID="5639e6818bba3be7fa72253bfb4ca3ed52dc79c5726b90a8bbf52c4520c10e44" exitCode=0
Oct 31 12:52:56.073026 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:52:56.072659    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4e8eb9-hxhgd" event={"ID":"01233c4a-9626-42cc-b7fb-e58e8f1c0278","Type":"ContainerDied","Data":"5639e6818bba3be7fa72253bfb4ca3ed52dc79c5726b90a8bbf52c4520c10e44"}
Oct 31 12:52:57.391115 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:52:57.391080    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4e8eb9-hxhgd"
Oct 31 12:52:57.441244 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:52:57.441212    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/01233c4a-9626-42cc-b7fb-e58e8f1c0278-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01233c4a-9626-42cc-b7fb-e58e8f1c0278-maas-api-xks-ca\") pod \"01233c4a-9626-42cc-b7fb-e58e8f1c0278\" (UID: \"01233c4a-9626-42cc-b7fb-e58e8f1c0278\") "
Oct 31 12:52:57.442069 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:52:57.441444    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01233c4a-9626-42cc-b7fb-e58e8f1c0278-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01233c4a-9626-42cc-b7fb-e58e8f1c0278-maas-api-openshift-ca\") pod \"01233c4a-9626-42cc-b7fb-e58e8f1c0278\" (UID: \"01233c4a-9626-42cc-b7fb-e58e8f1c0278\") "
Oct 31 12:52:57.442069 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:52:57.441543    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01233c4a-9626-42cc-b7fb-e58e8f1c0278-kube-api-access-fg94x\" (UniqueName: \"kubernetes.io/projected/01233c4a-9626-42cc-b7fb-e58e8f1c0278-kube-api-access-fg94x\") pod \"01233c4a-9626-42cc-b7fb-e58e8f1c0278\" (UID: \"01233c4a-9626-42cc-b7fb-e58e8f1c0278\") "
Oct 31 12:52:57.442069 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:52:57.441890    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01233c4a-9626-42cc-b7fb-e58e8f1c0278-maas-api-openshift-ca" pod "01233c4a-9626-42cc-b7fb-e58e8f1c0278" (UID: "01233c4a-9626-42cc-b7fb-e58e8f1c0278"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:52:57.442541 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:52:57.442503    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01233c4a-9626-42cc-b7fb-e58e8f1c0278-maas-api-openshift-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:52:57.443756 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:52:57.443729    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01233c4a-9626-42cc-b7fb-e58e8f1c0278-maas-api-xks-ca" pod "01233c4a-9626-42cc-b7fb-e58e8f1c0278" (UID: "01233c4a-9626-42cc-b7fb-e58e8f1c0278"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:52:57.443865 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:52:57.443795    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01233c4a-9626-42cc-b7fb-e58e8f1c0278-kube-api-access-fg94x" pod "01233c4a-9626-42cc-b7fb-e58e8f1c0278" (UID: "01233c4a-9626-42cc-b7fb-e58e8f1c0278"). InnerVolumeSpecName "kube-api-access-fg94x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:52:57.545559 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:52:57.543944    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fg94x\" (UniqueName: \"kubernetes.io/projected/01233c4a-9626-42cc-b7fb-e58e8f1c0278-kube-api-access-fg94x\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:52:57.545559 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:52:57.543977    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01233c4a-9626-42cc-b7fb-e58e8f1c0278-maas-api-xks-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:52:58.271514 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:52:58.244648    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4e8eb9-hxhgd" event={"ID":"01233c4a-9626-42cc-b7fb-e58e8f1c0278","Type":"ContainerDied","Data":"d9d1c17156a644f072274ca3858f6834d7d4267692b9580ec356ba7b9b5f650d"}
Oct 31 12:52:58.271514 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:52:58.244719    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d9d1c17156a644f072274ca3858f6834d7d4267692b9580ec356ba7b9b5f650d"
Oct 31 12:52:58.271514 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:52:58.244798    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4e8eb9-hxhgd"
Oct 31 12:52:58.436098 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:52:58.436062    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e43d8b-765bcc4ffb-g4s6x"]
Oct 31 12:52:58.436927 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:52:58.436891    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e43d8b-765bcc4ffb-g4s6x" podUID="dec1f857-9b34-458f-8954-56d2753466db" containerName="maas-api" containerID="cri-o://899bb28709d691b64688ef03645611a5e6dc2d0ef12a447bd5722fa22622a316" gracePeriod=30
Oct 31 12:52:58.943700 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:52:58.943601    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e43d8b-vf8xd"]
Oct 31 12:52:58.965988 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:52:58.955201    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e43d8b-vf8xd"]
Oct 31 12:52:59.095302 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:52:59.095208    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="571793e3-cfd3-47d9-b32a-452e2433c40d" path="/var/lib/kubelet/pods/571793e3-cfd3-47d9-b32a-452e2433c40d/volumes"
Oct 31 12:52:59.358511 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:52:59.338293    5060 generic.go:356] "Generic (PLEG): container finished" podID="dec1f857-9b34-458f-8954-56d2753466db" containerID="899bb28709d691b64688ef03645611a5e6dc2d0ef12a447bd5722fa22622a316" exitCode=0
Oct 31 12:52:59.358511 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:52:59.338442    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e43d8b-765bcc4ffb-g4s6x" event={"ID":"dec1f857-9b34-458f-8954-56d2753466db","Type":"ContainerDied","Data":"899bb28709d691b64688ef03645611a5e6dc2d0ef12a447bd5722fa22622a316"}
Oct 31 12:52:59.522365 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:52:59.521620    5060 patch_prober.go:29] interesting pod/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-mb4nn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Oct 31 12:52:59.522365 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:52:59.521705    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-mb4nn" podUID="0ade94c0-4cba-4ccd-87c0-cdd0cc2aa72d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Oct 31 12:52:59.780753 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:52:59.780343    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e43d8b-765bcc4ffb-g4s6x"
Oct 31 12:52:59.842520 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:52:59.838238    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dec1f857-9b34-458f-8954-56d2753466db-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dec1f857-9b34-458f-8954-56d2753466db-maas-api-tls\") pod \"dec1f857-9b34-458f-8954-56d2753466db\" (UID: \"dec1f857-9b34-458f-8954-56d2753466db\") "
Oct 31 12:52:59.842520 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:52:59.838293    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dec1f857-9b34-458f-8954-56d2753466db-kube-api-access-m86mk\" (UniqueName: \"kubernetes.io/projected/dec1f857-9b34-458f-8954-56d2753466db-kube-api-access-m86mk\") pod \"dec1f857-9b34-458f-8954-56d2753466db\" (UID: \"dec1f857-9b34-458f-8954-56d2753466db\") "
Oct 31 12:52:59.842520 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:52:59.838468    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dec1f857-9b34-458f-8954-56d2753466db-metrics-certs\" (UniqueName: \"kubernetes.io/secret/dec1f857-9b34-458f-8954-56d2753466db-metrics-certs\") pod \"dec1f857-9b34-458f-8954-56d2753466db\" (UID: \"dec1f857-9b34-458f-8954-56d2753466db\") "
Oct 31 12:52:59.842520 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:52:59.838630    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dec1f857-9b34-458f-8954-56d2753466db-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dec1f857-9b34-458f-8954-56d2753466db-openshift-service-ca\") pod \"dec1f857-9b34-458f-8954-56d2753466db\" (UID: \"dec1f857-9b34-458f-8954-56d2753466db\") "
Oct 31 12:52:59.844140 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:52:59.844087    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dec1f857-9b34-458f-8954-56d2753466db-openshift-service-ca" pod "dec1f857-9b34-458f-8954-56d2753466db" (UID: "dec1f857-9b34-458f-8954-56d2753466db"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:52:59.880714 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:52:59.878032    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dec1f857-9b34-458f-8954-56d2753466db-kube-api-access-m86mk" pod "dec1f857-9b34-458f-8954-56d2753466db" (UID: "dec1f857-9b34-458f-8954-56d2753466db"). InnerVolumeSpecName "kube-api-access-m86mk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:52:59.907485 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:52:59.907448    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dec1f857-9b34-458f-8954-56d2753466db-metrics-certs" pod "dec1f857-9b34-458f-8954-56d2753466db" (UID: "dec1f857-9b34-458f-8954-56d2753466db"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:52:59.908735 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:52:59.908697    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dec1f857-9b34-458f-8954-56d2753466db-maas-api-tls" pod "dec1f857-9b34-458f-8954-56d2753466db" (UID: "dec1f857-9b34-458f-8954-56d2753466db"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:52:59.944346 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:52:59.944308    5060 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dec1f857-9b34-458f-8954-56d2753466db-openshift-service-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:52:59.944604 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:52:59.944582    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dec1f857-9b34-458f-8954-56d2753466db-maas-api-tls\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:52:59.944718 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:52:59.944705    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m86mk\" (UniqueName: \"kubernetes.io/projected/dec1f857-9b34-458f-8954-56d2753466db-kube-api-access-m86mk\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:52:59.944811 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:52:59.944800    5060 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dec1f857-9b34-458f-8954-56d2753466db-metrics-certs\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:00.129089 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:53:00.129046    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7phb" containerName="extractor"
Oct 31 12:53:00.235620 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.235579    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-vqcct"]
Oct 31 12:53:00.237976 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.237948    5060 cpu_manager.go:414] "containerMap: removing container" podUID="dec1f857-9b34-458f-8954-56d2753466db" containerName="maas-api"
Oct 31 12:53:00.238144 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.238130    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dec1f857-9b34-458f-8954-56d2753466db" containerName="maas-api"
Oct 31 12:53:00.238290 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.238278    5060 cpu_manager.go:414] "containerMap: removing container" podUID="01233c4a-9626-42cc-b7fb-e58e8f1c0278" containerName="revoke-keys"
Oct 31 12:53:00.238371 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.238361    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01233c4a-9626-42cc-b7fb-e58e8f1c0278" containerName="revoke-keys"
Oct 31 12:53:00.238813 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.238796    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="dec1f857-9b34-458f-8954-56d2753466db" containerName="maas-api"
Oct 31 12:53:00.239017 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.239004    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="01233c4a-9626-42cc-b7fb-e58e8f1c0278" containerName="revoke-keys"
Oct 31 12:53:00.249191 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.249158    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-vqcct"]
Oct 31 12:53:00.249418 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.249402    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-vqcct"
Oct 31 12:53:00.297259 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.297225    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-ad1af7-openshift-default-dockercfg-9zj6x\""
Oct 31 12:53:00.309848 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.308745    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-mb4nn"
Oct 31 12:53:00.403446 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.403361    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-e98df8-openshift-default-779d98479-vkrj6"]
Oct 31 12:53:00.405003 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.404975    5060 cpu_manager.go:414] "containerMap: removing container" podUID="0ade94c0-4cba-4ccd-87c0-cdd0cc2aa72d" containerName="istio-proxy"
Oct 31 12:53:00.405655 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.405143    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ade94c0-4cba-4ccd-87c0-cdd0cc2aa72d" containerName="istio-proxy"
Oct 31 12:53:00.405655 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.405448    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ade94c0-4cba-4ccd-87c0-cdd0cc2aa72d" containerName="istio-proxy"
Oct 31 12:53:00.420037 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.420013    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-e98df8-openshift-default-779d98479-vkrj6"]
Oct 31 12:53:00.420190 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.420055    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-njl9z" event={"ID":"29b176fa-229b-433e-928f-41daf33d1f37","Type":"ContainerDied","Data":"6fafb77a1324904f28fa906f139fc8ee088f838268da7825463b6b19073c703d"}
Oct 31 12:53:00.420190 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.420059    5060 generic.go:356] "Generic (PLEG): container finished" podID="29b176fa-229b-433e-928f-41daf33d1f37" containerID="6fafb77a1324904f28fa906f139fc8ee088f838268da7825463b6b19073c703d" exitCode=0
Oct 31 12:53:00.420190 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.420116    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e98df8-openshift-default-779d98479-vkrj6"
Oct 31 12:53:00.421030 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.420883    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-jf8bm"
Oct 31 12:53:00.423106 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.423079    5060 generic.go:356] "Generic (PLEG): container finished" podID="360843c7-795f-4f22-84cd-76317addf9c2" containerID="b163f7ffd10c6999491edd6aadba751bcc983950890ceebea3672e7540b9ec48" exitCode=0
Oct 31 12:53:00.423205 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.423188    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-sc6ff" event={"ID":"360843c7-795f-4f22-84cd-76317addf9c2","Type":"ContainerDied","Data":"b163f7ffd10c6999491edd6aadba751bcc983950890ceebea3672e7540b9ec48"}
Oct 31 12:53:00.425098 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.425078    5060 generic.go:356] "Generic (PLEG): container finished" podID="0ade94c0-4cba-4ccd-87c0-cdd0cc2aa72d" containerID="b1792e1e8e3a369422a6939d613bdc1025975b26261bb7666cbc4972dcae2ebd" exitCode=0
Oct 31 12:53:00.425834 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.425124    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-mb4nn"
Oct 31 12:53:00.425834 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.425138    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-mb4nn" event={"ID":"0ade94c0-4cba-4ccd-87c0-cdd0cc2aa72d","Type":"ContainerDied","Data":"b1792e1e8e3a369422a6939d613bdc1025975b26261bb7666cbc4972dcae2ebd"}
Oct 31 12:53:00.425834 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.425465    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-mb4nn" event={"ID":"0ade94c0-4cba-4ccd-87c0-cdd0cc2aa72d","Type":"ContainerDied","Data":"43024daa959faca0f71635d631ac87ecd654825c8d1dbe8347f526c9464dfe4b"}
Oct 31 12:53:00.425834 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.425543    5060 scope.go:122] "RemoveContainer" containerID="b1792e1e8e3a369422a6939d613bdc1025975b26261bb7666cbc4972dcae2ebd"
Oct 31 12:53:00.425834 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.425811    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ade94c0-4cba-4ccd-87c0-cdd0cc2aa72d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ade94c0-4cba-4ccd-87c0-cdd0cc2aa72d-istio-data\") pod \"0ade94c0-4cba-4ccd-87c0-cdd0cc2aa72d\" (UID: \"0ade94c0-4cba-4ccd-87c0-cdd0cc2aa72d\") "
Oct 31 12:53:00.426884 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.425873    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ade94c0-4cba-4ccd-87c0-cdd0cc2aa72d-istio-token\" (UniqueName: \"kubernetes.io/projected/0ade94c0-4cba-4ccd-87c0-cdd0cc2aa72d-istio-token\") pod \"0ade94c0-4cba-4ccd-87c0-cdd0cc2aa72d\" (UID: \"0ade94c0-4cba-4ccd-87c0-cdd0cc2aa72d\") "
Oct 31 12:53:00.426884 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.425939    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0ade94c0-4cba-4ccd-87c0-cdd0cc2aa72d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ade94c0-4cba-4ccd-87c0-cdd0cc2aa72d-istio-podinfo\") pod \"0ade94c0-4cba-4ccd-87c0-cdd0cc2aa72d\" (UID: \"0ade94c0-4cba-4ccd-87c0-cdd0cc2aa72d\") "
Oct 31 12:53:00.426884 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.426045    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ade94c0-4cba-4ccd-87c0-cdd0cc2aa72d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ade94c0-4cba-4ccd-87c0-cdd0cc2aa72d-istio-envoy\") pod \"0ade94c0-4cba-4ccd-87c0-cdd0cc2aa72d\" (UID: \"0ade94c0-4cba-4ccd-87c0-cdd0cc2aa72d\") "
Oct 31 12:53:00.426884 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.426114    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ade94c0-4cba-4ccd-87c0-cdd0cc2aa72d-kube-api-access-gfj75\" (UniqueName: \"kubernetes.io/projected/0ade94c0-4cba-4ccd-87c0-cdd0cc2aa72d-kube-api-access-gfj75\") pod \"0ade94c0-4cba-4ccd-87c0-cdd0cc2aa72d\" (UID: \"0ade94c0-4cba-4ccd-87c0-cdd0cc2aa72d\") "
Oct 31 12:53:00.426884 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.426182    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ade94c0-4cba-4ccd-87c0-cdd0cc2aa72d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ade94c0-4cba-4ccd-87c0-cdd0cc2aa72d-workload-socket\") pod \"0ade94c0-4cba-4ccd-87c0-cdd0cc2aa72d\" (UID: \"0ade94c0-4cba-4ccd-87c0-cdd0cc2aa72d\") "
Oct 31 12:53:00.426884 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.426199    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ade94c0-4cba-4ccd-87c0-cdd0cc2aa72d-istio-data" pod "0ade94c0-4cba-4ccd-87c0-cdd0cc2aa72d" (UID: "0ade94c0-4cba-4ccd-87c0-cdd0cc2aa72d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:53:00.426884 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.426226    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ade94c0-4cba-4ccd-87c0-cdd0cc2aa72d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ade94c0-4cba-4ccd-87c0-cdd0cc2aa72d-workload-certs\") pod \"0ade94c0-4cba-4ccd-87c0-cdd0cc2aa72d\" (UID: \"0ade94c0-4cba-4ccd-87c0-cdd0cc2aa72d\") "
Oct 31 12:53:00.426884 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.426286    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ade94c0-4cba-4ccd-87c0-cdd0cc2aa72d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ade94c0-4cba-4ccd-87c0-cdd0cc2aa72d-credential-socket\") pod \"0ade94c0-4cba-4ccd-87c0-cdd0cc2aa72d\" (UID: \"0ade94c0-4cba-4ccd-87c0-cdd0cc2aa72d\") "
Oct 31 12:53:00.426884 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.426400    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ade94c0-4cba-4ccd-87c0-cdd0cc2aa72d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ade94c0-4cba-4ccd-87c0-cdd0cc2aa72d-istiod-ca-cert\") pod \"0ade94c0-4cba-4ccd-87c0-cdd0cc2aa72d\" (UID: \"0ade94c0-4cba-4ccd-87c0-cdd0cc2aa72d\") "
Oct 31 12:53:00.426884 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.426849    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2705ec7-400e-4265-b213-5b58ba42cd4c-workload-socket\") pod \"e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-vqcct\" (UID: \"b2705ec7-400e-4265-b213-5b58ba42cd4c\") " pod="openshift-ingress/e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-vqcct"
Oct 31 12:53:00.428359 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.426956    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2705ec7-400e-4265-b213-5b58ba42cd4c-istio-token\") pod \"e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-vqcct\" (UID: \"b2705ec7-400e-4265-b213-5b58ba42cd4c\") " pod="openshift-ingress/e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-vqcct"
Oct 31 12:53:00.428359 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.427031    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4l6w\" (UniqueName: \"kubernetes.io/projected/b2705ec7-400e-4265-b213-5b58ba42cd4c-kube-api-access-p4l6w\") pod \"e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-vqcct\" (UID: \"b2705ec7-400e-4265-b213-5b58ba42cd4c\") " pod="openshift-ingress/e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-vqcct"
Oct 31 12:53:00.428359 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.427108    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2705ec7-400e-4265-b213-5b58ba42cd4c-istiod-ca-cert\") pod \"e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-vqcct\" (UID: \"b2705ec7-400e-4265-b213-5b58ba42cd4c\") " pod="openshift-ingress/e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-vqcct"
Oct 31 12:53:00.428359 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.427149    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2705ec7-400e-4265-b213-5b58ba42cd4c-credential-socket\") pod \"e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-vqcct\" (UID: \"b2705ec7-400e-4265-b213-5b58ba42cd4c\") " pod="openshift-ingress/e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-vqcct"
Oct 31 12:53:00.428359 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.427172    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2705ec7-400e-4265-b213-5b58ba42cd4c-istio-data\") pod \"e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-vqcct\" (UID: \"b2705ec7-400e-4265-b213-5b58ba42cd4c\") " pod="openshift-ingress/e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-vqcct"
Oct 31 12:53:00.428359 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.427384    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2705ec7-400e-4265-b213-5b58ba42cd4c-istio-podinfo\") pod \"e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-vqcct\" (UID: \"b2705ec7-400e-4265-b213-5b58ba42cd4c\") " pod="openshift-ingress/e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-vqcct"
Oct 31 12:53:00.428359 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.427505    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2705ec7-400e-4265-b213-5b58ba42cd4c-workload-certs\") pod \"e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-vqcct\" (UID: \"b2705ec7-400e-4265-b213-5b58ba42cd4c\") " pod="openshift-ingress/e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-vqcct"
Oct 31 12:53:00.428359 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.427550    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2705ec7-400e-4265-b213-5b58ba42cd4c-istio-envoy\") pod \"e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-vqcct\" (UID: \"b2705ec7-400e-4265-b213-5b58ba42cd4c\") " pod="openshift-ingress/e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-vqcct"
Oct 31 12:53:00.428359 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.427681    5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ade94c0-4cba-4ccd-87c0-cdd0cc2aa72d-istio-data\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:00.428359 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.427840    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ade94c0-4cba-4ccd-87c0-cdd0cc2aa72d-istiod-ca-cert" pod "0ade94c0-4cba-4ccd-87c0-cdd0cc2aa72d" (UID: "0ade94c0-4cba-4ccd-87c0-cdd0cc2aa72d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:53:00.430655 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.428519    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ade94c0-4cba-4ccd-87c0-cdd0cc2aa72d-workload-socket" pod "0ade94c0-4cba-4ccd-87c0-cdd0cc2aa72d" (UID: "0ade94c0-4cba-4ccd-87c0-cdd0cc2aa72d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:53:00.430655 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.429074    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ade94c0-4cba-4ccd-87c0-cdd0cc2aa72d-credential-socket" pod "0ade94c0-4cba-4ccd-87c0-cdd0cc2aa72d" (UID: "0ade94c0-4cba-4ccd-87c0-cdd0cc2aa72d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:53:00.431289 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.431263    5060 generic.go:356] "Generic (PLEG): container finished" podID="02360407-0dfa-45b7-b276-6487dcfb02f0" containerID="7c91c181e45356241d7139f6aeedfaae2dd5b0a8dd32e6953ea1f26c79face5f" exitCode=0
Oct 31 12:53:00.431379 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.431341    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-jf8bm" event={"ID":"02360407-0dfa-45b7-b276-6487dcfb02f0","Type":"ContainerDied","Data":"7c91c181e45356241d7139f6aeedfaae2dd5b0a8dd32e6953ea1f26c79face5f"}
Oct 31 12:53:00.431440 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.431379    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-jf8bm" event={"ID":"02360407-0dfa-45b7-b276-6487dcfb02f0","Type":"ContainerDied","Data":"2a012caa6d497a800ee0a3dee28d50bf49ef23cc7a94e6a9c4e43a3b864e055d"}
Oct 31 12:53:00.431539 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.431456    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-jf8bm"
Oct 31 12:53:00.433614 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.433587    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ade94c0-4cba-4ccd-87c0-cdd0cc2aa72d-workload-certs" pod "0ade94c0-4cba-4ccd-87c0-cdd0cc2aa72d" (UID: "0ade94c0-4cba-4ccd-87c0-cdd0cc2aa72d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:53:00.439027 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.438998    5060 generic.go:356] "Generic (PLEG): container finished" podID="86c84864-127f-4507-8f60-2dbb684c687a" containerID="b0e2771f19a497662c194c4899a869898da0bab9a01207f1478d9e573f31a211" exitCode=0
Oct 31 12:53:00.439126 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.439092    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-gxs26" event={"ID":"86c84864-127f-4507-8f60-2dbb684c687a","Type":"ContainerDied","Data":"b0e2771f19a497662c194c4899a869898da0bab9a01207f1478d9e573f31a211"}
Oct 31 12:53:00.628072 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.530989    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ade94c0-4cba-4ccd-87c0-cdd0cc2aa72d-istio-token" pod "0ade94c0-4cba-4ccd-87c0-cdd0cc2aa72d" (UID: "0ade94c0-4cba-4ccd-87c0-cdd0cc2aa72d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:53:00.636414 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.634540    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02360407-0dfa-45b7-b276-6487dcfb02f0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02360407-0dfa-45b7-b276-6487dcfb02f0-istio-envoy\") pod \"02360407-0dfa-45b7-b276-6487dcfb02f0\" (UID: \"02360407-0dfa-45b7-b276-6487dcfb02f0\") "
Oct 31 12:53:00.636414 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.634614    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02360407-0dfa-45b7-b276-6487dcfb02f0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02360407-0dfa-45b7-b276-6487dcfb02f0-istiod-ca-cert\") pod \"02360407-0dfa-45b7-b276-6487dcfb02f0\" (UID: \"02360407-0dfa-45b7-b276-6487dcfb02f0\") "
Oct 31 12:53:00.636414 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.634702    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02360407-0dfa-45b7-b276-6487dcfb02f0-kube-api-access-986cp\" (UniqueName: \"kubernetes.io/projected/02360407-0dfa-45b7-b276-6487dcfb02f0-kube-api-access-986cp\") pod \"02360407-0dfa-45b7-b276-6487dcfb02f0\" (UID: \"02360407-0dfa-45b7-b276-6487dcfb02f0\") "
Oct 31 12:53:00.636414 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.634838    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ade94c0-4cba-4ccd-87c0-cdd0cc2aa72d-istio-envoy" pod "0ade94c0-4cba-4ccd-87c0-cdd0cc2aa72d" (UID: "0ade94c0-4cba-4ccd-87c0-cdd0cc2aa72d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:53:00.636414 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.634867    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02360407-0dfa-45b7-b276-6487dcfb02f0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02360407-0dfa-45b7-b276-6487dcfb02f0-credential-socket\") pod \"02360407-0dfa-45b7-b276-6487dcfb02f0\" (UID: \"02360407-0dfa-45b7-b276-6487dcfb02f0\") "
Oct 31 12:53:00.636414 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.635081    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02360407-0dfa-45b7-b276-6487dcfb02f0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02360407-0dfa-45b7-b276-6487dcfb02f0-istio-data\") pod \"02360407-0dfa-45b7-b276-6487dcfb02f0\" (UID: \"02360407-0dfa-45b7-b276-6487dcfb02f0\") "
Oct 31 12:53:00.636414 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.635142    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02360407-0dfa-45b7-b276-6487dcfb02f0-istio-token\" (UniqueName: \"kubernetes.io/projected/02360407-0dfa-45b7-b276-6487dcfb02f0-istio-token\") pod \"02360407-0dfa-45b7-b276-6487dcfb02f0\" (UID: \"02360407-0dfa-45b7-b276-6487dcfb02f0\") "
Oct 31 12:53:00.636414 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.635205    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02360407-0dfa-45b7-b276-6487dcfb02f0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02360407-0dfa-45b7-b276-6487dcfb02f0-workload-certs\") pod \"02360407-0dfa-45b7-b276-6487dcfb02f0\" (UID: \"02360407-0dfa-45b7-b276-6487dcfb02f0\") "
Oct 31 12:53:00.636414 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.635271    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/02360407-0dfa-45b7-b276-6487dcfb02f0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02360407-0dfa-45b7-b276-6487dcfb02f0-istio-podinfo\") pod \"02360407-0dfa-45b7-b276-6487dcfb02f0\" (UID: \"02360407-0dfa-45b7-b276-6487dcfb02f0\") "
Oct 31 12:53:00.636414 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.635354    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02360407-0dfa-45b7-b276-6487dcfb02f0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02360407-0dfa-45b7-b276-6487dcfb02f0-workload-socket\") pod \"02360407-0dfa-45b7-b276-6487dcfb02f0\" (UID: \"02360407-0dfa-45b7-b276-6487dcfb02f0\") "
Oct 31 12:53:00.636414 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.635416    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ade94c0-4cba-4ccd-87c0-cdd0cc2aa72d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ade94c0-4cba-4ccd-87c0-cdd0cc2aa72d-istio-envoy\") pod \"0ade94c0-4cba-4ccd-87c0-cdd0cc2aa72d\" (UID: \"0ade94c0-4cba-4ccd-87c0-cdd0cc2aa72d\") "
Oct 31 12:53:00.636414 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.635708    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d62f8e4-b062-4dd4-a23d-0f66566c83da-credential-socket\") pod \"e2e-inf-a-e98df8-openshift-default-779d98479-vkrj6\" (UID: \"0d62f8e4-b062-4dd4-a23d-0f66566c83da\") " pod="openshift-ingress/e2e-inf-a-e98df8-openshift-default-779d98479-vkrj6"
Oct 31 12:53:00.636414 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.635758    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2705ec7-400e-4265-b213-5b58ba42cd4c-istio-podinfo\") pod \"e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-vqcct\" (UID: \"b2705ec7-400e-4265-b213-5b58ba42cd4c\") " pod="openshift-ingress/e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-vqcct"
Oct 31 12:53:00.636414 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.635873    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2705ec7-400e-4265-b213-5b58ba42cd4c-workload-certs\") pod \"e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-vqcct\" (UID: \"b2705ec7-400e-4265-b213-5b58ba42cd4c\") " pod="openshift-ingress/e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-vqcct"
Oct 31 12:53:00.636414 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.635918    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2705ec7-400e-4265-b213-5b58ba42cd4c-istio-envoy\") pod \"e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-vqcct\" (UID: \"b2705ec7-400e-4265-b213-5b58ba42cd4c\") " pod="openshift-ingress/e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-vqcct"
Oct 31 12:53:00.636414 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.636054    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2705ec7-400e-4265-b213-5b58ba42cd4c-workload-socket\") pod \"e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-vqcct\" (UID: \"b2705ec7-400e-4265-b213-5b58ba42cd4c\") " pod="openshift-ingress/e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-vqcct"
Oct 31 12:53:00.638964 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.637202    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d62f8e4-b062-4dd4-a23d-0f66566c83da-istio-data\") pod \"e2e-inf-a-e98df8-openshift-default-779d98479-vkrj6\" (UID: \"0d62f8e4-b062-4dd4-a23d-0f66566c83da\") " pod="openshift-ingress/e2e-inf-a-e98df8-openshift-default-779d98479-vkrj6"
Oct 31 12:53:00.638964 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.637268    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2705ec7-400e-4265-b213-5b58ba42cd4c-istio-token\") pod \"e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-vqcct\" (UID: \"b2705ec7-400e-4265-b213-5b58ba42cd4c\") " pod="openshift-ingress/e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-vqcct"
Oct 31 12:53:00.638964 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.637334    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d62f8e4-b062-4dd4-a23d-0f66566c83da-istio-podinfo\") pod \"e2e-inf-a-e98df8-openshift-default-779d98479-vkrj6\" (UID: \"0d62f8e4-b062-4dd4-a23d-0f66566c83da\") " pod="openshift-ingress/e2e-inf-a-e98df8-openshift-default-779d98479-vkrj6"
Oct 31 12:53:00.638964 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.637381    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4l6w\" (UniqueName: \"kubernetes.io/projected/b2705ec7-400e-4265-b213-5b58ba42cd4c-kube-api-access-p4l6w\") pod \"e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-vqcct\" (UID: \"b2705ec7-400e-4265-b213-5b58ba42cd4c\") " pod="openshift-ingress/e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-vqcct"
Oct 31 12:53:00.638964 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.637468    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2705ec7-400e-4265-b213-5b58ba42cd4c-istiod-ca-cert\") pod \"e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-vqcct\" (UID: \"b2705ec7-400e-4265-b213-5b58ba42cd4c\") " pod="openshift-ingress/e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-vqcct"
Oct 31 12:53:00.638964 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.637522    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bspb4\" (UniqueName: \"kubernetes.io/projected/0d62f8e4-b062-4dd4-a23d-0f66566c83da-kube-api-access-bspb4\") pod \"e2e-inf-a-e98df8-openshift-default-779d98479-vkrj6\" (UID: \"0d62f8e4-b062-4dd4-a23d-0f66566c83da\") " pod="openshift-ingress/e2e-inf-a-e98df8-openshift-default-779d98479-vkrj6"
Oct 31 12:53:00.638964 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.637557    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2705ec7-400e-4265-b213-5b58ba42cd4c-credential-socket\") pod \"e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-vqcct\" (UID: \"b2705ec7-400e-4265-b213-5b58ba42cd4c\") " pod="openshift-ingress/e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-vqcct"
Oct 31 12:53:00.638964 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.637592    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2705ec7-400e-4265-b213-5b58ba42cd4c-istio-data\") pod \"e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-vqcct\" (UID: \"b2705ec7-400e-4265-b213-5b58ba42cd4c\") " pod="openshift-ingress/e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-vqcct"
Oct 31 12:53:00.638964 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.637621    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02360407-0dfa-45b7-b276-6487dcfb02f0-workload-socket" pod "02360407-0dfa-45b7-b276-6487dcfb02f0" (UID: "02360407-0dfa-45b7-b276-6487dcfb02f0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:53:00.638964 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.637770    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d62f8e4-b062-4dd4-a23d-0f66566c83da-istio-envoy\") pod \"e2e-inf-a-e98df8-openshift-default-779d98479-vkrj6\" (UID: \"0d62f8e4-b062-4dd4-a23d-0f66566c83da\") " pod="openshift-ingress/e2e-inf-a-e98df8-openshift-default-779d98479-vkrj6"
Oct 31 12:53:00.638964 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.637807    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d62f8e4-b062-4dd4-a23d-0f66566c83da-istiod-ca-cert\") pod \"e2e-inf-a-e98df8-openshift-default-779d98479-vkrj6\" (UID: \"0d62f8e4-b062-4dd4-a23d-0f66566c83da\") " pod="openshift-ingress/e2e-inf-a-e98df8-openshift-default-779d98479-vkrj6"
Oct 31 12:53:00.638964 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.637836    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d62f8e4-b062-4dd4-a23d-0f66566c83da-workload-socket\") pod \"e2e-inf-a-e98df8-openshift-default-779d98479-vkrj6\" (UID: \"0d62f8e4-b062-4dd4-a23d-0f66566c83da\") " pod="openshift-ingress/e2e-inf-a-e98df8-openshift-default-779d98479-vkrj6"
Oct 31 12:53:00.638964 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.637907    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d62f8e4-b062-4dd4-a23d-0f66566c83da-istio-token\") pod \"e2e-inf-a-e98df8-openshift-default-779d98479-vkrj6\" (UID: \"0d62f8e4-b062-4dd4-a23d-0f66566c83da\") " pod="openshift-ingress/e2e-inf-a-e98df8-openshift-default-779d98479-vkrj6"
Oct 31 12:53:00.638964 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.637964    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d62f8e4-b062-4dd4-a23d-0f66566c83da-workload-certs\") pod \"e2e-inf-a-e98df8-openshift-default-779d98479-vkrj6\" (UID: \"0d62f8e4-b062-4dd4-a23d-0f66566c83da\") " pod="openshift-ingress/e2e-inf-a-e98df8-openshift-default-779d98479-vkrj6"
Oct 31 12:53:00.638964 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.637998    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2705ec7-400e-4265-b213-5b58ba42cd4c-istio-data\") pod \"e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-vqcct\" (UID: \"b2705ec7-400e-4265-b213-5b58ba42cd4c\") " pod="openshift-ingress/e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-vqcct"
Oct 31 12:53:00.638964 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.638031    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02360407-0dfa-45b7-b276-6487dcfb02f0-credential-socket" pod "02360407-0dfa-45b7-b276-6487dcfb02f0" (UID: "02360407-0dfa-45b7-b276-6487dcfb02f0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:53:00.638964 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.638198    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02360407-0dfa-45b7-b276-6487dcfb02f0-istiod-ca-cert" pod "02360407-0dfa-45b7-b276-6487dcfb02f0" (UID: "02360407-0dfa-45b7-b276-6487dcfb02f0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:53:00.649844 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.639008    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02360407-0dfa-45b7-b276-6487dcfb02f0-istio-data" pod "02360407-0dfa-45b7-b276-6487dcfb02f0" (UID: "02360407-0dfa-45b7-b276-6487dcfb02f0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:53:00.649844 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.639242    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2705ec7-400e-4265-b213-5b58ba42cd4c-workload-certs\") pod \"e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-vqcct\" (UID: \"b2705ec7-400e-4265-b213-5b58ba42cd4c\") " pod="openshift-ingress/e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-vqcct"
Oct 31 12:53:00.649844 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.641352    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2705ec7-400e-4265-b213-5b58ba42cd4c-workload-socket\") pod \"e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-vqcct\" (UID: \"b2705ec7-400e-4265-b213-5b58ba42cd4c\") " pod="openshift-ingress/e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-vqcct"
Oct 31 12:53:00.649844 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.641911    5060 generic.go:356] "Generic (PLEG): container finished" podID="38bdca08-eb2b-43c5-8c07-7059e5c9e199" containerID="4c7d3ca2a9c8a503e740a602271d240f30dea66d2fed340dac15ec3cb71094ef" exitCode=0
Oct 31 12:53:00.649844 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.642005    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-sv4w2" event={"ID":"38bdca08-eb2b-43c5-8c07-7059e5c9e199","Type":"ContainerDied","Data":"4c7d3ca2a9c8a503e740a602271d240f30dea66d2fed340dac15ec3cb71094ef"}
Oct 31 12:53:00.649844 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.647803    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2705ec7-400e-4265-b213-5b58ba42cd4c-credential-socket\") pod \"e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-vqcct\" (UID: \"b2705ec7-400e-4265-b213-5b58ba42cd4c\") " pod="openshift-ingress/e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-vqcct"
Oct 31 12:53:00.649844 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.648000    5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ade94c0-4cba-4ccd-87c0-cdd0cc2aa72d-istio-token\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:00.649844 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.648025    5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02360407-0dfa-45b7-b276-6487dcfb02f0-workload-socket\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:00.649844 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.648046    5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ade94c0-4cba-4ccd-87c0-cdd0cc2aa72d-workload-socket\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:00.649844 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.648063    5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ade94c0-4cba-4ccd-87c0-cdd0cc2aa72d-workload-certs\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:00.649844 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.648078    5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ade94c0-4cba-4ccd-87c0-cdd0cc2aa72d-credential-socket\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:00.649844 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.648095    5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ade94c0-4cba-4ccd-87c0-cdd0cc2aa72d-istiod-ca-cert\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:00.649844 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:53:00.649790    5060 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/0ade94c0-4cba-4ccd-87c0-cdd0cc2aa72d/volumes/kubernetes.io~empty-dir/istio-envoy
Oct 31 12:53:00.649844 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.649815    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ade94c0-4cba-4ccd-87c0-cdd0cc2aa72d-istio-envoy" pod "0ade94c0-4cba-4ccd-87c0-cdd0cc2aa72d" (UID: "0ade94c0-4cba-4ccd-87c0-cdd0cc2aa72d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:53:00.650687 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.650597    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2705ec7-400e-4265-b213-5b58ba42cd4c-istio-envoy\") pod \"e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-vqcct\" (UID: \"b2705ec7-400e-4265-b213-5b58ba42cd4c\") " pod="openshift-ingress/e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-vqcct"
Oct 31 12:53:00.651360 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.651246    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2705ec7-400e-4265-b213-5b58ba42cd4c-istio-podinfo\") pod \"e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-vqcct\" (UID: \"b2705ec7-400e-4265-b213-5b58ba42cd4c\") " pod="openshift-ingress/e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-vqcct"
Oct 31 12:53:00.671338 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.653229    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/02360407-0dfa-45b7-b276-6487dcfb02f0-istio-podinfo" pod "02360407-0dfa-45b7-b276-6487dcfb02f0" (UID: "02360407-0dfa-45b7-b276-6487dcfb02f0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 12:53:00.671338 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.660805    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02360407-0dfa-45b7-b276-6487dcfb02f0-workload-certs" pod "02360407-0dfa-45b7-b276-6487dcfb02f0" (UID: "02360407-0dfa-45b7-b276-6487dcfb02f0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:53:00.710668 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.687069    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2705ec7-400e-4265-b213-5b58ba42cd4c-istio-token\") pod \"e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-vqcct\" (UID: \"b2705ec7-400e-4265-b213-5b58ba42cd4c\") " pod="openshift-ingress/e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-vqcct"
Oct 31 12:53:00.710668 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.706784    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02360407-0dfa-45b7-b276-6487dcfb02f0-kube-api-access-986cp" pod "02360407-0dfa-45b7-b276-6487dcfb02f0" (UID: "02360407-0dfa-45b7-b276-6487dcfb02f0"). InnerVolumeSpecName "kube-api-access-986cp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:53:00.712195 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.711072    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02360407-0dfa-45b7-b276-6487dcfb02f0-istio-envoy" pod "02360407-0dfa-45b7-b276-6487dcfb02f0" (UID: "02360407-0dfa-45b7-b276-6487dcfb02f0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:53:00.712195 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.711143    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0ade94c0-4cba-4ccd-87c0-cdd0cc2aa72d-istio-podinfo" pod "0ade94c0-4cba-4ccd-87c0-cdd0cc2aa72d" (UID: "0ade94c0-4cba-4ccd-87c0-cdd0cc2aa72d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 12:53:00.714425 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.713633    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02360407-0dfa-45b7-b276-6487dcfb02f0-istio-token" pod "02360407-0dfa-45b7-b276-6487dcfb02f0" (UID: "02360407-0dfa-45b7-b276-6487dcfb02f0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:53:00.719167 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.718897    5060 scope.go:122] "RemoveContainer" containerID="b1792e1e8e3a369422a6939d613bdc1025975b26261bb7666cbc4972dcae2ebd"
Oct 31 12:53:00.727512 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.719689    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2705ec7-400e-4265-b213-5b58ba42cd4c-istiod-ca-cert\") pod \"e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-vqcct\" (UID: \"b2705ec7-400e-4265-b213-5b58ba42cd4c\") " pod="openshift-ingress/e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-vqcct"
Oct 31 12:53:00.737990 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.730677    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-zg66r"]
Oct 31 12:53:00.737990 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.732058    5060 cpu_manager.go:414] "containerMap: removing container" podUID="02360407-0dfa-45b7-b276-6487dcfb02f0" containerName="istio-proxy"
Oct 31 12:53:00.737990 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.732081    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02360407-0dfa-45b7-b276-6487dcfb02f0" containerName="istio-proxy"
Oct 31 12:53:00.737990 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.732136    5060 cpu_manager.go:414] "containerMap: removing container" podUID="02360407-0dfa-45b7-b276-6487dcfb02f0" containerName="istio-proxy"
Oct 31 12:53:00.737990 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.732144    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02360407-0dfa-45b7-b276-6487dcfb02f0" containerName="istio-proxy"
Oct 31 12:53:00.737990 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.732348    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="02360407-0dfa-45b7-b276-6487dcfb02f0" containerName="istio-proxy"
Oct 31 12:53:00.737990 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.732370    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="02360407-0dfa-45b7-b276-6487dcfb02f0" containerName="istio-proxy"
Oct 31 12:53:00.740991 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:53:00.739679    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1792e1e8e3a369422a6939d613bdc1025975b26261bb7666cbc4972dcae2ebd\": container with ID starting with b1792e1e8e3a369422a6939d613bdc1025975b26261bb7666cbc4972dcae2ebd not found: ID does not exist" containerID="b1792e1e8e3a369422a6939d613bdc1025975b26261bb7666cbc4972dcae2ebd"
Oct 31 12:53:00.740991 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.739722    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1792e1e8e3a369422a6939d613bdc1025975b26261bb7666cbc4972dcae2ebd"} err="failed to get container status \"b1792e1e8e3a369422a6939d613bdc1025975b26261bb7666cbc4972dcae2ebd\": rpc error: code = NotFound desc = could not find container \"b1792e1e8e3a369422a6939d613bdc1025975b26261bb7666cbc4972dcae2ebd\": container with ID starting with b1792e1e8e3a369422a6939d613bdc1025975b26261bb7666cbc4972dcae2ebd not found: ID does not exist"
Oct 31 12:53:00.740991 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.739748    5060 scope.go:122] "RemoveContainer" containerID="7c91c181e45356241d7139f6aeedfaae2dd5b0a8dd32e6953ea1f26c79face5f"
Oct 31 12:53:00.740991 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.740370    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e43d8b-765bcc4ffb-g4s6x"
Oct 31 12:53:00.741265 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.741086    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ade94c0-4cba-4ccd-87c0-cdd0cc2aa72d-kube-api-access-gfj75" pod "0ade94c0-4cba-4ccd-87c0-cdd0cc2aa72d" (UID: "0ade94c0-4cba-4ccd-87c0-cdd0cc2aa72d"). InnerVolumeSpecName "kube-api-access-gfj75". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:53:00.742170 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.742118    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4l6w\" (UniqueName: \"kubernetes.io/projected/b2705ec7-400e-4265-b213-5b58ba42cd4c-kube-api-access-p4l6w\") pod \"e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-vqcct\" (UID: \"b2705ec7-400e-4265-b213-5b58ba42cd4c\") " pod="openshift-ingress/e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-vqcct"
Oct 31 12:53:00.749061 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.749038    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-zg66r"]
Oct 31 12:53:00.749174 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.749089    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e43d8b-765bcc4ffb-g4s6x" event={"ID":"dec1f857-9b34-458f-8954-56d2753466db","Type":"ContainerDied","Data":"6ed65c0b483f070f3f7aa452be1cab42bbb3e9d9363fa146ba884b44f66d70d4"}
Oct 31 12:53:00.749258 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.749188    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-zg66r"
Oct 31 12:53:00.749421 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.749190    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d62f8e4-b062-4dd4-a23d-0f66566c83da-workload-certs\") pod \"e2e-inf-a-e98df8-openshift-default-779d98479-vkrj6\" (UID: \"0d62f8e4-b062-4dd4-a23d-0f66566c83da\") " pod="openshift-ingress/e2e-inf-a-e98df8-openshift-default-779d98479-vkrj6"
Oct 31 12:53:00.749607 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.749558    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d62f8e4-b062-4dd4-a23d-0f66566c83da-workload-certs\") pod \"e2e-inf-a-e98df8-openshift-default-779d98479-vkrj6\" (UID: \"0d62f8e4-b062-4dd4-a23d-0f66566c83da\") " pod="openshift-ingress/e2e-inf-a-e98df8-openshift-default-779d98479-vkrj6"
Oct 31 12:53:00.749737 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.749720    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d62f8e4-b062-4dd4-a23d-0f66566c83da-credential-socket\") pod \"e2e-inf-a-e98df8-openshift-default-779d98479-vkrj6\" (UID: \"0d62f8e4-b062-4dd4-a23d-0f66566c83da\") " pod="openshift-ingress/e2e-inf-a-e98df8-openshift-default-779d98479-vkrj6"
Oct 31 12:53:00.750003 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.749944    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d62f8e4-b062-4dd4-a23d-0f66566c83da-istio-data\") pod \"e2e-inf-a-e98df8-openshift-default-779d98479-vkrj6\" (UID: \"0d62f8e4-b062-4dd4-a23d-0f66566c83da\") " pod="openshift-ingress/e2e-inf-a-e98df8-openshift-default-779d98479-vkrj6"
Oct 31 12:53:00.750178 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.750024    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d62f8e4-b062-4dd4-a23d-0f66566c83da-istio-podinfo\") pod \"e2e-inf-a-e98df8-openshift-default-779d98479-vkrj6\" (UID: \"0d62f8e4-b062-4dd4-a23d-0f66566c83da\") " pod="openshift-ingress/e2e-inf-a-e98df8-openshift-default-779d98479-vkrj6"
Oct 31 12:53:00.750178 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.750070    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d62f8e4-b062-4dd4-a23d-0f66566c83da-credential-socket\") pod \"e2e-inf-a-e98df8-openshift-default-779d98479-vkrj6\" (UID: \"0d62f8e4-b062-4dd4-a23d-0f66566c83da\") " pod="openshift-ingress/e2e-inf-a-e98df8-openshift-default-779d98479-vkrj6"
Oct 31 12:53:00.750178 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.750124    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bspb4\" (UniqueName: \"kubernetes.io/projected/0d62f8e4-b062-4dd4-a23d-0f66566c83da-kube-api-access-bspb4\") pod \"e2e-inf-a-e98df8-openshift-default-779d98479-vkrj6\" (UID: \"0d62f8e4-b062-4dd4-a23d-0f66566c83da\") " pod="openshift-ingress/e2e-inf-a-e98df8-openshift-default-779d98479-vkrj6"
Oct 31 12:53:00.750364 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.750225    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d62f8e4-b062-4dd4-a23d-0f66566c83da-istio-envoy\") pod \"e2e-inf-a-e98df8-openshift-default-779d98479-vkrj6\" (UID: \"0d62f8e4-b062-4dd4-a23d-0f66566c83da\") " pod="openshift-ingress/e2e-inf-a-e98df8-openshift-default-779d98479-vkrj6"
Oct 31 12:53:00.750364 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.750256    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d62f8e4-b062-4dd4-a23d-0f66566c83da-istiod-ca-cert\") pod \"e2e-inf-a-e98df8-openshift-default-779d98479-vkrj6\" (UID: \"0d62f8e4-b062-4dd4-a23d-0f66566c83da\") " pod="openshift-ingress/e2e-inf-a-e98df8-openshift-default-779d98479-vkrj6"
Oct 31 12:53:00.750364 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.750278    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d62f8e4-b062-4dd4-a23d-0f66566c83da-workload-socket\") pod \"e2e-inf-a-e98df8-openshift-default-779d98479-vkrj6\" (UID: \"0d62f8e4-b062-4dd4-a23d-0f66566c83da\") " pod="openshift-ingress/e2e-inf-a-e98df8-openshift-default-779d98479-vkrj6"
Oct 31 12:53:00.750364 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.750330    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d62f8e4-b062-4dd4-a23d-0f66566c83da-istio-token\") pod \"e2e-inf-a-e98df8-openshift-default-779d98479-vkrj6\" (UID: \"0d62f8e4-b062-4dd4-a23d-0f66566c83da\") " pod="openshift-ingress/e2e-inf-a-e98df8-openshift-default-779d98479-vkrj6"
Oct 31 12:53:00.751472 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.750283    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d62f8e4-b062-4dd4-a23d-0f66566c83da-istio-data\") pod \"e2e-inf-a-e98df8-openshift-default-779d98479-vkrj6\" (UID: \"0d62f8e4-b062-4dd4-a23d-0f66566c83da\") " pod="openshift-ingress/e2e-inf-a-e98df8-openshift-default-779d98479-vkrj6"
Oct 31 12:53:00.751472 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.750902    5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02360407-0dfa-45b7-b276-6487dcfb02f0-istio-token\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:00.751472 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.751174    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d62f8e4-b062-4dd4-a23d-0f66566c83da-workload-socket\") pod \"e2e-inf-a-e98df8-openshift-default-779d98479-vkrj6\" (UID: \"0d62f8e4-b062-4dd4-a23d-0f66566c83da\") " pod="openshift-ingress/e2e-inf-a-e98df8-openshift-default-779d98479-vkrj6"
Oct 31 12:53:00.751472 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.751207    5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02360407-0dfa-45b7-b276-6487dcfb02f0-workload-certs\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:00.751472 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.751225    5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ade94c0-4cba-4ccd-87c0-cdd0cc2aa72d-istio-podinfo\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:00.751472 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.751232    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d62f8e4-b062-4dd4-a23d-0f66566c83da-istiod-ca-cert\") pod \"e2e-inf-a-e98df8-openshift-default-779d98479-vkrj6\" (UID: \"0d62f8e4-b062-4dd4-a23d-0f66566c83da\") " pod="openshift-ingress/e2e-inf-a-e98df8-openshift-default-779d98479-vkrj6"
Oct 31 12:53:00.751472 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.751242    5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02360407-0dfa-45b7-b276-6487dcfb02f0-istio-podinfo\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:00.751472 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.751271    5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ade94c0-4cba-4ccd-87c0-cdd0cc2aa72d-istio-envoy\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:00.751472 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.751286    5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02360407-0dfa-45b7-b276-6487dcfb02f0-istio-envoy\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:00.751472 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.751300    5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02360407-0dfa-45b7-b276-6487dcfb02f0-istiod-ca-cert\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:00.751472 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.751316    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gfj75\" (UniqueName: \"kubernetes.io/projected/0ade94c0-4cba-4ccd-87c0-cdd0cc2aa72d-kube-api-access-gfj75\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:00.751472 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.751332    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-986cp\" (UniqueName: \"kubernetes.io/projected/02360407-0dfa-45b7-b276-6487dcfb02f0-kube-api-access-986cp\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:00.751472 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.751348    5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02360407-0dfa-45b7-b276-6487dcfb02f0-credential-socket\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:00.751472 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.751362    5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02360407-0dfa-45b7-b276-6487dcfb02f0-istio-data\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:00.753067 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.753022    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d62f8e4-b062-4dd4-a23d-0f66566c83da-istio-envoy\") pod \"e2e-inf-a-e98df8-openshift-default-779d98479-vkrj6\" (UID: \"0d62f8e4-b062-4dd4-a23d-0f66566c83da\") " pod="openshift-ingress/e2e-inf-a-e98df8-openshift-default-779d98479-vkrj6"
Oct 31 12:53:00.753595 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.753524    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d62f8e4-b062-4dd4-a23d-0f66566c83da-istio-podinfo\") pod \"e2e-inf-a-e98df8-openshift-default-779d98479-vkrj6\" (UID: \"0d62f8e4-b062-4dd4-a23d-0f66566c83da\") " pod="openshift-ingress/e2e-inf-a-e98df8-openshift-default-779d98479-vkrj6"
Oct 31 12:53:00.772693 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.766250    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-sv4w2"
Oct 31 12:53:00.772693 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.771969    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-sc6ff"
Oct 31 12:53:00.806102 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.788376    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bspb4\" (UniqueName: \"kubernetes.io/projected/0d62f8e4-b062-4dd4-a23d-0f66566c83da-kube-api-access-bspb4\") pod \"e2e-inf-a-e98df8-openshift-default-779d98479-vkrj6\" (UID: \"0d62f8e4-b062-4dd4-a23d-0f66566c83da\") " pod="openshift-ingress/e2e-inf-a-e98df8-openshift-default-779d98479-vkrj6"
Oct 31 12:53:00.806102 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.790938    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d62f8e4-b062-4dd4-a23d-0f66566c83da-istio-token\") pod \"e2e-inf-a-e98df8-openshift-default-779d98479-vkrj6\" (UID: \"0d62f8e4-b062-4dd4-a23d-0f66566c83da\") " pod="openshift-ingress/e2e-inf-a-e98df8-openshift-default-779d98479-vkrj6"
Oct 31 12:53:00.806102 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.801337    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-gxs26"
Oct 31 12:53:00.806102 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.803393    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-njl9z"
Oct 31 12:53:00.818697 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.816299    5060 scope.go:122] "RemoveContainer" containerID="bf1be783bcbba3e5a2ef5eb2d7ebf98c19e54cbde409381b7e1695dd214e1156"
Oct 31 12:53:00.839529 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.839478    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-ds5zz"]
Oct 31 12:53:00.840710 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.840686    5060 scope.go:122] "RemoveContainer" containerID="7c91c181e45356241d7139f6aeedfaae2dd5b0a8dd32e6953ea1f26c79face5f"
Oct 31 12:53:00.841429 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.840838    5060 cpu_manager.go:414] "containerMap: removing container" podUID="38bdca08-eb2b-43c5-8c07-7059e5c9e199" containerName="istio-proxy"
Oct 31 12:53:00.841429 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.840864    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38bdca08-eb2b-43c5-8c07-7059e5c9e199" containerName="istio-proxy"
Oct 31 12:53:00.841429 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.840883    5060 cpu_manager.go:414] "containerMap: removing container" podUID="360843c7-795f-4f22-84cd-76317addf9c2" containerName="istio-proxy"
Oct 31 12:53:00.841429 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.840891    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="360843c7-795f-4f22-84cd-76317addf9c2" containerName="istio-proxy"
Oct 31 12:53:00.841429 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.840924    5060 cpu_manager.go:414] "containerMap: removing container" podUID="86c84864-127f-4507-8f60-2dbb684c687a" containerName="istio-proxy"
Oct 31 12:53:00.841429 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.840931    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86c84864-127f-4507-8f60-2dbb684c687a" containerName="istio-proxy"
Oct 31 12:53:00.841429 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.840952    5060 cpu_manager.go:414] "containerMap: removing container" podUID="29b176fa-229b-433e-928f-41daf33d1f37" containerName="istio-proxy"
Oct 31 12:53:00.841429 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.840959    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29b176fa-229b-433e-928f-41daf33d1f37" containerName="istio-proxy"
Oct 31 12:53:00.841429 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.841063    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="86c84864-127f-4507-8f60-2dbb684c687a" containerName="istio-proxy"
Oct 31 12:53:00.841429 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.841072    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="29b176fa-229b-433e-928f-41daf33d1f37" containerName="istio-proxy"
Oct 31 12:53:00.841429 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.841084    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="38bdca08-eb2b-43c5-8c07-7059e5c9e199" containerName="istio-proxy"
Oct 31 12:53:00.841429 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.841091    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="360843c7-795f-4f22-84cd-76317addf9c2" containerName="istio-proxy"
Oct 31 12:53:00.841429 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:53:00.841101    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c91c181e45356241d7139f6aeedfaae2dd5b0a8dd32e6953ea1f26c79face5f\": container with ID starting with 7c91c181e45356241d7139f6aeedfaae2dd5b0a8dd32e6953ea1f26c79face5f not found: ID does not exist" containerID="7c91c181e45356241d7139f6aeedfaae2dd5b0a8dd32e6953ea1f26c79face5f"
Oct 31 12:53:00.841429 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.841125    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c91c181e45356241d7139f6aeedfaae2dd5b0a8dd32e6953ea1f26c79face5f"} err="failed to get container status \"7c91c181e45356241d7139f6aeedfaae2dd5b0a8dd32e6953ea1f26c79face5f\": rpc error: code = NotFound desc = could not find container \"7c91c181e45356241d7139f6aeedfaae2dd5b0a8dd32e6953ea1f26c79face5f\": container with ID starting with 7c91c181e45356241d7139f6aeedfaae2dd5b0a8dd32e6953ea1f26c79face5f not found: ID does not exist"
Oct 31 12:53:00.841429 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.841143    5060 scope.go:122] "RemoveContainer" containerID="bf1be783bcbba3e5a2ef5eb2d7ebf98c19e54cbde409381b7e1695dd214e1156"
Oct 31 12:53:00.841429 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:53:00.841399    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf1be783bcbba3e5a2ef5eb2d7ebf98c19e54cbde409381b7e1695dd214e1156\": container with ID starting with bf1be783bcbba3e5a2ef5eb2d7ebf98c19e54cbde409381b7e1695dd214e1156 not found: ID does not exist" containerID="bf1be783bcbba3e5a2ef5eb2d7ebf98c19e54cbde409381b7e1695dd214e1156"
Oct 31 12:53:00.841429 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.841426    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf1be783bcbba3e5a2ef5eb2d7ebf98c19e54cbde409381b7e1695dd214e1156"} err="failed to get container status \"bf1be783bcbba3e5a2ef5eb2d7ebf98c19e54cbde409381b7e1695dd214e1156\": rpc error: code = NotFound desc = could not find container \"bf1be783bcbba3e5a2ef5eb2d7ebf98c19e54cbde409381b7e1695dd214e1156\": container with ID starting with bf1be783bcbba3e5a2ef5eb2d7ebf98c19e54cbde409381b7e1695dd214e1156 not found: ID does not exist"
Oct 31 12:53:00.842827 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.841443    5060 scope.go:122] "RemoveContainer" containerID="899bb28709d691b64688ef03645611a5e6dc2d0ef12a447bd5722fa22622a316"
Oct 31 12:53:00.847459 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.847437    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-ds5zz"
Oct 31 12:53:00.899938 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.874693    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4rz4\" (UniqueName: \"kubernetes.io/projected/ea38d319-4637-47c0-bf7a-125b61dac2fe-kube-api-access-v4rz4\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-zg66r\" (UID: \"ea38d319-4637-47c0-bf7a-125b61dac2fe\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-zg66r"
Oct 31 12:53:00.899938 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.874799    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea38d319-4637-47c0-bf7a-125b61dac2fe-istiod-ca-cert\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-zg66r\" (UID: \"ea38d319-4637-47c0-bf7a-125b61dac2fe\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-zg66r"
Oct 31 12:53:00.899938 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.874902    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea38d319-4637-47c0-bf7a-125b61dac2fe-istio-token\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-zg66r\" (UID: \"ea38d319-4637-47c0-bf7a-125b61dac2fe\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-zg66r"
Oct 31 12:53:00.899938 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.875028    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea38d319-4637-47c0-bf7a-125b61dac2fe-workload-socket\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-zg66r\" (UID: \"ea38d319-4637-47c0-bf7a-125b61dac2fe\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-zg66r"
Oct 31 12:53:00.899938 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.875068    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea38d319-4637-47c0-bf7a-125b61dac2fe-workload-certs\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-zg66r\" (UID: \"ea38d319-4637-47c0-bf7a-125b61dac2fe\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-zg66r"
Oct 31 12:53:00.899938 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.875111    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea38d319-4637-47c0-bf7a-125b61dac2fe-istio-envoy\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-zg66r\" (UID: \"ea38d319-4637-47c0-bf7a-125b61dac2fe\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-zg66r"
Oct 31 12:53:00.899938 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.877076    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea38d319-4637-47c0-bf7a-125b61dac2fe-istio-data\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-zg66r\" (UID: \"ea38d319-4637-47c0-bf7a-125b61dac2fe\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-zg66r"
Oct 31 12:53:00.899938 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.877155    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea38d319-4637-47c0-bf7a-125b61dac2fe-istio-podinfo\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-zg66r\" (UID: \"ea38d319-4637-47c0-bf7a-125b61dac2fe\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-zg66r"
Oct 31 12:53:00.899938 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.877203    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea38d319-4637-47c0-bf7a-125b61dac2fe-credential-socket\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-zg66r\" (UID: \"ea38d319-4637-47c0-bf7a-125b61dac2fe\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-zg66r"
Oct 31 12:53:00.899938 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.890551    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e43d8b-765bcc4ffb-g4s6x"]
Oct 31 12:53:00.907620 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.907592    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-ds5zz"]
Oct 31 12:53:00.920793 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.917827    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-vqcct"
Oct 31 12:53:00.926437 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.926414    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e43d8b-765bcc4ffb-g4s6x"]
Oct 31 12:53:01.066532 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.988094    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29b176fa-229b-433e-928f-41daf33d1f37-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29b176fa-229b-433e-928f-41daf33d1f37-workload-certs\") pod \"29b176fa-229b-433e-928f-41daf33d1f37\" (UID: \"29b176fa-229b-433e-928f-41daf33d1f37\") "
Oct 31 12:53:01.066532 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.988156    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/38bdca08-eb2b-43c5-8c07-7059e5c9e199-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38bdca08-eb2b-43c5-8c07-7059e5c9e199-istio-podinfo\") pod \"38bdca08-eb2b-43c5-8c07-7059e5c9e199\" (UID: \"38bdca08-eb2b-43c5-8c07-7059e5c9e199\") "
Oct 31 12:53:01.066532 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.988188    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29b176fa-229b-433e-928f-41daf33d1f37-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29b176fa-229b-433e-928f-41daf33d1f37-workload-socket\") pod \"29b176fa-229b-433e-928f-41daf33d1f37\" (UID: \"29b176fa-229b-433e-928f-41daf33d1f37\") "
Oct 31 12:53:01.066532 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.988221    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/360843c7-795f-4f22-84cd-76317addf9c2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/360843c7-795f-4f22-84cd-76317addf9c2-credential-socket\") pod \"360843c7-795f-4f22-84cd-76317addf9c2\" (UID: \"360843c7-795f-4f22-84cd-76317addf9c2\") "
Oct 31 12:53:01.066532 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.988256    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38bdca08-eb2b-43c5-8c07-7059e5c9e199-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38bdca08-eb2b-43c5-8c07-7059e5c9e199-istiod-ca-cert\") pod \"38bdca08-eb2b-43c5-8c07-7059e5c9e199\" (UID: \"38bdca08-eb2b-43c5-8c07-7059e5c9e199\") "
Oct 31 12:53:01.066532 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.988287    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38bdca08-eb2b-43c5-8c07-7059e5c9e199-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38bdca08-eb2b-43c5-8c07-7059e5c9e199-workload-certs\") pod \"38bdca08-eb2b-43c5-8c07-7059e5c9e199\" (UID: \"38bdca08-eb2b-43c5-8c07-7059e5c9e199\") "
Oct 31 12:53:01.066532 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.988315    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29b176fa-229b-433e-928f-41daf33d1f37-kube-api-access-g9rjz\" (UniqueName: \"kubernetes.io/projected/29b176fa-229b-433e-928f-41daf33d1f37-kube-api-access-g9rjz\") pod \"29b176fa-229b-433e-928f-41daf33d1f37\" (UID: \"29b176fa-229b-433e-928f-41daf33d1f37\") "
Oct 31 12:53:01.066532 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.988390    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/360843c7-795f-4f22-84cd-76317addf9c2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/360843c7-795f-4f22-84cd-76317addf9c2-istio-envoy\") pod \"360843c7-795f-4f22-84cd-76317addf9c2\" (UID: \"360843c7-795f-4f22-84cd-76317addf9c2\") "
Oct 31 12:53:01.066532 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.988422    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38bdca08-eb2b-43c5-8c07-7059e5c9e199-istio-token\" (UniqueName: \"kubernetes.io/projected/38bdca08-eb2b-43c5-8c07-7059e5c9e199-istio-token\") pod \"38bdca08-eb2b-43c5-8c07-7059e5c9e199\" (UID: \"38bdca08-eb2b-43c5-8c07-7059e5c9e199\") "
Oct 31 12:53:01.066532 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.988461    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/360843c7-795f-4f22-84cd-76317addf9c2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/360843c7-795f-4f22-84cd-76317addf9c2-istiod-ca-cert\") pod \"360843c7-795f-4f22-84cd-76317addf9c2\" (UID: \"360843c7-795f-4f22-84cd-76317addf9c2\") "
Oct 31 12:53:01.066532 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.988518    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29b176fa-229b-433e-928f-41daf33d1f37-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29b176fa-229b-433e-928f-41daf33d1f37-istio-envoy\") pod \"29b176fa-229b-433e-928f-41daf33d1f37\" (UID: \"29b176fa-229b-433e-928f-41daf33d1f37\") "
Oct 31 12:53:01.066532 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.988577    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86c84864-127f-4507-8f60-2dbb684c687a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86c84864-127f-4507-8f60-2dbb684c687a-workload-certs\") pod \"86c84864-127f-4507-8f60-2dbb684c687a\" (UID: \"86c84864-127f-4507-8f60-2dbb684c687a\") "
Oct 31 12:53:01.066532 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.988615    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/86c84864-127f-4507-8f60-2dbb684c687a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86c84864-127f-4507-8f60-2dbb684c687a-istio-podinfo\") pod \"86c84864-127f-4507-8f60-2dbb684c687a\" (UID: \"86c84864-127f-4507-8f60-2dbb684c687a\") "
Oct 31 12:53:01.066532 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.988645    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86c84864-127f-4507-8f60-2dbb684c687a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86c84864-127f-4507-8f60-2dbb684c687a-credential-socket\") pod \"86c84864-127f-4507-8f60-2dbb684c687a\" (UID: \"86c84864-127f-4507-8f60-2dbb684c687a\") "
Oct 31 12:53:01.066532 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.988682    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29b176fa-229b-433e-928f-41daf33d1f37-istio-token\" (UniqueName: \"kubernetes.io/projected/29b176fa-229b-433e-928f-41daf33d1f37-istio-token\") pod \"29b176fa-229b-433e-928f-41daf33d1f37\" (UID: \"29b176fa-229b-433e-928f-41daf33d1f37\") "
Oct 31 12:53:01.066532 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.988734    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/360843c7-795f-4f22-84cd-76317addf9c2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/360843c7-795f-4f22-84cd-76317addf9c2-istio-podinfo\") pod \"360843c7-795f-4f22-84cd-76317addf9c2\" (UID: \"360843c7-795f-4f22-84cd-76317addf9c2\") "
Oct 31 12:53:01.066532 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.988763    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38bdca08-eb2b-43c5-8c07-7059e5c9e199-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38bdca08-eb2b-43c5-8c07-7059e5c9e199-istio-data\") pod \"38bdca08-eb2b-43c5-8c07-7059e5c9e199\" (UID: \"38bdca08-eb2b-43c5-8c07-7059e5c9e199\") "
Oct 31 12:53:01.066532 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.988825    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86c84864-127f-4507-8f60-2dbb684c687a-istio-token\" (UniqueName: \"kubernetes.io/projected/86c84864-127f-4507-8f60-2dbb684c687a-istio-token\") pod \"86c84864-127f-4507-8f60-2dbb684c687a\" (UID: \"86c84864-127f-4507-8f60-2dbb684c687a\") "
Oct 31 12:53:01.066532 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.988858    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/360843c7-795f-4f22-84cd-76317addf9c2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/360843c7-795f-4f22-84cd-76317addf9c2-istio-data\") pod \"360843c7-795f-4f22-84cd-76317addf9c2\" (UID: \"360843c7-795f-4f22-84cd-76317addf9c2\") "
Oct 31 12:53:01.066532 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.988923    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29b176fa-229b-433e-928f-41daf33d1f37-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29b176fa-229b-433e-928f-41daf33d1f37-istio-data\") pod \"29b176fa-229b-433e-928f-41daf33d1f37\" (UID: \"29b176fa-229b-433e-928f-41daf33d1f37\") "
Oct 31 12:53:01.066532 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.988980    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/360843c7-795f-4f22-84cd-76317addf9c2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/360843c7-795f-4f22-84cd-76317addf9c2-workload-certs\") pod \"360843c7-795f-4f22-84cd-76317addf9c2\" (UID: \"360843c7-795f-4f22-84cd-76317addf9c2\") "
Oct 31 12:53:01.066532 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.989030    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86c84864-127f-4507-8f60-2dbb684c687a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86c84864-127f-4507-8f60-2dbb684c687a-workload-socket\") pod \"86c84864-127f-4507-8f60-2dbb684c687a\" (UID: \"86c84864-127f-4507-8f60-2dbb684c687a\") "
Oct 31 12:53:01.066532 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.989081    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86c84864-127f-4507-8f60-2dbb684c687a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86c84864-127f-4507-8f60-2dbb684c687a-istio-data\") pod \"86c84864-127f-4507-8f60-2dbb684c687a\" (UID: \"86c84864-127f-4507-8f60-2dbb684c687a\") "
Oct 31 12:53:01.066532 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.989109    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86c84864-127f-4507-8f60-2dbb684c687a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86c84864-127f-4507-8f60-2dbb684c687a-istio-envoy\") pod \"86c84864-127f-4507-8f60-2dbb684c687a\" (UID: \"86c84864-127f-4507-8f60-2dbb684c687a\") "
Oct 31 12:53:01.066532 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.989135    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29b176fa-229b-433e-928f-41daf33d1f37-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29b176fa-229b-433e-928f-41daf33d1f37-credential-socket\") pod \"29b176fa-229b-433e-928f-41daf33d1f37\" (UID: \"29b176fa-229b-433e-928f-41daf33d1f37\") "
Oct 31 12:53:01.066532 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.989240    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38bdca08-eb2b-43c5-8c07-7059e5c9e199-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38bdca08-eb2b-43c5-8c07-7059e5c9e199-istio-envoy\") pod \"38bdca08-eb2b-43c5-8c07-7059e5c9e199\" (UID: \"38bdca08-eb2b-43c5-8c07-7059e5c9e199\") "
Oct 31 12:53:01.066532 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.989307    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/360843c7-795f-4f22-84cd-76317addf9c2-istio-token\" (UniqueName: \"kubernetes.io/projected/360843c7-795f-4f22-84cd-76317addf9c2-istio-token\") pod \"360843c7-795f-4f22-84cd-76317addf9c2\" (UID: \"360843c7-795f-4f22-84cd-76317addf9c2\") "
Oct 31 12:53:01.066532 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.989362    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86c84864-127f-4507-8f60-2dbb684c687a-kube-api-access-x42tb\" (UniqueName: \"kubernetes.io/projected/86c84864-127f-4507-8f60-2dbb684c687a-kube-api-access-x42tb\") pod \"86c84864-127f-4507-8f60-2dbb684c687a\" (UID: \"86c84864-127f-4507-8f60-2dbb684c687a\") "
Oct 31 12:53:01.066532 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.989412    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29b176fa-229b-433e-928f-41daf33d1f37-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29b176fa-229b-433e-928f-41daf33d1f37-istiod-ca-cert\") pod \"29b176fa-229b-433e-928f-41daf33d1f37\" (UID: \"29b176fa-229b-433e-928f-41daf33d1f37\") "
Oct 31 12:53:01.066532 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.989453    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/360843c7-795f-4f22-84cd-76317addf9c2-kube-api-access-mnvvb\" (UniqueName: \"kubernetes.io/projected/360843c7-795f-4f22-84cd-76317addf9c2-kube-api-access-mnvvb\") pod \"360843c7-795f-4f22-84cd-76317addf9c2\" (UID: \"360843c7-795f-4f22-84cd-76317addf9c2\") "
Oct 31 12:53:01.066532 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.989732    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38bdca08-eb2b-43c5-8c07-7059e5c9e199-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38bdca08-eb2b-43c5-8c07-7059e5c9e199-workload-socket\") pod \"38bdca08-eb2b-43c5-8c07-7059e5c9e199\" (UID: \"38bdca08-eb2b-43c5-8c07-7059e5c9e199\") "
Oct 31 12:53:01.066532 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.989808    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38bdca08-eb2b-43c5-8c07-7059e5c9e199-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38bdca08-eb2b-43c5-8c07-7059e5c9e199-credential-socket\") pod \"38bdca08-eb2b-43c5-8c07-7059e5c9e199\" (UID: \"38bdca08-eb2b-43c5-8c07-7059e5c9e199\") "
Oct 31 12:53:01.066532 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.989855    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/29b176fa-229b-433e-928f-41daf33d1f37-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29b176fa-229b-433e-928f-41daf33d1f37-istio-podinfo\") pod \"29b176fa-229b-433e-928f-41daf33d1f37\" (UID: \"29b176fa-229b-433e-928f-41daf33d1f37\") "
Oct 31 12:53:01.066532 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.989965    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/360843c7-795f-4f22-84cd-76317addf9c2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/360843c7-795f-4f22-84cd-76317addf9c2-workload-socket\") pod \"360843c7-795f-4f22-84cd-76317addf9c2\" (UID: \"360843c7-795f-4f22-84cd-76317addf9c2\") "
Oct 31 12:53:01.066532 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.990017    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86c84864-127f-4507-8f60-2dbb684c687a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86c84864-127f-4507-8f60-2dbb684c687a-istiod-ca-cert\") pod \"86c84864-127f-4507-8f60-2dbb684c687a\" (UID: \"86c84864-127f-4507-8f60-2dbb684c687a\") "
Oct 31 12:53:01.066532 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.990079    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38bdca08-eb2b-43c5-8c07-7059e5c9e199-kube-api-access-nqgl6\" (UniqueName: \"kubernetes.io/projected/38bdca08-eb2b-43c5-8c07-7059e5c9e199-kube-api-access-nqgl6\") pod \"38bdca08-eb2b-43c5-8c07-7059e5c9e199\" (UID: \"38bdca08-eb2b-43c5-8c07-7059e5c9e199\") "
Oct 31 12:53:01.066532 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.990342    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea38d319-4637-47c0-bf7a-125b61dac2fe-credential-socket\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-zg66r\" (UID: \"ea38d319-4637-47c0-bf7a-125b61dac2fe\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-zg66r"
Oct 31 12:53:01.066532 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:00.990398    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0070ea70-06e9-4733-8d11-ecb4cb993ff7-workload-socket\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-ds5zz\" (UID: \"0070ea70-06e9-4733-8d11-ecb4cb993ff7\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-ds5zz"
Oct 31 12:53:01.066532 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:01.056646    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e98df8-openshift-default-779d98479-vkrj6"
Oct 31 12:53:01.066532 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:01.057588    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/360843c7-795f-4f22-84cd-76317addf9c2-istio-data" pod "360843c7-795f-4f22-84cd-76317addf9c2" (UID: "360843c7-795f-4f22-84cd-76317addf9c2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:53:01.111067 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:01.068745    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/360843c7-795f-4f22-84cd-76317addf9c2-istiod-ca-cert" pod "360843c7-795f-4f22-84cd-76317addf9c2" (UID: "360843c7-795f-4f22-84cd-76317addf9c2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:53:01.111067 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:01.073411    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38bdca08-eb2b-43c5-8c07-7059e5c9e199-workload-certs" pod "38bdca08-eb2b-43c5-8c07-7059e5c9e199" (UID: "38bdca08-eb2b-43c5-8c07-7059e5c9e199"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:53:01.111067 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:01.073693    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29b176fa-229b-433e-928f-41daf33d1f37-workload-socket" pod "29b176fa-229b-433e-928f-41daf33d1f37" (UID: "29b176fa-229b-433e-928f-41daf33d1f37"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:53:01.111067 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:01.073884    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/360843c7-795f-4f22-84cd-76317addf9c2-credential-socket" pod "360843c7-795f-4f22-84cd-76317addf9c2" (UID: "360843c7-795f-4f22-84cd-76317addf9c2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:53:01.111067 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:01.093715    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38bdca08-eb2b-43c5-8c07-7059e5c9e199-istiod-ca-cert" pod "38bdca08-eb2b-43c5-8c07-7059e5c9e199" (UID: "38bdca08-eb2b-43c5-8c07-7059e5c9e199"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:53:01.128511 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:01.121782    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29b176fa-229b-433e-928f-41daf33d1f37-workload-certs" pod "29b176fa-229b-433e-928f-41daf33d1f37" (UID: "29b176fa-229b-433e-928f-41daf33d1f37"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:53:01.140313 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:01.123615    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38bdca08-eb2b-43c5-8c07-7059e5c9e199-istio-data" pod "38bdca08-eb2b-43c5-8c07-7059e5c9e199" (UID: "38bdca08-eb2b-43c5-8c07-7059e5c9e199"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:53:01.140807 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:01.140760    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4rz4\" (UniqueName: \"kubernetes.io/projected/ea38d319-4637-47c0-bf7a-125b61dac2fe-kube-api-access-v4rz4\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-zg66r\" (UID: \"ea38d319-4637-47c0-bf7a-125b61dac2fe\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-zg66r"
Oct 31 12:53:01.166727 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:01.141194    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea38d319-4637-47c0-bf7a-125b61dac2fe-istiod-ca-cert\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-zg66r\" (UID: \"ea38d319-4637-47c0-bf7a-125b61dac2fe\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-zg66r"
Oct 31 12:53:01.166727 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:01.141238    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0070ea70-06e9-4733-8d11-ecb4cb993ff7-istiod-ca-cert\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-ds5zz\" (UID: \"0070ea70-06e9-4733-8d11-ecb4cb993ff7\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-ds5zz"
Oct 31 12:53:01.166727 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:01.141344    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7l89\" (UniqueName: \"kubernetes.io/projected/0070ea70-06e9-4733-8d11-ecb4cb993ff7-kube-api-access-w7l89\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-ds5zz\" (UID: \"0070ea70-06e9-4733-8d11-ecb4cb993ff7\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-ds5zz"
Oct 31 12:53:01.166727 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:01.141446    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0070ea70-06e9-4733-8d11-ecb4cb993ff7-istio-data\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-ds5zz\" (UID: \"0070ea70-06e9-4733-8d11-ecb4cb993ff7\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-ds5zz"
Oct 31 12:53:01.166727 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:01.141547    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea38d319-4637-47c0-bf7a-125b61dac2fe-istio-token\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-zg66r\" (UID: \"ea38d319-4637-47c0-bf7a-125b61dac2fe\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-zg66r"
Oct 31 12:53:01.166727 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:01.141736    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0070ea70-06e9-4733-8d11-ecb4cb993ff7-workload-certs\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-ds5zz\" (UID: \"0070ea70-06e9-4733-8d11-ecb4cb993ff7\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-ds5zz"
Oct 31 12:53:01.166727 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:01.142037    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/360843c7-795f-4f22-84cd-76317addf9c2-workload-certs" pod "360843c7-795f-4f22-84cd-76317addf9c2" (UID: "360843c7-795f-4f22-84cd-76317addf9c2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:53:01.166727 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:01.142256    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86c84864-127f-4507-8f60-2dbb684c687a-workload-socket" pod "86c84864-127f-4507-8f60-2dbb684c687a" (UID: "86c84864-127f-4507-8f60-2dbb684c687a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:53:01.166727 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:01.142447    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86c84864-127f-4507-8f60-2dbb684c687a-istio-data" pod "86c84864-127f-4507-8f60-2dbb684c687a" (UID: "86c84864-127f-4507-8f60-2dbb684c687a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:53:01.166727 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:01.144095    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea38d319-4637-47c0-bf7a-125b61dac2fe-istiod-ca-cert\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-zg66r\" (UID: \"ea38d319-4637-47c0-bf7a-125b61dac2fe\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-zg66r"
Oct 31 12:53:01.166727 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:01.144665    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea38d319-4637-47c0-bf7a-125b61dac2fe-workload-socket\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-zg66r\" (UID: \"ea38d319-4637-47c0-bf7a-125b61dac2fe\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-zg66r"
Oct 31 12:53:01.166727 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:01.144736    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea38d319-4637-47c0-bf7a-125b61dac2fe-workload-certs\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-zg66r\" (UID: \"ea38d319-4637-47c0-bf7a-125b61dac2fe\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-zg66r"
Oct 31 12:53:01.166727 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:01.144818    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0070ea70-06e9-4733-8d11-ecb4cb993ff7-istio-token\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-ds5zz\" (UID: \"0070ea70-06e9-4733-8d11-ecb4cb993ff7\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-ds5zz"
Oct 31 12:53:01.166727 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:01.144865    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea38d319-4637-47c0-bf7a-125b61dac2fe-istio-envoy\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-zg66r\" (UID: \"ea38d319-4637-47c0-bf7a-125b61dac2fe\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-zg66r"
Oct 31 12:53:01.166727 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:01.145033    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0070ea70-06e9-4733-8d11-ecb4cb993ff7-istio-podinfo\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-ds5zz\" (UID: \"0070ea70-06e9-4733-8d11-ecb4cb993ff7\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-ds5zz"
Oct 31 12:53:01.166727 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:01.145092    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea38d319-4637-47c0-bf7a-125b61dac2fe-istio-data\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-zg66r\" (UID: \"ea38d319-4637-47c0-bf7a-125b61dac2fe\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-zg66r"
Oct 31 12:53:01.166727 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:01.145224    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea38d319-4637-47c0-bf7a-125b61dac2fe-istio-podinfo\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-zg66r\" (UID: \"ea38d319-4637-47c0-bf7a-125b61dac2fe\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-zg66r"
Oct 31 12:53:01.166727 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:01.145265    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0070ea70-06e9-4733-8d11-ecb4cb993ff7-credential-socket\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-ds5zz\" (UID: \"0070ea70-06e9-4733-8d11-ecb4cb993ff7\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-ds5zz"
Oct 31 12:53:01.166727 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:01.145326    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0070ea70-06e9-4733-8d11-ecb4cb993ff7-istio-envoy\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-ds5zz\" (UID: \"0070ea70-06e9-4733-8d11-ecb4cb993ff7\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-ds5zz"
Oct 31 12:53:01.166727 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:01.145620    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29b176fa-229b-433e-928f-41daf33d1f37-credential-socket" pod "29b176fa-229b-433e-928f-41daf33d1f37" (UID: "29b176fa-229b-433e-928f-41daf33d1f37"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:53:01.166727 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:01.146105    5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29b176fa-229b-433e-928f-41daf33d1f37-workload-certs\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:01.166727 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:01.146129    5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29b176fa-229b-433e-928f-41daf33d1f37-workload-socket\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:01.166727 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:01.146160    5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/360843c7-795f-4f22-84cd-76317addf9c2-credential-socket\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:01.166727 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:01.146175    5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38bdca08-eb2b-43c5-8c07-7059e5c9e199-istiod-ca-cert\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:01.166727 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:01.146191    5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38bdca08-eb2b-43c5-8c07-7059e5c9e199-workload-certs\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:01.166727 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:01.146206    5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/360843c7-795f-4f22-84cd-76317addf9c2-istiod-ca-cert\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:01.166727 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:01.146236    5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38bdca08-eb2b-43c5-8c07-7059e5c9e199-istio-data\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:01.166727 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:01.146252    5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/360843c7-795f-4f22-84cd-76317addf9c2-istio-data\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:01.166727 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:01.146270    5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/360843c7-795f-4f22-84cd-76317addf9c2-workload-certs\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:01.166727 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:01.146285    5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86c84864-127f-4507-8f60-2dbb684c687a-workload-socket\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:01.166727 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:01.146323    5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86c84864-127f-4507-8f60-2dbb684c687a-istio-data\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:01.166727 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:01.146755    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea38d319-4637-47c0-bf7a-125b61dac2fe-workload-socket\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-zg66r\" (UID: \"ea38d319-4637-47c0-bf7a-125b61dac2fe\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-zg66r"
Oct 31 12:53:01.166727 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:01.147093    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea38d319-4637-47c0-bf7a-125b61dac2fe-workload-certs\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-zg66r\" (UID: \"ea38d319-4637-47c0-bf7a-125b61dac2fe\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-zg66r"
Oct 31 12:53:01.208930 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:01.178328    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea38d319-4637-47c0-bf7a-125b61dac2fe-istio-data\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-zg66r\" (UID: \"ea38d319-4637-47c0-bf7a-125b61dac2fe\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-zg66r"
Oct 31 12:53:01.208930 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:01.184699    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4rz4\" (UniqueName: \"kubernetes.io/projected/ea38d319-4637-47c0-bf7a-125b61dac2fe-kube-api-access-v4rz4\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-zg66r\" (UID: \"ea38d319-4637-47c0-bf7a-125b61dac2fe\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-zg66r"
Oct 31 12:53:01.208930 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:01.188098    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea38d319-4637-47c0-bf7a-125b61dac2fe-istio-podinfo\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-zg66r\" (UID: \"ea38d319-4637-47c0-bf7a-125b61dac2fe\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-zg66r"
Oct 31 12:53:01.208930 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:01.200512    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29b176fa-229b-433e-928f-41daf33d1f37-istio-data" pod "29b176fa-229b-433e-928f-41daf33d1f37" (UID: "29b176fa-229b-433e-928f-41daf33d1f37"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:53:01.208930 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:01.202376    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86c84864-127f-4507-8f60-2dbb684c687a-workload-certs" pod "86c84864-127f-4507-8f60-2dbb684c687a" (UID: "86c84864-127f-4507-8f60-2dbb684c687a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:53:01.221481 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:01.221160    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea38d319-4637-47c0-bf7a-125b61dac2fe-istio-token\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-zg66r\" (UID: \"ea38d319-4637-47c0-bf7a-125b61dac2fe\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-zg66r"
Oct 31 12:53:01.221688 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:01.221609    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86c84864-127f-4507-8f60-2dbb684c687a-credential-socket" pod "86c84864-127f-4507-8f60-2dbb684c687a" (UID: "86c84864-127f-4507-8f60-2dbb684c687a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:53:01.225573 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:01.221917    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86c84864-127f-4507-8f60-2dbb684c687a-istiod-ca-cert" pod "86c84864-127f-4507-8f60-2dbb684c687a" (UID: "86c84864-127f-4507-8f60-2dbb684c687a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:53:01.225573 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:01.222291    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29b176fa-229b-433e-928f-41daf33d1f37-istiod-ca-cert" pod "29b176fa-229b-433e-928f-41daf33d1f37" (UID: "29b176fa-229b-433e-928f-41daf33d1f37"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:53:01.230903 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:01.230868    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38bdca08-eb2b-43c5-8c07-7059e5c9e199-workload-socket" pod "38bdca08-eb2b-43c5-8c07-7059e5c9e199" (UID: "38bdca08-eb2b-43c5-8c07-7059e5c9e199"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:53:01.233670 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:01.233613    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38bdca08-eb2b-43c5-8c07-7059e5c9e199-credential-socket" pod "38bdca08-eb2b-43c5-8c07-7059e5c9e199" (UID: "38bdca08-eb2b-43c5-8c07-7059e5c9e199"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:53:01.234650 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:01.234624    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/360843c7-795f-4f22-84cd-76317addf9c2-workload-socket" pod "360843c7-795f-4f22-84cd-76317addf9c2" (UID: "360843c7-795f-4f22-84cd-76317addf9c2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:53:01.246239 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:01.245995    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86c84864-127f-4507-8f60-2dbb684c687a-istio-envoy" pod "86c84864-127f-4507-8f60-2dbb684c687a" (UID: "86c84864-127f-4507-8f60-2dbb684c687a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:53:01.246239 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:01.246174    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/360843c7-795f-4f22-84cd-76317addf9c2-istio-token" pod "360843c7-795f-4f22-84cd-76317addf9c2" (UID: "360843c7-795f-4f22-84cd-76317addf9c2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:53:01.247046 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:01.246654    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/360843c7-795f-4f22-84cd-76317addf9c2-kube-api-access-mnvvb" pod "360843c7-795f-4f22-84cd-76317addf9c2" (UID: "360843c7-795f-4f22-84cd-76317addf9c2"). InnerVolumeSpecName "kube-api-access-mnvvb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:53:01.248678 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:01.248648    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38bdca08-eb2b-43c5-8c07-7059e5c9e199-kube-api-access-nqgl6" pod "38bdca08-eb2b-43c5-8c07-7059e5c9e199" (UID: "38bdca08-eb2b-43c5-8c07-7059e5c9e199"). InnerVolumeSpecName "kube-api-access-nqgl6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:53:01.250235 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:01.250204    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0070ea70-06e9-4733-8d11-ecb4cb993ff7-workload-certs\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-ds5zz\" (UID: \"0070ea70-06e9-4733-8d11-ecb4cb993ff7\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-ds5zz"
Oct 31 12:53:01.251674 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:01.251645    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0070ea70-06e9-4733-8d11-ecb4cb993ff7-istio-token\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-ds5zz\" (UID: \"0070ea70-06e9-4733-8d11-ecb4cb993ff7\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-ds5zz"
Oct 31 12:53:01.251833 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:01.251814    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0070ea70-06e9-4733-8d11-ecb4cb993ff7-istio-podinfo\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-ds5zz\" (UID: \"0070ea70-06e9-4733-8d11-ecb4cb993ff7\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-ds5zz"
Oct 31 12:53:01.251945 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:01.251928    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0070ea70-06e9-4733-8d11-ecb4cb993ff7-credential-socket\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-ds5zz\" (UID: \"0070ea70-06e9-4733-8d11-ecb4cb993ff7\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-ds5zz"
Oct 31 12:53:01.251993 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:01.251965    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0070ea70-06e9-4733-8d11-ecb4cb993ff7-istio-envoy\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-ds5zz\" (UID: \"0070ea70-06e9-4733-8d11-ecb4cb993ff7\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-ds5zz"
Oct 31 12:53:01.252031 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:01.252019    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0070ea70-06e9-4733-8d11-ecb4cb993ff7-workload-socket\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-ds5zz\" (UID: \"0070ea70-06e9-4733-8d11-ecb4cb993ff7\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-ds5zz"
Oct 31 12:53:01.252216 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:01.252198    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0070ea70-06e9-4733-8d11-ecb4cb993ff7-istiod-ca-cert\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-ds5zz\" (UID: \"0070ea70-06e9-4733-8d11-ecb4cb993ff7\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-ds5zz"
Oct 31 12:53:01.252410 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:01.252394    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7l89\" (UniqueName: \"kubernetes.io/projected/0070ea70-06e9-4733-8d11-ecb4cb993ff7-kube-api-access-w7l89\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-ds5zz\" (UID: \"0070ea70-06e9-4733-8d11-ecb4cb993ff7\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-ds5zz"
Oct 31 12:53:01.252593 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:01.252573    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0070ea70-06e9-4733-8d11-ecb4cb993ff7-istio-data\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-ds5zz\" (UID: \"0070ea70-06e9-4733-8d11-ecb4cb993ff7\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-ds5zz"
Oct 31 12:53:01.252811 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:01.252793    5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38bdca08-eb2b-43c5-8c07-7059e5c9e199-credential-socket\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:01.252915 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:01.252901    5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/360843c7-795f-4f22-84cd-76317addf9c2-workload-socket\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:01.253026 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:01.253012    5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86c84864-127f-4507-8f60-2dbb684c687a-istiod-ca-cert\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:01.253125 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:01.253112    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nqgl6\" (UniqueName: \"kubernetes.io/projected/38bdca08-eb2b-43c5-8c07-7059e5c9e199-kube-api-access-nqgl6\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:01.253230 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:01.253216    5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86c84864-127f-4507-8f60-2dbb684c687a-workload-certs\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:01.253328 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:01.253316    5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86c84864-127f-4507-8f60-2dbb684c687a-credential-socket\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:01.253424 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:01.253411    5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29b176fa-229b-433e-928f-41daf33d1f37-istio-data\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:01.253549 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:01.253535    5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86c84864-127f-4507-8f60-2dbb684c687a-istio-envoy\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:01.253654 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:01.253641    5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29b176fa-229b-433e-928f-41daf33d1f37-credential-socket\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:01.253756 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:01.253743    5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/360843c7-795f-4f22-84cd-76317addf9c2-istio-token\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:01.253852 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:01.253838    5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29b176fa-229b-433e-928f-41daf33d1f37-istiod-ca-cert\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:01.253946 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:01.253933    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mnvvb\" (UniqueName: \"kubernetes.io/projected/360843c7-795f-4f22-84cd-76317addf9c2-kube-api-access-mnvvb\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:01.254043 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:01.254030    5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38bdca08-eb2b-43c5-8c07-7059e5c9e199-workload-socket\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:01.266561 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:01.254720    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0070ea70-06e9-4733-8d11-ecb4cb993ff7-istio-data\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-ds5zz\" (UID: \"0070ea70-06e9-4733-8d11-ecb4cb993ff7\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-ds5zz"
Oct 31 12:53:01.266768 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:01.263795    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea38d319-4637-47c0-bf7a-125b61dac2fe-credential-socket\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-zg66r\" (UID: \"ea38d319-4637-47c0-bf7a-125b61dac2fe\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-zg66r"
Oct 31 12:53:01.267144 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:01.267123    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0070ea70-06e9-4733-8d11-ecb4cb993ff7-credential-socket\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-ds5zz\" (UID: \"0070ea70-06e9-4733-8d11-ecb4cb993ff7\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-ds5zz"
Oct 31 12:53:01.310625 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:01.310527    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38bdca08-eb2b-43c5-8c07-7059e5c9e199-istio-envoy" pod "38bdca08-eb2b-43c5-8c07-7059e5c9e199" (UID: "38bdca08-eb2b-43c5-8c07-7059e5c9e199"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:53:01.310816 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:01.310796    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea38d319-4637-47c0-bf7a-125b61dac2fe-istio-envoy\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-zg66r\" (UID: \"ea38d319-4637-47c0-bf7a-125b61dac2fe\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-zg66r"
Oct 31 12:53:01.311468 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:01.310924    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86c84864-127f-4507-8f60-2dbb684c687a-istio-token" pod "86c84864-127f-4507-8f60-2dbb684c687a" (UID: "86c84864-127f-4507-8f60-2dbb684c687a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:53:01.311468 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:01.310996    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/29b176fa-229b-433e-928f-41daf33d1f37-istio-podinfo" pod "29b176fa-229b-433e-928f-41daf33d1f37" (UID: "29b176fa-229b-433e-928f-41daf33d1f37"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 12:53:01.311468 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:01.311334    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0070ea70-06e9-4733-8d11-ecb4cb993ff7-istio-podinfo\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-ds5zz\" (UID: \"0070ea70-06e9-4733-8d11-ecb4cb993ff7\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-ds5zz"
Oct 31 12:53:01.312393 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:01.311934    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38bdca08-eb2b-43c5-8c07-7059e5c9e199-istio-token" pod "38bdca08-eb2b-43c5-8c07-7059e5c9e199" (UID: "38bdca08-eb2b-43c5-8c07-7059e5c9e199"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:53:01.312393 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:01.312022    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29b176fa-229b-433e-928f-41daf33d1f37-istio-envoy" pod "29b176fa-229b-433e-928f-41daf33d1f37" (UID: "29b176fa-229b-433e-928f-41daf33d1f37"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:53:01.312393 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:01.312077    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/38bdca08-eb2b-43c5-8c07-7059e5c9e199-istio-podinfo" pod "38bdca08-eb2b-43c5-8c07-7059e5c9e199" (UID: "38bdca08-eb2b-43c5-8c07-7059e5c9e199"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 12:53:01.312393 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:01.312138    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29b176fa-229b-433e-928f-41daf33d1f37-istio-token" pod "29b176fa-229b-433e-928f-41daf33d1f37" (UID: "29b176fa-229b-433e-928f-41daf33d1f37"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:53:01.312393 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:01.312185    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/360843c7-795f-4f22-84cd-76317addf9c2-istio-envoy" pod "360843c7-795f-4f22-84cd-76317addf9c2" (UID: "360843c7-795f-4f22-84cd-76317addf9c2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:53:01.312393 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:01.312231    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/360843c7-795f-4f22-84cd-76317addf9c2-istio-podinfo" pod "360843c7-795f-4f22-84cd-76317addf9c2" (UID: "360843c7-795f-4f22-84cd-76317addf9c2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 12:53:01.312393 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:01.312285    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29b176fa-229b-433e-928f-41daf33d1f37-kube-api-access-g9rjz" pod "29b176fa-229b-433e-928f-41daf33d1f37" (UID: "29b176fa-229b-433e-928f-41daf33d1f37"). InnerVolumeSpecName "kube-api-access-g9rjz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:53:01.312393 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:01.312337    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86c84864-127f-4507-8f60-2dbb684c687a-kube-api-access-x42tb" pod "86c84864-127f-4507-8f60-2dbb684c687a" (UID: "86c84864-127f-4507-8f60-2dbb684c687a"). InnerVolumeSpecName "kube-api-access-x42tb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:53:01.324228 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:01.313659    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/86c84864-127f-4507-8f60-2dbb684c687a-istio-podinfo" pod "86c84864-127f-4507-8f60-2dbb684c687a" (UID: "86c84864-127f-4507-8f60-2dbb684c687a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 12:53:01.324228 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:01.316800    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dec1f857-9b34-458f-8954-56d2753466db" path="/var/lib/kubelet/pods/dec1f857-9b34-458f-8954-56d2753466db/volumes"
Oct 31 12:53:01.324228 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:01.322027    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0070ea70-06e9-4733-8d11-ecb4cb993ff7-workload-certs\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-ds5zz\" (UID: \"0070ea70-06e9-4733-8d11-ecb4cb993ff7\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-ds5zz"
Oct 31 12:53:01.328770 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:01.327855    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0070ea70-06e9-4733-8d11-ecb4cb993ff7-istio-envoy\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-ds5zz\" (UID: \"0070ea70-06e9-4733-8d11-ecb4cb993ff7\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-ds5zz"
Oct 31 12:53:01.331680 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:01.330668    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0070ea70-06e9-4733-8d11-ecb4cb993ff7-workload-socket\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-ds5zz\" (UID: \"0070ea70-06e9-4733-8d11-ecb4cb993ff7\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-ds5zz"
Oct 31 12:53:01.331680 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:01.331440    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0070ea70-06e9-4733-8d11-ecb4cb993ff7-istiod-ca-cert\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-ds5zz\" (UID: \"0070ea70-06e9-4733-8d11-ecb4cb993ff7\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-ds5zz"
Oct 31 12:53:01.336532 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:01.335720    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-b9pn6"]
Oct 31 12:53:01.364253 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:01.363257    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7l89\" (UniqueName: \"kubernetes.io/projected/0070ea70-06e9-4733-8d11-ecb4cb993ff7-kube-api-access-w7l89\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-ds5zz\" (UID: \"0070ea70-06e9-4733-8d11-ecb4cb993ff7\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-ds5zz"
Oct 31 12:53:01.364253 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:01.363978    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0070ea70-06e9-4733-8d11-ecb4cb993ff7-istio-token\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-ds5zz\" (UID: \"0070ea70-06e9-4733-8d11-ecb4cb993ff7\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-ds5zz"
Oct 31 12:53:01.438537 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:01.404719    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-zg66r"
Oct 31 12:53:01.556099 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:01.553258    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-ds5zz"
Oct 31 12:53:01.595179 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:01.590836    5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38bdca08-eb2b-43c5-8c07-7059e5c9e199-istio-envoy\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:01.595179 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:01.593857    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x42tb\" (UniqueName: \"kubernetes.io/projected/86c84864-127f-4507-8f60-2dbb684c687a-kube-api-access-x42tb\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:01.595179 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:01.593885    5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29b176fa-229b-433e-928f-41daf33d1f37-istio-podinfo\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:01.595179 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:01.593961    5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38bdca08-eb2b-43c5-8c07-7059e5c9e199-istio-podinfo\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:01.595179 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:01.593999    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9rjz\" (UniqueName: \"kubernetes.io/projected/29b176fa-229b-433e-928f-41daf33d1f37-kube-api-access-g9rjz\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:01.595179 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:01.594018    5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/360843c7-795f-4f22-84cd-76317addf9c2-istio-envoy\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:01.595179 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:01.594048    5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38bdca08-eb2b-43c5-8c07-7059e5c9e199-istio-token\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:01.595179 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:01.594070    5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29b176fa-229b-433e-928f-41daf33d1f37-istio-envoy\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:01.595179 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:01.594085    5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86c84864-127f-4507-8f60-2dbb684c687a-istio-podinfo\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:01.595179 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:01.594101    5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29b176fa-229b-433e-928f-41daf33d1f37-istio-token\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:01.595179 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:01.594129    5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/360843c7-795f-4f22-84cd-76317addf9c2-istio-podinfo\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:01.595179 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:01.594150    5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86c84864-127f-4507-8f60-2dbb684c687a-istio-token\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:01.641578 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:01.639474    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-jf8bm"]
Oct 31 12:53:01.642131 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:01.642107    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-jf8bm"]
Oct 31 12:53:01.642288 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:01.642273    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-b9pn6"]
Oct 31 12:53:01.642559 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:01.642542    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-b9pn6"
Oct 31 12:53:01.712326 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:01.712289    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41d07641-1c30-4e59-b402-fd7505e781ba-istio-podinfo\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-b9pn6\" (UID: \"41d07641-1c30-4e59-b402-fd7505e781ba\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-b9pn6"
Oct 31 12:53:01.712698 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:01.712672    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41d07641-1c30-4e59-b402-fd7505e781ba-workload-certs\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-b9pn6\" (UID: \"41d07641-1c30-4e59-b402-fd7505e781ba\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-b9pn6"
Oct 31 12:53:01.712860 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:01.712842    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41d07641-1c30-4e59-b402-fd7505e781ba-istio-envoy\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-b9pn6\" (UID: \"41d07641-1c30-4e59-b402-fd7505e781ba\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-b9pn6"
Oct 31 12:53:01.713008 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:01.712989    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42dl6\" (UniqueName: \"kubernetes.io/projected/41d07641-1c30-4e59-b402-fd7505e781ba-kube-api-access-42dl6\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-b9pn6\" (UID: \"41d07641-1c30-4e59-b402-fd7505e781ba\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-b9pn6"
Oct 31 12:53:01.713140 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:01.713120    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41d07641-1c30-4e59-b402-fd7505e781ba-istio-token\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-b9pn6\" (UID: \"41d07641-1c30-4e59-b402-fd7505e781ba\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-b9pn6"
Oct 31 12:53:01.713303 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:01.713286    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41d07641-1c30-4e59-b402-fd7505e781ba-istio-data\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-b9pn6\" (UID: \"41d07641-1c30-4e59-b402-fd7505e781ba\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-b9pn6"
Oct 31 12:53:01.713438 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:01.713422    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41d07641-1c30-4e59-b402-fd7505e781ba-istiod-ca-cert\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-b9pn6\" (UID: \"41d07641-1c30-4e59-b402-fd7505e781ba\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-b9pn6"
Oct 31 12:53:01.727734 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:01.714178    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-7stx7"]
Oct 31 12:53:01.727734 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:01.723149    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41d07641-1c30-4e59-b402-fd7505e781ba-credential-socket\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-b9pn6\" (UID: \"41d07641-1c30-4e59-b402-fd7505e781ba\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-b9pn6"
Oct 31 12:53:01.727734 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:01.723379    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41d07641-1c30-4e59-b402-fd7505e781ba-workload-socket\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-b9pn6\" (UID: \"41d07641-1c30-4e59-b402-fd7505e781ba\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-b9pn6"
Oct 31 12:53:01.787658 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:01.764663    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:53:01.787658 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:01.764764    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:53:01.787658 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:01.764812    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:53:01.929736 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:01.847246    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41d07641-1c30-4e59-b402-fd7505e781ba-istio-podinfo\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-b9pn6\" (UID: \"41d07641-1c30-4e59-b402-fd7505e781ba\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-b9pn6"
Oct 31 12:53:01.929736 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:01.849013    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41d07641-1c30-4e59-b402-fd7505e781ba-workload-certs\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-b9pn6\" (UID: \"41d07641-1c30-4e59-b402-fd7505e781ba\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-b9pn6"
Oct 31 12:53:01.929736 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:01.849096    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41d07641-1c30-4e59-b402-fd7505e781ba-istio-envoy\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-b9pn6\" (UID: \"41d07641-1c30-4e59-b402-fd7505e781ba\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-b9pn6"
Oct 31 12:53:01.929736 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:01.849133    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42dl6\" (UniqueName: \"kubernetes.io/projected/41d07641-1c30-4e59-b402-fd7505e781ba-kube-api-access-42dl6\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-b9pn6\" (UID: \"41d07641-1c30-4e59-b402-fd7505e781ba\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-b9pn6"
Oct 31 12:53:01.929736 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:01.849182    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41d07641-1c30-4e59-b402-fd7505e781ba-istio-token\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-b9pn6\" (UID: \"41d07641-1c30-4e59-b402-fd7505e781ba\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-b9pn6"
Oct 31 12:53:01.929736 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:01.849315    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41d07641-1c30-4e59-b402-fd7505e781ba-istio-data\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-b9pn6\" (UID: \"41d07641-1c30-4e59-b402-fd7505e781ba\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-b9pn6"
Oct 31 12:53:01.929736 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:01.849355    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41d07641-1c30-4e59-b402-fd7505e781ba-istiod-ca-cert\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-b9pn6\" (UID: \"41d07641-1c30-4e59-b402-fd7505e781ba\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-b9pn6"
Oct 31 12:53:01.929736 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:01.849410    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41d07641-1c30-4e59-b402-fd7505e781ba-credential-socket\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-b9pn6\" (UID: \"41d07641-1c30-4e59-b402-fd7505e781ba\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-b9pn6"
Oct 31 12:53:01.929736 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:01.849722    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41d07641-1c30-4e59-b402-fd7505e781ba-workload-socket\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-b9pn6\" (UID: \"41d07641-1c30-4e59-b402-fd7505e781ba\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-b9pn6"
Oct 31 12:53:01.929736 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:01.850697    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41d07641-1c30-4e59-b402-fd7505e781ba-workload-socket\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-b9pn6\" (UID: \"41d07641-1c30-4e59-b402-fd7505e781ba\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-b9pn6"
Oct 31 12:53:01.929736 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:01.862835    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41d07641-1c30-4e59-b402-fd7505e781ba-workload-certs\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-b9pn6\" (UID: \"41d07641-1c30-4e59-b402-fd7505e781ba\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-b9pn6"
Oct 31 12:53:01.929736 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:01.869068    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41d07641-1c30-4e59-b402-fd7505e781ba-istio-data\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-b9pn6\" (UID: \"41d07641-1c30-4e59-b402-fd7505e781ba\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-b9pn6"
Oct 31 12:53:01.929736 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:01.871093    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41d07641-1c30-4e59-b402-fd7505e781ba-istiod-ca-cert\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-b9pn6\" (UID: \"41d07641-1c30-4e59-b402-fd7505e781ba\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-b9pn6"
Oct 31 12:53:01.929736 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:01.873683    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41d07641-1c30-4e59-b402-fd7505e781ba-credential-socket\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-b9pn6\" (UID: \"41d07641-1c30-4e59-b402-fd7505e781ba\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-b9pn6"
Oct 31 12:53:01.929736 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:01.873999    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-7stx7"]
Oct 31 12:53:01.929736 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:01.874143    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-7stx7"
Oct 31 12:53:01.929736 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:01.875930    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41d07641-1c30-4e59-b402-fd7505e781ba-istio-envoy\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-b9pn6\" (UID: \"41d07641-1c30-4e59-b402-fd7505e781ba\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-b9pn6"
Oct 31 12:53:01.929736 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:01.887194    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-d086f3-openshift-default-dockercfg-hcwf6\""
Oct 31 12:53:02.013515 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:01.996571    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42dl6\" (UniqueName: \"kubernetes.io/projected/41d07641-1c30-4e59-b402-fd7505e781ba-kube-api-access-42dl6\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-b9pn6\" (UID: \"41d07641-1c30-4e59-b402-fd7505e781ba\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-b9pn6"
Oct 31 12:53:02.015818 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:02.012628    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41d07641-1c30-4e59-b402-fd7505e781ba-istio-podinfo\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-b9pn6\" (UID: \"41d07641-1c30-4e59-b402-fd7505e781ba\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-b9pn6"
Oct 31 12:53:02.016014 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:02.015193    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25899808-1323-425c-9359-3b56f99ad5cb-workload-certs\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-7stx7\" (UID: \"25899808-1323-425c-9359-3b56f99ad5cb\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-7stx7"
Oct 31 12:53:02.016169 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:02.016149    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-665vv\" (UniqueName: \"kubernetes.io/projected/25899808-1323-425c-9359-3b56f99ad5cb-kube-api-access-665vv\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-7stx7\" (UID: \"25899808-1323-425c-9359-3b56f99ad5cb\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-7stx7"
Oct 31 12:53:02.016408 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:02.016387    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25899808-1323-425c-9359-3b56f99ad5cb-istio-token\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-7stx7\" (UID: \"25899808-1323-425c-9359-3b56f99ad5cb\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-7stx7"
Oct 31 12:53:02.020008 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:02.019974    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25899808-1323-425c-9359-3b56f99ad5cb-istiod-ca-cert\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-7stx7\" (UID: \"25899808-1323-425c-9359-3b56f99ad5cb\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-7stx7"
Oct 31 12:53:02.020204 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:02.020183    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25899808-1323-425c-9359-3b56f99ad5cb-istio-data\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-7stx7\" (UID: \"25899808-1323-425c-9359-3b56f99ad5cb\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-7stx7"
Oct 31 12:53:02.020359 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:02.020340    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25899808-1323-425c-9359-3b56f99ad5cb-credential-socket\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-7stx7\" (UID: \"25899808-1323-425c-9359-3b56f99ad5cb\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-7stx7"
Oct 31 12:53:02.020570 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:02.020554    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25899808-1323-425c-9359-3b56f99ad5cb-workload-socket\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-7stx7\" (UID: \"25899808-1323-425c-9359-3b56f99ad5cb\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-7stx7"
Oct 31 12:53:02.020961 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:02.020935    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25899808-1323-425c-9359-3b56f99ad5cb-istio-podinfo\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-7stx7\" (UID: \"25899808-1323-425c-9359-3b56f99ad5cb\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-7stx7"
Oct 31 12:53:02.021104 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:02.021085    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25899808-1323-425c-9359-3b56f99ad5cb-istio-envoy\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-7stx7\" (UID: \"25899808-1323-425c-9359-3b56f99ad5cb\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-7stx7"
Oct 31 12:53:02.024656 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:02.023165    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-1a7d0e-748bf567d5-j7bb7"]
Oct 31 12:53:02.043372 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:02.043316    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-1a7d0e-748bf567d5-j7bb7"]
Oct 31 12:53:02.045922 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:02.045032    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1a7d0e-748bf567d5-j7bb7"
Oct 31 12:53:02.051232 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:02.051202    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-6jkzx\""
Oct 31 12:53:02.051884 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:02.051860    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-1a7d0e-serving-cert\""
Oct 31 12:53:02.051992 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:02.051974    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 31 12:53:02.052912 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:02.052804    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 12:53:02.052912 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:02.052889    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 12:53:02.070568 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:02.060316    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41d07641-1c30-4e59-b402-fd7505e781ba-istio-token\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-b9pn6\" (UID: \"41d07641-1c30-4e59-b402-fd7505e781ba\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-b9pn6"
Oct 31 12:53:02.110971 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:02.100238    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-njl9z"
Oct 31 12:53:02.110971 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:02.102057    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-njl9z" event={"ID":"29b176fa-229b-433e-928f-41daf33d1f37","Type":"ContainerDied","Data":"0ff6d98fc0142fabd43a8291115c4a043efbd1b829dd1790a1fa3daf5ee1d88c"}
Oct 31 12:53:02.110971 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:02.102115    5060 scope.go:122] "RemoveContainer" containerID="6fafb77a1324904f28fa906f139fc8ee088f838268da7825463b6b19073c703d"
Oct 31 12:53:02.110971 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:02.106876    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-mb4nn"]
Oct 31 12:53:02.110971 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:02.107570    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-sc6ff" event={"ID":"360843c7-795f-4f22-84cd-76317addf9c2","Type":"ContainerDied","Data":"aff6b7cd9c407005e7fba993db40d14ce04e6cf0f1a2e8566c594d3f71a04ad6"}
Oct 31 12:53:02.110971 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:02.107686    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-sc6ff"
Oct 31 12:53:02.118001 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:02.117700    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-mb4nn"]
Oct 31 12:53:02.172050 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:02.122938    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-665vv\" (UniqueName: \"kubernetes.io/projected/25899808-1323-425c-9359-3b56f99ad5cb-kube-api-access-665vv\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-7stx7\" (UID: \"25899808-1323-425c-9359-3b56f99ad5cb\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-7stx7"
Oct 31 12:53:02.172050 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:02.122983    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25899808-1323-425c-9359-3b56f99ad5cb-istio-token\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-7stx7\" (UID: \"25899808-1323-425c-9359-3b56f99ad5cb\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-7stx7"
Oct 31 12:53:02.172050 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:02.123182    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xccp\" (UniqueName: \"kubernetes.io/projected/bcb6aa33-17fe-45ac-93a5-ff86419c2246-kube-api-access-6xccp\") pod \"maas-api-e2e-inf-b-1a7d0e-748bf567d5-j7bb7\" (UID: \"bcb6aa33-17fe-45ac-93a5-ff86419c2246\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1a7d0e-748bf567d5-j7bb7"
Oct 31 12:53:02.172050 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:02.123330    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25899808-1323-425c-9359-3b56f99ad5cb-istiod-ca-cert\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-7stx7\" (UID: \"25899808-1323-425c-9359-3b56f99ad5cb\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-7stx7"
Oct 31 12:53:02.172050 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:02.123381    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25899808-1323-425c-9359-3b56f99ad5cb-istio-data\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-7stx7\" (UID: \"25899808-1323-425c-9359-3b56f99ad5cb\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-7stx7"
Oct 31 12:53:02.172050 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:02.123444    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25899808-1323-425c-9359-3b56f99ad5cb-credential-socket\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-7stx7\" (UID: \"25899808-1323-425c-9359-3b56f99ad5cb\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-7stx7"
Oct 31 12:53:02.172050 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:02.123643    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bcb6aa33-17fe-45ac-93a5-ff86419c2246-openshift-service-ca\") pod \"maas-api-e2e-inf-b-1a7d0e-748bf567d5-j7bb7\" (UID: \"bcb6aa33-17fe-45ac-93a5-ff86419c2246\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1a7d0e-748bf567d5-j7bb7"
Oct 31 12:53:02.172050 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:02.123689    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25899808-1323-425c-9359-3b56f99ad5cb-workload-socket\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-7stx7\" (UID: \"25899808-1323-425c-9359-3b56f99ad5cb\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-7stx7"
Oct 31 12:53:02.172050 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:02.123714    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bcb6aa33-17fe-45ac-93a5-ff86419c2246-metrics-certs\") pod \"maas-api-e2e-inf-b-1a7d0e-748bf567d5-j7bb7\" (UID: \"bcb6aa33-17fe-45ac-93a5-ff86419c2246\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1a7d0e-748bf567d5-j7bb7"
Oct 31 12:53:02.172050 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:02.123882    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bcb6aa33-17fe-45ac-93a5-ff86419c2246-maas-api-tls\") pod \"maas-api-e2e-inf-b-1a7d0e-748bf567d5-j7bb7\" (UID: \"bcb6aa33-17fe-45ac-93a5-ff86419c2246\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1a7d0e-748bf567d5-j7bb7"
Oct 31 12:53:02.172050 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:02.124053    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25899808-1323-425c-9359-3b56f99ad5cb-istio-podinfo\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-7stx7\" (UID: \"25899808-1323-425c-9359-3b56f99ad5cb\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-7stx7"
Oct 31 12:53:02.172050 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:02.124088    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25899808-1323-425c-9359-3b56f99ad5cb-istio-envoy\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-7stx7\" (UID: \"25899808-1323-425c-9359-3b56f99ad5cb\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-7stx7"
Oct 31 12:53:02.172050 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:02.124163    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25899808-1323-425c-9359-3b56f99ad5cb-workload-certs\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-7stx7\" (UID: \"25899808-1323-425c-9359-3b56f99ad5cb\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-7stx7"
Oct 31 12:53:02.172050 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:02.124630    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25899808-1323-425c-9359-3b56f99ad5cb-workload-certs\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-7stx7\" (UID: \"25899808-1323-425c-9359-3b56f99ad5cb\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-7stx7"
Oct 31 12:53:02.172050 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:02.127147    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25899808-1323-425c-9359-3b56f99ad5cb-istiod-ca-cert\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-7stx7\" (UID: \"25899808-1323-425c-9359-3b56f99ad5cb\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-7stx7"
Oct 31 12:53:02.172050 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:02.127413    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25899808-1323-425c-9359-3b56f99ad5cb-istio-data\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-7stx7\" (UID: \"25899808-1323-425c-9359-3b56f99ad5cb\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-7stx7"
Oct 31 12:53:02.172050 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:02.127684    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25899808-1323-425c-9359-3b56f99ad5cb-credential-socket\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-7stx7\" (UID: \"25899808-1323-425c-9359-3b56f99ad5cb\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-7stx7"
Oct 31 12:53:02.172050 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:02.128248    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25899808-1323-425c-9359-3b56f99ad5cb-workload-socket\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-7stx7\" (UID: \"25899808-1323-425c-9359-3b56f99ad5cb\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-7stx7"
Oct 31 12:53:02.172050 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:02.132404    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25899808-1323-425c-9359-3b56f99ad5cb-istio-podinfo\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-7stx7\" (UID: \"25899808-1323-425c-9359-3b56f99ad5cb\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-7stx7"
Oct 31 12:53:02.172050 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:02.145106    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-665vv\" (UniqueName: \"kubernetes.io/projected/25899808-1323-425c-9359-3b56f99ad5cb-kube-api-access-665vv\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-7stx7\" (UID: \"25899808-1323-425c-9359-3b56f99ad5cb\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-7stx7"
Oct 31 12:53:02.172050 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:02.163342    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25899808-1323-425c-9359-3b56f99ad5cb-istio-envoy\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-7stx7\" (UID: \"25899808-1323-425c-9359-3b56f99ad5cb\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-7stx7"
Oct 31 12:53:02.201855 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:02.175378    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25899808-1323-425c-9359-3b56f99ad5cb-istio-token\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-7stx7\" (UID: \"25899808-1323-425c-9359-3b56f99ad5cb\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-7stx7"
Oct 31 12:53:02.201855 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:02.191177    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-gxs26" event={"ID":"86c84864-127f-4507-8f60-2dbb684c687a","Type":"ContainerDied","Data":"f5ad51ea2614a5cee4c36c4d3a879b5c62ac7cce423e7f83ae904f0d74e3d1ef"}
Oct 31 12:53:02.201855 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:02.191347    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-gxs26"
Oct 31 12:53:02.246579 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:02.241441    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bcb6aa33-17fe-45ac-93a5-ff86419c2246-maas-api-tls\") pod \"maas-api-e2e-inf-b-1a7d0e-748bf567d5-j7bb7\" (UID: \"bcb6aa33-17fe-45ac-93a5-ff86419c2246\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1a7d0e-748bf567d5-j7bb7"
Oct 31 12:53:02.246579 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:02.241746    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xccp\" (UniqueName: \"kubernetes.io/projected/bcb6aa33-17fe-45ac-93a5-ff86419c2246-kube-api-access-6xccp\") pod \"maas-api-e2e-inf-b-1a7d0e-748bf567d5-j7bb7\" (UID: \"bcb6aa33-17fe-45ac-93a5-ff86419c2246\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1a7d0e-748bf567d5-j7bb7"
Oct 31 12:53:02.246579 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:02.241884    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bcb6aa33-17fe-45ac-93a5-ff86419c2246-openshift-service-ca\") pod \"maas-api-e2e-inf-b-1a7d0e-748bf567d5-j7bb7\" (UID: \"bcb6aa33-17fe-45ac-93a5-ff86419c2246\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1a7d0e-748bf567d5-j7bb7"
Oct 31 12:53:02.246579 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:02.241918    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bcb6aa33-17fe-45ac-93a5-ff86419c2246-metrics-certs\") pod \"maas-api-e2e-inf-b-1a7d0e-748bf567d5-j7bb7\" (UID: \"bcb6aa33-17fe-45ac-93a5-ff86419c2246\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1a7d0e-748bf567d5-j7bb7"
Oct 31 12:53:02.318732 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:02.276283    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bcb6aa33-17fe-45ac-93a5-ff86419c2246-openshift-service-ca\") pod \"maas-api-e2e-inf-b-1a7d0e-748bf567d5-j7bb7\" (UID: \"bcb6aa33-17fe-45ac-93a5-ff86419c2246\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1a7d0e-748bf567d5-j7bb7"
Oct 31 12:53:02.318732 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:02.316911    5060 scope.go:122] "RemoveContainer" containerID="b163f7ffd10c6999491edd6aadba751bcc983950890ceebea3672e7540b9ec48"
Oct 31 12:53:02.328607 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:02.320322    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xccp\" (UniqueName: \"kubernetes.io/projected/bcb6aa33-17fe-45ac-93a5-ff86419c2246-kube-api-access-6xccp\") pod \"maas-api-e2e-inf-b-1a7d0e-748bf567d5-j7bb7\" (UID: \"bcb6aa33-17fe-45ac-93a5-ff86419c2246\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1a7d0e-748bf567d5-j7bb7"
Oct 31 12:53:02.328607 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:02.323763    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:53:02.328607 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:02.323846    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:53:02.328607 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:02.323888    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:53:02.328607 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:02.325053    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-vqcct"]
Oct 31 12:53:02.328607 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:02.325113    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-vqcct" event={"ID":"b2705ec7-400e-4265-b213-5b58ba42cd4c","Type":"ContainerStarted","Data":"9397b449c7a1cb0a61faf30b9fe3db9e6a5a7e467d08d4929cb1534380a29c64"}
Oct 31 12:53:02.332430 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:02.332406    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bcb6aa33-17fe-45ac-93a5-ff86419c2246-metrics-certs\") pod \"maas-api-e2e-inf-b-1a7d0e-748bf567d5-j7bb7\" (UID: \"bcb6aa33-17fe-45ac-93a5-ff86419c2246\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1a7d0e-748bf567d5-j7bb7"
Oct 31 12:53:02.335618 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:02.335127    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bcb6aa33-17fe-45ac-93a5-ff86419c2246-maas-api-tls\") pod \"maas-api-e2e-inf-b-1a7d0e-748bf567d5-j7bb7\" (UID: \"bcb6aa33-17fe-45ac-93a5-ff86419c2246\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1a7d0e-748bf567d5-j7bb7"
Oct 31 12:53:02.337076 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:02.336867    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-b9pn6"
Oct 31 12:53:02.378360 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:02.344577    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-sv4w2" event={"ID":"38bdca08-eb2b-43c5-8c07-7059e5c9e199","Type":"ContainerDied","Data":"184ceb960be55dc2f1bb257c3c9e1df3710eec41a06d9484d0ed29d40209fa5f"}
Oct 31 12:53:02.378360 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:02.344808    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-sv4w2"
Oct 31 12:53:02.413018 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:02.412817    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-7stx7"
Oct 31 12:53:02.423502 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:02.423444    5060 scope.go:122] "RemoveContainer" containerID="b0e2771f19a497662c194c4899a869898da0bab9a01207f1478d9e573f31a211"
Oct 31 12:53:02.425064 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:02.425030    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-e98df8-openshift-default-779d98479-vkrj6"]
Oct 31 12:53:02.434052 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:02.434023    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-njl9z"]
Oct 31 12:53:02.440667 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:02.440617    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-njl9z"]
Oct 31 12:53:02.449111 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:02.449086    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-gxs26"]
Oct 31 12:53:02.498837 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:02.498750    5060 scope.go:122] "RemoveContainer" containerID="4c7d3ca2a9c8a503e740a602271d240f30dea66d2fed340dac15ec3cb71094ef"
Oct 31 12:53:02.501188 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:02.501038    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-gxs26"]
Oct 31 12:53:02.534328 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:02.534300    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-sc6ff"]
Oct 31 12:53:02.541278 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:02.541231    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-sc6ff"]
Oct 31 12:53:02.558019 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:02.557988    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-sv4w2"]
Oct 31 12:53:02.574299 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:02.574264    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1a7d0e-748bf567d5-j7bb7"
Oct 31 12:53:02.614640 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:02.614603    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-ds5zz"]
Oct 31 12:53:02.621349 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:02.621311    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-sv4w2"]
Oct 31 12:53:02.642300 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:53:02.642271    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0070ea70_06e9_4733_8d11_ecb4cb993ff7.slice/crio-1f61ced7df877c2828cf38dfb610fe4f1b1fd81be04e13a04f5d6084d5426bed WatchSource:0}: Error finding container 1f61ced7df877c2828cf38dfb610fe4f1b1fd81be04e13a04f5d6084d5426bed: Status 404 returned error can't find the container with id 1f61ced7df877c2828cf38dfb610fe4f1b1fd81be04e13a04f5d6084d5426bed
Oct 31 12:53:02.646305 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:02.646267    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:53:02.646480 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:02.646462    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:53:02.646673 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:02.646656    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:53:02.685513 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:02.663049    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-zg66r"]
Oct 31 12:53:02.742938 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:02.742705    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:53:02.742938 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:02.742798    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:53:02.742938 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:02.742894    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:53:03.136815 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:03.136537    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02360407-0dfa-45b7-b276-6487dcfb02f0" path="/var/lib/kubelet/pods/02360407-0dfa-45b7-b276-6487dcfb02f0/volumes"
Oct 31 12:53:03.137188 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:53:03.137160    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41d07641_1c30_4e59_b402_fd7505e781ba.slice/crio-9cd5c391d8f2cb2f6b53ce112e11f43d8ae53ed19c51b8c9b83e057598cac628 WatchSource:0}: Error finding container 9cd5c391d8f2cb2f6b53ce112e11f43d8ae53ed19c51b8c9b83e057598cac628: Status 404 returned error can't find the container with id 9cd5c391d8f2cb2f6b53ce112e11f43d8ae53ed19c51b8c9b83e057598cac628
Oct 31 12:53:03.137739 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:03.137688    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ade94c0-4cba-4ccd-87c0-cdd0cc2aa72d" path="/var/lib/kubelet/pods/0ade94c0-4cba-4ccd-87c0-cdd0cc2aa72d/volumes"
Oct 31 12:53:03.138743 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:03.138721    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29b176fa-229b-433e-928f-41daf33d1f37" path="/var/lib/kubelet/pods/29b176fa-229b-433e-928f-41daf33d1f37/volumes"
Oct 31 12:53:03.140038 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:03.139990    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="360843c7-795f-4f22-84cd-76317addf9c2" path="/var/lib/kubelet/pods/360843c7-795f-4f22-84cd-76317addf9c2/volumes"
Oct 31 12:53:03.140872 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:03.140848    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38bdca08-eb2b-43c5-8c07-7059e5c9e199" path="/var/lib/kubelet/pods/38bdca08-eb2b-43c5-8c07-7059e5c9e199/volumes"
Oct 31 12:53:03.141703 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:03.141683    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86c84864-127f-4507-8f60-2dbb684c687a" path="/var/lib/kubelet/pods/86c84864-127f-4507-8f60-2dbb684c687a/volumes"
Oct 31 12:53:03.142905 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:03.142883    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-7stx7"]
Oct 31 12:53:03.143076 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:03.143061    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-b9pn6"]
Oct 31 12:53:03.218913 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:03.218862    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:53:03.219222 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:03.219192    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:53:03.219373 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:03.219352    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:53:03.264043 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:03.255680    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:53:03.264043 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:03.255773    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:53:03.264043 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:03.255820    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:53:03.440847 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:03.438444    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-1a7d0e-748bf567d5-j7bb7"]
Oct 31 12:53:03.453293 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:03.453256    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-zg66r" event={"ID":"ea38d319-4637-47c0-bf7a-125b61dac2fe","Type":"ContainerStarted","Data":"7852c16adc99dafa80d1c9727e019f8620e0370ced0b974d88f7a36f05a969b9"}
Oct 31 12:53:03.454016 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:03.453994    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-zg66r" event={"ID":"ea38d319-4637-47c0-bf7a-125b61dac2fe","Type":"ContainerStarted","Data":"cbd9f0525d16e5a919d517fb91c92cd7b9618dad5c913f647af6ecca2905f2b6"}
Oct 31 12:53:03.522551 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:03.504255    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-zg66r" podStartSLOduration=181.504229489 podStartE2EDuration="3m1.504229489s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:50:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:53:03.493071637 +0000 UTC m=+1546.596139352" watchObservedRunningTime="2026-10-31 12:53:03.504229489 +0000 UTC m=+1546.607297201"
Oct 31 12:53:03.556399 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:03.556295    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-b9pn6" event={"ID":"41d07641-1c30-4e59-b402-fd7505e781ba","Type":"ContainerStarted","Data":"9cd5c391d8f2cb2f6b53ce112e11f43d8ae53ed19c51b8c9b83e057598cac628"}
Oct 31 12:53:03.653111 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:03.653015    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-ds5zz" event={"ID":"0070ea70-06e9-4733-8d11-ecb4cb993ff7","Type":"ContainerStarted","Data":"854af3eb3ad954edf26df321b78fd4ad65ec92e0b1d06ab91d690d848515f96b"}
Oct 31 12:53:03.655552 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:03.653650    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-ds5zz" event={"ID":"0070ea70-06e9-4733-8d11-ecb4cb993ff7","Type":"ContainerStarted","Data":"1f61ced7df877c2828cf38dfb610fe4f1b1fd81be04e13a04f5d6084d5426bed"}
Oct 31 12:53:03.827814 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:03.825117    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e98df8-openshift-default-779d98479-vkrj6" event={"ID":"0d62f8e4-b062-4dd4-a23d-0f66566c83da","Type":"ContainerStarted","Data":"75f2642303e354ccd77cddad60790638c3b1c480e432f7aad1fb90ca360b1992"}
Oct 31 12:53:03.827814 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:03.825189    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e98df8-openshift-default-779d98479-vkrj6" event={"ID":"0d62f8e4-b062-4dd4-a23d-0f66566c83da","Type":"ContainerStarted","Data":"a2648314fc5d81306544915fa7cbabfcf11182a2ab6539373f7a5accb94bef42"}
Oct 31 12:53:03.932911 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:03.928214    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-ds5zz" podStartSLOduration=180.928190188 podStartE2EDuration="3m0.928190188s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:50:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:53:03.894272264 +0000 UTC m=+1546.997339977" watchObservedRunningTime="2026-10-31 12:53:03.928190188 +0000 UTC m=+1547.031257901"
Oct 31 12:53:03.976904 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:03.976251    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-e98df8-openshift-default-779d98479-vkrj6" podStartSLOduration=181.976229939 podStartE2EDuration="3m1.976229939s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:50:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:53:03.969247484 +0000 UTC m=+1547.072315199" watchObservedRunningTime="2026-10-31 12:53:03.976229939 +0000 UTC m=+1547.079297656"
Oct 31 12:53:04.032450 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:04.032410    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1a7d0e-748bf567d5-j7bb7" event={"ID":"bcb6aa33-17fe-45ac-93a5-ff86419c2246","Type":"ContainerStarted","Data":"15da662a7fc426bcf64aae0262d9d26bd1ded3097c751a3dec6c86b57d2f4aa4"}
Oct 31 12:53:04.154153 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:04.133807    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-e98df8-openshift-default-779d98479-vkrj6"
Oct 31 12:53:04.210619 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:04.176677    5060 patch_prober.go:29] interesting pod/e2e-inf-a-e98df8-openshift-default-779d98479-vkrj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body=
Oct 31 12:53:04.210619 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:04.176746    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e98df8-openshift-default-779d98479-vkrj6" podUID="0d62f8e4-b062-4dd4-a23d-0f66566c83da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused"
Oct 31 12:53:04.369987 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:04.369947    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-7stx7" event={"ID":"25899808-1323-425c-9359-3b56f99ad5cb","Type":"ContainerStarted","Data":"d75286916ab35ebe5672ab15aea942360fc21a1b36049d2f87476892c002b60a"}
Oct 31 12:53:04.619623 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:04.532107    5060 patch_prober.go:29] interesting pod/e2e-inf-a-4e8eb9-openshift-default-768d574d85-zg66r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body=
Oct 31 12:53:04.700812 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:04.594285    5060 patch_prober.go:29] interesting pod/e2e-inf-a-4e8eb9-openshift-default-768d574d85-ds5zz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" start-of-body=
Oct 31 12:53:04.700812 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:04.681656    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-ds5zz" podUID="0070ea70-06e9-4733-8d11-ecb4cb993ff7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused"
Oct 31 12:53:04.700812 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:04.691074    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-zg66r" podUID="ea38d319-4637-47c0-bf7a-125b61dac2fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused"
Oct 31 12:53:04.749518 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:04.749261    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-zg66r"
Oct 31 12:53:04.749518 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:04.749339    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-ds5zz"
Oct 31 12:53:04.804430 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:04.798479    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-vqcct" event={"ID":"b2705ec7-400e-4265-b213-5b58ba42cd4c","Type":"ContainerStarted","Data":"3975283cd17e0a73b11ef41c996088b132983657497571571be49b0e29d04ee0"}
Oct 31 12:53:04.901568 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:04.895626    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/premium-w2-e6c79d-kserve-6468bf46d4-h66f8" podUID="cd6a266f-8d12-4089-83ee-c75c716bf26c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:53:05.066052 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:05.066007    5060 patch_prober.go:29] interesting pod/e2e-inf-a-e98df8-openshift-default-779d98479-vkrj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body=
Oct 31 12:53:05.066668 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:05.066265    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e98df8-openshift-default-779d98479-vkrj6" podUID="0d62f8e4-b062-4dd4-a23d-0f66566c83da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused"
Oct 31 12:53:05.137559 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:05.122128    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-vqcct"
Oct 31 12:53:05.177709 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:05.033176    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x2qzv" podUID="c247ed2a-aa90-424b-9b7e-4f2936d67e3f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded"
Oct 31 12:53:05.202803 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:05.202750    5060 patch_prober.go:29] interesting pod/e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-vqcct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" start-of-body=
Oct 31 12:53:05.203117 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:05.203066    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-vqcct" podUID="b2705ec7-400e-4265-b213-5b58ba42cd4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused"
Oct 31 12:53:05.441565 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:05.441449    5060 patch_prober.go:29] interesting pod/e2e-inf-a-4e8eb9-openshift-default-768d574d85-zg66r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body=
Oct 31 12:53:05.441841 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:05.441806    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-zg66r" podUID="ea38d319-4637-47c0-bf7a-125b61dac2fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused"
Oct 31 12:53:05.570552 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:05.550987    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-vqcct" podStartSLOduration=124.550962766 podStartE2EDuration="2m4.550962766s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:51:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:53:05.122592148 +0000 UTC m=+1548.225659861" watchObservedRunningTime="2026-10-31 12:53:05.550962766 +0000 UTC m=+1548.654030480"
Oct 31 12:53:05.571679 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:05.571597    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-2-w5-b25d3b-kserve-76f798d59f-vvzhg" podUID="394ed76c-84ac-4612-9acb-a7e36cb68d89" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:53:05.639516 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:05.600901    5060 patch_prober.go:29] interesting pod/e2e-inf-a-4e8eb9-openshift-default-768d574d85-ds5zz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" start-of-body=
Oct 31 12:53:05.686143 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:05.673038    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-ds5zz" podUID="0070ea70-06e9-4733-8d11-ecb4cb993ff7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused"
Oct 31 12:53:06.085566 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:06.000281    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/premium-w2-e6c79d-kserve-6468bf46d4-h66f8" podUID="cd6a266f-8d12-4089-83ee-c75c716bf26c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:53:06.086656 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:06.044706    5060 patch_prober.go:29] interesting pod/e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-vqcct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" start-of-body=
Oct 31 12:53:06.086887 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:06.086855    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-vqcct" podUID="b2705ec7-400e-4265-b213-5b58ba42cd4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused"
Oct 31 12:53:06.136812 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:06.062016    5060 patch_prober.go:29] interesting pod/e2e-inf-a-e98df8-openshift-default-779d98479-vkrj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body=
Oct 31 12:53:06.136812 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:06.087018    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e98df8-openshift-default-779d98479-vkrj6" podUID="0d62f8e4-b062-4dd4-a23d-0f66566c83da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused"
Oct 31 12:53:06.136812 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:06.087680    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e6c79d/premium-w2-e6c79d-kserve-6468bf46d4-h66f8"
Oct 31 12:53:06.168094 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:06.168049    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x2qzv" podUID="c247ed2a-aa90-424b-9b7e-4f2936d67e3f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:53:06.168735 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:06.168711    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x2qzv"
Oct 31 12:53:06.505570 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:06.504756    5060 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-2-w5-b25d3b-kserve-76f798d59f-vvzhg" podUID="394ed76c-84ac-4612-9acb-a7e36cb68d89" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:53:06.523659 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:06.506078    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/simulator-w4-c67de2-kserve-868679cf9b-gqjbx" podUID="e104d7da-1fb2-4c99-a480-b4ae84c4f273" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded"
Oct 31 12:53:06.575516 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:06.574376    5060 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-c67de2/simulator-w4-c67de2-kserve-868679cf9b-gqjbx" podUID="e104d7da-1fb2-4c99-a480-b4ae84c4f273" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:53:06.575516 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:06.574554    5060 patch_prober.go:29] interesting pod/e2e-inf-a-4e8eb9-openshift-default-768d574d85-zg66r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body=
Oct 31 12:53:06.575516 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:06.574602    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-zg66r" podUID="ea38d319-4637-47c0-bf7a-125b61dac2fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused"
Oct 31 12:53:06.603522 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:06.597622    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-2-w5-b25d3b-kserve-76f798d59f-vvzhg" podUID="394ed76c-84ac-4612-9acb-a7e36cb68d89" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:53:06.603522 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:06.598178    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-2-w5-b25d3b-kserve-76f798d59f-vvzhg"
Oct 31 12:53:06.650864 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:06.629843    5060 patch_prober.go:29] interesting pod/e2e-inf-a-4e8eb9-openshift-default-768d574d85-ds5zz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" start-of-body=
Oct 31 12:53:06.651186 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:06.651144    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-ds5zz" podUID="0070ea70-06e9-4733-8d11-ecb4cb993ff7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused"
Oct 31 12:53:06.652447 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:06.652414    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-b9pn6" event={"ID":"41d07641-1c30-4e59-b402-fd7505e781ba","Type":"ContainerStarted","Data":"dfb44bb00e52e91a04fb972e54c41471fb0f804fc6f3769d653cbd1212f00dea"}
Oct 31 12:53:06.787516 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:06.783957    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-7stx7" event={"ID":"25899808-1323-425c-9359-3b56f99ad5cb","Type":"ContainerStarted","Data":"26e44b9d2ea1cc00868da3adafd4ba6e1bbce4e6f6194a2b35f7de329d87b522"}
Oct 31 12:53:06.948338 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:06.948304    5060 patch_prober.go:29] interesting pod/e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-vqcct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" start-of-body=
Oct 31 12:53:06.948650 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:06.948620    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-vqcct" podUID="b2705ec7-400e-4265-b213-5b58ba42cd4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused"
Oct 31 12:53:07.097662 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:07.097593    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/premium-w2-e6c79d-kserve-6468bf46d4-h66f8" podUID="cd6a266f-8d12-4089-83ee-c75c716bf26c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:53:07.256778 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:07.256724    5060 patch_prober.go:29] interesting pod/e2e-inf-a-e98df8-openshift-default-779d98479-vkrj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body=
Oct 31 12:53:07.257117 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:07.257083    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e98df8-openshift-default-779d98479-vkrj6" podUID="0d62f8e4-b062-4dd4-a23d-0f66566c83da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused"
Oct 31 12:53:07.409921 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:07.387199    5060 patch_prober.go:29] interesting pod/e2e-inf-a-4e8eb9-openshift-default-768d574d85-b9pn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" start-of-body=
Oct 31 12:53:07.409921 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:07.387281    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-b9pn6" podUID="41d07641-1c30-4e59-b402-fd7505e781ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused"
Oct 31 12:53:07.409921 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:07.388285    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x2qzv" podUID="c247ed2a-aa90-424b-9b7e-4f2936d67e3f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded"
Oct 31 12:53:07.410646 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:07.410600    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vmzbr" podUID="7049f597-e139-4f0e-afbd-ec6ea0effab9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:53:07.468120 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:07.468052    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-rbn8q" podUID="dea929f7-61c3-4921-9bdb-106f763e067e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded"
Oct 31 12:53:07.521346 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:07.521306    5060 patch_prober.go:29] interesting pod/e2e-inf-a-4e8eb9-openshift-default-768d574d85-zg66r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body=
Oct 31 12:53:07.561517 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:07.552619    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-zg66r" podUID="ea38d319-4637-47c0-bf7a-125b61dac2fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused"
Oct 31 12:53:07.561517 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:07.537589    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-rm7cl" podUID="8292eaa8-0a02-46be-8654-3c4136ff604f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded"
Oct 31 12:53:07.635664 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:07.597235    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/simulator-w4-c67de2-kserve-868679cf9b-gqjbx" podUID="e104d7da-1fb2-4c99-a480-b4ae84c4f273" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded"
Oct 31 12:53:07.635664 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:07.598616    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-2-w5-b25d3b-kserve-76f798d59f-vvzhg" podUID="394ed76c-84ac-4612-9acb-a7e36cb68d89" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:53:07.814549 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:07.781838    5060 patch_prober.go:29] interesting pod/e2e-inf-a-4e8eb9-openshift-default-768d574d85-ds5zz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" start-of-body=
Oct 31 12:53:07.814549 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:07.781908    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-ds5zz" podUID="0070ea70-06e9-4733-8d11-ecb4cb993ff7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused"
Oct 31 12:53:07.987547 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:07.987218    5060 patch_prober.go:29] interesting pod/e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-vqcct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" start-of-body=
Oct 31 12:53:07.987547 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:07.987279    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-vqcct" podUID="b2705ec7-400e-4265-b213-5b58ba42cd4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused"
Oct 31 12:53:08.154542 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:08.145180    5060 patch_prober.go:29] interesting pod/e2e-inf-a-e98df8-openshift-default-779d98479-vkrj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body=
Oct 31 12:53:08.164626 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:08.164569    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e98df8-openshift-default-779d98479-vkrj6" podUID="0d62f8e4-b062-4dd4-a23d-0f66566c83da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused"
Oct 31 12:53:08.290959 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:08.290921    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/premium-w2-e6c79d-kserve-6468bf46d4-h66f8" podUID="cd6a266f-8d12-4089-83ee-c75c716bf26c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Oct 31 12:53:08.446626 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:08.410672    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x2qzv" podUID="c247ed2a-aa90-424b-9b7e-4f2936d67e3f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:53:08.479814 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:53:08.464696    5060 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.335s"
Oct 31 12:53:08.479814 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:08.464759    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-b9pn6"
Oct 31 12:53:08.479814 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:08.464844    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c67de2/simulator-w4-c67de2-kserve-868679cf9b-gqjbx"
Oct 31 12:53:08.479814 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:08.464866    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-7stx7"
Oct 31 12:53:08.479814 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:08.464941    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-7stx7"
Oct 31 12:53:08.479814 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:08.464981    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c67de2/simulator-w4-c67de2-kserve-868679cf9b-gqjbx"
Oct 31 12:53:08.599517 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:08.548409    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-b9pn6" podStartSLOduration=185.548385924 podStartE2EDuration="3m5.548385924s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:50:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:53:07.048358647 +0000 UTC m=+1550.151426363" watchObservedRunningTime="2026-10-31 12:53:08.548385924 +0000 UTC m=+1551.651453637"
Oct 31 12:53:08.671528 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:08.646338    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-2-w5-b25d3b-kserve-76f798d59f-vvzhg" podUID="394ed76c-84ac-4612-9acb-a7e36cb68d89" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": context deadline exceeded"
Oct 31 12:53:08.681519 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:08.647681    5060 patch_prober.go:29] interesting pod/e2e-inf-a-4e8eb9-openshift-default-768d574d85-b9pn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" start-of-body=
Oct 31 12:53:08.681796 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:08.681766    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-b9pn6" podUID="41d07641-1c30-4e59-b402-fd7505e781ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused"
Oct 31 12:53:08.704740 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:08.704643    5060 patch_prober.go:29] interesting pod/e2e-inf-a-4e8eb9-openshift-default-768d574d85-zg66r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body=
Oct 31 12:53:08.705656 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:08.704927    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-zg66r" podUID="ea38d319-4637-47c0-bf7a-125b61dac2fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused"
Oct 31 12:53:08.706696 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:08.706673    5060 patch_prober.go:29] interesting pod/e2e-inf-a-4e8eb9-openshift-default-768d574d85-ds5zz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" start-of-body=
Oct 31 12:53:08.706875 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:08.706850    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-ds5zz" podUID="0070ea70-06e9-4733-8d11-ecb4cb993ff7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused"
Oct 31 12:53:08.991565 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:08.937612    5060 patch_prober.go:29] interesting pod/e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-vqcct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" start-of-body=
Oct 31 12:53:08.991565 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:08.937672    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-vqcct" podUID="b2705ec7-400e-4265-b213-5b58ba42cd4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused"
Oct 31 12:53:08.991565 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:08.964458    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1a7d0e-748bf567d5-j7bb7" event={"ID":"bcb6aa33-17fe-45ac-93a5-ff86419c2246","Type":"ContainerStarted","Data":"464aa43494598df7326debe86e667e43f5ad090d5bf196f25672da303e988785"}
Oct 31 12:53:08.991565 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:08.973520    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-7stx7"
Oct 31 12:53:08.991565 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:08.973579    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1a7d0e-748bf567d5-j7bb7"
Oct 31 12:53:09.019312 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:09.019138    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-7stx7"
Oct 31 12:53:09.089708 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:09.089672    5060 patch_prober.go:29] interesting pod/e2e-inf-a-e98df8-openshift-default-779d98479-vkrj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body=
Oct 31 12:53:09.090047 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:09.090018    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e98df8-openshift-default-779d98479-vkrj6" podUID="0d62f8e4-b062-4dd4-a23d-0f66566c83da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused"
Oct 31 12:53:09.227512 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:09.196875    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-7stx7" podStartSLOduration=183.196854031 podStartE2EDuration="3m3.196854031s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:50:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:53:07.567180253 +0000 UTC m=+1550.670247966" watchObservedRunningTime="2026-10-31 12:53:09.196854031 +0000 UTC m=+1552.299921745"
Oct 31 12:53:09.249572 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:09.231351    5060 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-e6c79d/premium-w2-e6c79d-kserve-6468bf46d4-h66f8" podUID="cd6a266f-8d12-4089-83ee-c75c716bf26c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:53:09.492925 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:09.463234    5060 patch_prober.go:29] interesting pod/e2e-inf-a-4e8eb9-openshift-default-768d574d85-b9pn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" start-of-body=
Oct 31 12:53:09.492925 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:09.463298    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-b9pn6" podUID="41d07641-1c30-4e59-b402-fd7505e781ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused"
Oct 31 12:53:09.492925 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:09.464201    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/premium-w2-e6c79d-kserve-6468bf46d4-h66f8" podUID="cd6a266f-8d12-4089-83ee-c75c716bf26c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Oct 31 12:53:09.492925 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:09.485167    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x2qzv" podUID="c247ed2a-aa90-424b-9b7e-4f2936d67e3f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded"
Oct 31 12:53:09.534092 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:09.494575    5060 patch_prober.go:29] interesting pod/e2e-inf-a-4e8eb9-openshift-default-768d574d85-zg66r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body=
Oct 31 12:53:09.534092 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:09.494631    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-zg66r" podUID="ea38d319-4637-47c0-bf7a-125b61dac2fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused"
Oct 31 12:53:09.562148 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:09.551655    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/simulator-w4-c67de2-kserve-868679cf9b-gqjbx" podUID="e104d7da-1fb2-4c99-a480-b4ae84c4f273" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:53:09.562148 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:09.558390    5060 patch_prober.go:29] interesting pod/e2e-inf-a-4e8eb9-openshift-default-768d574d85-ds5zz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" start-of-body=
Oct 31 12:53:09.562148 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:09.558445    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-ds5zz" podUID="0070ea70-06e9-4733-8d11-ecb4cb993ff7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused"
Oct 31 12:53:09.694281 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:09.684666    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-2-w5-b25d3b-kserve-76f798d59f-vvzhg" podUID="394ed76c-84ac-4612-9acb-a7e36cb68d89" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": context deadline exceeded"
Oct 31 12:53:09.951669 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:09.944791    5060 patch_prober.go:29] interesting pod/e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-vqcct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" start-of-body=
Oct 31 12:53:09.951669 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:09.944854    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-vqcct" podUID="b2705ec7-400e-4265-b213-5b58ba42cd4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused"
Oct 31 12:53:10.080085 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:10.080035    5060 patch_prober.go:29] interesting pod/e2e-inf-a-e43d8b-openshift-default-5578dcfc7f-5f9ss container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 12:53:10.080440 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:10.080407    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-e43d8b-openshift-default-5578dcfc7f-5f9ss" podUID="a0ba8ab3-65d2-4132-9896-80f1fb46b5c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:53:10.140526 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:10.132640    5060 patch_prober.go:29] interesting pod/e2e-inf-a-e98df8-openshift-default-779d98479-vkrj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body=
Oct 31 12:53:10.140526 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:10.132703    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e98df8-openshift-default-779d98479-vkrj6" podUID="0d62f8e4-b062-4dd4-a23d-0f66566c83da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused"
Oct 31 12:53:10.394564 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:10.369709    5060 patch_prober.go:29] interesting pod/e2e-inf-a-4e8eb9-openshift-default-768d574d85-b9pn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" start-of-body=
Oct 31 12:53:10.394564 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:10.369773    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-b9pn6" podUID="41d07641-1c30-4e59-b402-fd7505e781ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused"
Oct 31 12:53:10.509517 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:10.468792    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/premium-w2-e6c79d-kserve-6468bf46d4-h66f8" podUID="cd6a266f-8d12-4089-83ee-c75c716bf26c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Oct 31 12:53:10.510804 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:10.510036    5060 patch_prober.go:29] interesting pod/e2e-inf-a-4e8eb9-openshift-default-768d574d85-zg66r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body=
Oct 31 12:53:10.510804 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:10.510102    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-zg66r" podUID="ea38d319-4637-47c0-bf7a-125b61dac2fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused"
Oct 31 12:53:10.510804 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:10.510537    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x2qzv" podUID="c247ed2a-aa90-424b-9b7e-4f2936d67e3f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded"
Oct 31 12:53:10.599521 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:10.567743    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d176a5-57f6cd6df5-tzc5c"]
Oct 31 12:53:10.599521 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:10.568278    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/simulator-w4-c67de2-kserve-868679cf9b-gqjbx" podUID="e104d7da-1fb2-4c99-a480-b4ae84c4f273" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:53:10.646586 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:10.624608    5060 patch_prober.go:29] interesting pod/e2e-inf-a-4e8eb9-openshift-default-768d574d85-ds5zz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" start-of-body=
Oct 31 12:53:10.646586 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:10.624671    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-ds5zz" podUID="0070ea70-06e9-4733-8d11-ecb4cb993ff7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused"
Oct 31 12:53:10.646586 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:10.624707    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d176a5-57f6cd6df5-tzc5c"]
Oct 31 12:53:10.646586 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:10.624794    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c67de2/simulator-w4-c67de2-kserve-868679cf9b-gqjbx"
Oct 31 12:53:10.646586 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:10.625419    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d176a5-57f6cd6df5-tzc5c"
Oct 31 12:53:10.646586 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:10.642038    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-d176a5-dockercfg-4b7ln\""
Oct 31 12:53:10.654624 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:10.654583    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-d176a5\""
Oct 31 12:53:10.800588 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:10.742600    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-2-w5-b25d3b-kserve-76f798d59f-vvzhg" podUID="394ed76c-84ac-4612-9acb-a7e36cb68d89" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:53:11.059663 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:10.938628    5060 patch_prober.go:29] interesting pod/e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-vqcct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" start-of-body=
Oct 31 12:53:11.059663 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:10.938689    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-vqcct" podUID="b2705ec7-400e-4265-b213-5b58ba42cd4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused"
Oct 31 12:53:11.060117 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:11.060080    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-e43d8b-openshift-default-5578dcfc7f-5f9ss"]
Oct 31 12:53:11.060504 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:11.060445    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-e43d8b-openshift-default-5578dcfc7f-5f9ss" podUID="a0ba8ab3-65d2-4132-9896-80f1fb46b5c5" containerName="istio-proxy" containerID="cri-o://643c50baeec7939f03f59f571525c956c954acc4db8b464eb73930c894d69b75" gracePeriod=30
Oct 31 12:53:11.162649 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:11.162607    5060 patch_prober.go:29] interesting pod/e2e-inf-a-e98df8-openshift-default-779d98479-vkrj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body=
Oct 31 12:53:11.162947 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:11.162917    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e98df8-openshift-default-779d98479-vkrj6" podUID="0d62f8e4-b062-4dd4-a23d-0f66566c83da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused"
Oct 31 12:53:11.283953 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:11.280778    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ltmb\" (UniqueName: \"kubernetes.io/projected/40133e45-b954-420a-aa2c-8e5b9c873ef4-kube-api-access-7ltmb\") pod \"payload-pre-processing-e2e-inf-a-d176a5-57f6cd6df5-tzc5c\" (UID: \"40133e45-b954-420a-aa2c-8e5b9c873ef4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d176a5-57f6cd6df5-tzc5c"
Oct 31 12:53:11.283953 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:11.280943    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40133e45-b954-420a-aa2c-8e5b9c873ef4-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-d176a5-57f6cd6df5-tzc5c\" (UID: \"40133e45-b954-420a-aa2c-8e5b9c873ef4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d176a5-57f6cd6df5-tzc5c"
Oct 31 12:53:11.366574 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:11.363808    5060 patch_prober.go:29] interesting pod/e2e-inf-a-4e8eb9-openshift-default-768d574d85-b9pn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" start-of-body=
Oct 31 12:53:11.413766 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:11.413704    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-b9pn6" podUID="41d07641-1c30-4e59-b402-fd7505e781ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused"
Oct 31 12:53:11.414388 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:11.391412    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40133e45-b954-420a-aa2c-8e5b9c873ef4-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-d176a5-57f6cd6df5-tzc5c\" (UID: \"40133e45-b954-420a-aa2c-8e5b9c873ef4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d176a5-57f6cd6df5-tzc5c"
Oct 31 12:53:11.414621 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:11.390311    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40133e45-b954-420a-aa2c-8e5b9c873ef4-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-d176a5-57f6cd6df5-tzc5c\" (UID: \"40133e45-b954-420a-aa2c-8e5b9c873ef4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d176a5-57f6cd6df5-tzc5c"
Oct 31 12:53:11.482117 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:11.482082    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ltmb\" (UniqueName: \"kubernetes.io/projected/40133e45-b954-420a-aa2c-8e5b9c873ef4-kube-api-access-7ltmb\") pod \"payload-pre-processing-e2e-inf-a-d176a5-57f6cd6df5-tzc5c\" (UID: \"40133e45-b954-420a-aa2c-8e5b9c873ef4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d176a5-57f6cd6df5-tzc5c"
Oct 31 12:53:11.482622 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:11.481582    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/premium-w2-e6c79d-kserve-6468bf46d4-h66f8" podUID="cd6a266f-8d12-4089-83ee-c75c716bf26c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:53:11.577927 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:11.552856    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x2qzv" podUID="c247ed2a-aa90-424b-9b7e-4f2936d67e3f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded"
Oct 31 12:53:11.590543 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:11.480097    5060 patch_prober.go:29] interesting pod/e2e-inf-a-4e8eb9-openshift-default-768d574d85-zg66r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body=
Oct 31 12:53:11.590543 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:11.589295    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-zg66r" podUID="ea38d319-4637-47c0-bf7a-125b61dac2fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused"
Oct 31 12:53:11.598881 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:11.591664    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/simulator-w4-c67de2-kserve-868679cf9b-gqjbx" podUID="e104d7da-1fb2-4c99-a480-b4ae84c4f273" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded"
Oct 31 12:53:11.609619 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:11.607816    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1a7d0e-748bf567d5-j7bb7" podStartSLOduration=182.154039571 podStartE2EDuration="3m3.607793893s" totalImagesPullingTime="1.453754322s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:50:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:53:09.041941845 +0000 UTC m=+1552.145009559" watchObservedRunningTime="2026-10-31 12:53:11.607793893 +0000 UTC m=+1554.710861605"
Oct 31 12:53:11.645568 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:11.643199    5060 patch_prober.go:29] interesting pod/e2e-inf-a-4e8eb9-openshift-default-768d574d85-ds5zz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" start-of-body=
Oct 31 12:53:11.645568 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:11.643271    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-ds5zz" podUID="0070ea70-06e9-4733-8d11-ecb4cb993ff7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused"
Oct 31 12:53:11.800516 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:11.799529    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-2-w5-b25d3b-kserve-76f798d59f-vvzhg" podUID="394ed76c-84ac-4612-9acb-a7e36cb68d89" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:53:11.863929 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:11.860218    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ltmb\" (UniqueName: \"kubernetes.io/projected/40133e45-b954-420a-aa2c-8e5b9c873ef4-kube-api-access-7ltmb\") pod \"payload-pre-processing-e2e-inf-a-d176a5-57f6cd6df5-tzc5c\" (UID: \"40133e45-b954-420a-aa2c-8e5b9c873ef4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d176a5-57f6cd6df5-tzc5c"
Oct 31 12:53:11.966560 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:11.956608    5060 patch_prober.go:29] interesting pod/e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-vqcct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" start-of-body=
Oct 31 12:53:11.966560 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:11.956669    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-vqcct" podUID="b2705ec7-400e-4265-b213-5b58ba42cd4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused"
Oct 31 12:53:12.251205 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:12.249694    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d176a5-57f6cd6df5-tzc5c"
Oct 31 12:53:12.393355 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:12.351712    5060 patch_prober.go:29] interesting pod/e2e-inf-a-e98df8-openshift-default-779d98479-vkrj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body=
Oct 31 12:53:12.393355 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:12.351796    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e98df8-openshift-default-779d98479-vkrj6" podUID="0d62f8e4-b062-4dd4-a23d-0f66566c83da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused"
Oct 31 12:53:12.602544 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:12.587069    5060 patch_prober.go:29] interesting pod/e2e-inf-a-4e8eb9-openshift-default-768d574d85-b9pn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" start-of-body=
Oct 31 12:53:12.602544 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:12.587161    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-b9pn6" podUID="41d07641-1c30-4e59-b402-fd7505e781ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused"
Oct 31 12:53:12.615205 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:12.615148    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wnh97"]
Oct 31 12:53:12.616267 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:12.616228    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wnh97" podUID="330523c8-06c9-42a2-ac3f-9dd4ad62c392" containerName="istio-proxy" containerID="cri-o://d190e7520eebe52b0c749ad77fb372857ff3f35218eb3dc8fea952d63280f3d9" gracePeriod=30
Oct 31 12:53:12.646060 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:12.646011    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/simulator-w4-c67de2-kserve-868679cf9b-gqjbx" podUID="e104d7da-1fb2-4c99-a480-b4ae84c4f273" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:53:12.682408 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:12.672576    5060 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x2qzv" podUID="c247ed2a-aa90-424b-9b7e-4f2936d67e3f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": context deadline exceeded"
Oct 31 12:53:12.706535 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:12.704318    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x2qzv" podUID="c247ed2a-aa90-424b-9b7e-4f2936d67e3f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:53:12.706535 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:12.705177    5060 patch_prober.go:29] interesting pod/e2e-inf-a-4e8eb9-openshift-default-768d574d85-ds5zz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" start-of-body=
Oct 31 12:53:12.706535 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:12.705241    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-ds5zz" podUID="0070ea70-06e9-4733-8d11-ecb4cb993ff7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused"
Oct 31 12:53:12.708235 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:12.708190    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/premium-w2-e6c79d-kserve-6468bf46d4-h66f8" podUID="cd6a266f-8d12-4089-83ee-c75c716bf26c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Oct 31 12:53:12.804112 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:12.804072    5060 patch_prober.go:29] interesting pod/e2e-inf-a-4e8eb9-openshift-default-768d574d85-zg66r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body=
Oct 31 12:53:12.846509 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:12.833864    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-zg66r" podUID="ea38d319-4637-47c0-bf7a-125b61dac2fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused"
Oct 31 12:53:12.846509 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:12.837765    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-2-w5-b25d3b-kserve-76f798d59f-vvzhg" podUID="394ed76c-84ac-4612-9acb-a7e36cb68d89" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:53:13.319559 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:13.319517    5060 patch_prober.go:29] interesting pod/e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-vqcct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" start-of-body=
Oct 31 12:53:13.320348 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:13.319764    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-vqcct" podUID="b2705ec7-400e-4265-b213-5b58ba42cd4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused"
Oct 31 12:53:13.320348 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:13.319895    5060 patch_prober.go:29] interesting pod/e2e-inf-a-e98df8-openshift-default-779d98479-vkrj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body=
Oct 31 12:53:13.320348 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:13.319920    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e98df8-openshift-default-779d98479-vkrj6" podUID="0d62f8e4-b062-4dd4-a23d-0f66566c83da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused"
Oct 31 12:53:13.400771 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:13.400726    5060 patch_prober.go:29] interesting pod/e2e-inf-a-4e8eb9-openshift-default-768d574d85-b9pn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" start-of-body=
Oct 31 12:53:13.401081 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:13.401035    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-b9pn6" podUID="41d07641-1c30-4e59-b402-fd7505e781ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused"
Oct 31 12:53:13.716354 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:13.716245    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/premium-w2-e6c79d-kserve-6468bf46d4-h66f8" podUID="cd6a266f-8d12-4089-83ee-c75c716bf26c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:53:13.860906 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:13.767073    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x2qzv" podUID="c247ed2a-aa90-424b-9b7e-4f2936d67e3f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:53:13.860906 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:13.773507    5060 patch_prober.go:29] interesting pod/e2e-inf-a-4e8eb9-openshift-default-768d574d85-ds5zz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" start-of-body=
Oct 31 12:53:13.860906 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:13.821776    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-ds5zz" podUID="0070ea70-06e9-4733-8d11-ecb4cb993ff7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused"
Oct 31 12:53:13.860906 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:13.821665    5060 patch_prober.go:29] interesting pod/e2e-inf-a-4e8eb9-openshift-default-768d574d85-zg66r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body=
Oct 31 12:53:13.860906 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:13.821904    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-zg66r" podUID="ea38d319-4637-47c0-bf7a-125b61dac2fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused"
Oct 31 12:53:13.872555 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:13.863626    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-2-w5-b25d3b-kserve-76f798d59f-vvzhg" podUID="394ed76c-84ac-4612-9acb-a7e36cb68d89" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": context deadline exceeded"
Oct 31 12:53:13.886031 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:13.885971    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/simulator-w4-c67de2-kserve-868679cf9b-gqjbx" podUID="e104d7da-1fb2-4c99-a480-b4ae84c4f273" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded"
Oct 31 12:53:14.035667 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:14.035628    5060 patch_prober.go:29] interesting pod/e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-vqcct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" start-of-body=
Oct 31 12:53:14.056004 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:14.055574    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-vqcct" podUID="b2705ec7-400e-4265-b213-5b58ba42cd4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused"
Oct 31 12:53:14.058946 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:14.058908    5060 patch_prober.go:29] interesting pod/e2e-inf-a-e98df8-openshift-default-779d98479-vkrj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body=
Oct 31 12:53:14.059149 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:14.059123    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e98df8-openshift-default-779d98479-vkrj6" podUID="0d62f8e4-b062-4dd4-a23d-0f66566c83da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused"
Oct 31 12:53:14.217919 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:53:14.217087    5060 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.087s"
Oct 31 12:53:14.217919 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:14.217159    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-xmr62"]
Oct 31 12:53:14.217919 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:14.217218    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-dm6jq"]
Oct 31 12:53:14.217919 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:14.217427    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-dm6jq" podUID="6750d8d2-c91b-4d08-9553-2fe15afb061a" containerName="istio-proxy" containerID="cri-o://4a4fa34a0917cc911ea4ce7ecf3411bc3464d8a28cbdc5a67b9fc4e5b2ac6fa9" gracePeriod=30
Oct 31 12:53:14.217919 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:14.217723    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-xmr62" podUID="e1e6c7f5-d872-4549-b950-1e751b6374ab" containerName="istio-proxy" containerID="cri-o://15eb93212f666238582935bcfb7fb66c4776d563bb22273bc4eb80596e90659b" gracePeriod=30
Oct 31 12:53:14.346830 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:14.346738    5060 patch_prober.go:29] interesting pod/e2e-inf-a-4e8eb9-openshift-default-768d574d85-b9pn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" start-of-body=
Oct 31 12:53:14.347126 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:14.347086    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-b9pn6" podUID="41d07641-1c30-4e59-b402-fd7505e781ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused"
Oct 31 12:53:14.460234 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:14.458290    5060 patch_prober.go:29] interesting pod/e2e-inf-a-4e8eb9-openshift-default-768d574d85-zg66r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body=
Oct 31 12:53:14.460234 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:14.458350    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-zg66r" podUID="ea38d319-4637-47c0-bf7a-125b61dac2fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused"
Oct 31 12:53:14.513577 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:14.513537    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4e8eb9-hxhgd"]
Oct 31 12:53:14.549697 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:14.549663    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4e8eb9-hxhgd"]
Oct 31 12:53:14.562789 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:14.562751    5060 patch_prober.go:29] interesting pod/e2e-inf-a-4e8eb9-openshift-default-768d574d85-ds5zz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" start-of-body=
Oct 31 12:53:14.563057 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:14.563025    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-ds5zz" podUID="0070ea70-06e9-4733-8d11-ecb4cb993ff7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused"
Oct 31 12:53:14.965932 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:14.965851    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/simulator-w4-c67de2-kserve-868679cf9b-gqjbx" podUID="e104d7da-1fb2-4c99-a480-b4ae84c4f273" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:53:14.986224 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:14.969677    5060 patch_prober.go:29] interesting pod/e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-vqcct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" start-of-body=
Oct 31 12:53:14.986224 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:14.969728    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-vqcct" podUID="b2705ec7-400e-4265-b213-5b58ba42cd4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused"
Oct 31 12:53:14.986224 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:14.983908    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/premium-w2-e6c79d-kserve-6468bf46d4-h66f8" podUID="cd6a266f-8d12-4089-83ee-c75c716bf26c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Oct 31 12:53:14.986224 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:14.984257    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x2qzv" podUID="c247ed2a-aa90-424b-9b7e-4f2936d67e3f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:53:14.986914 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:14.986882    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-2-w5-b25d3b-kserve-76f798d59f-vvzhg" podUID="394ed76c-84ac-4612-9acb-a7e36cb68d89" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": context deadline exceeded"
Oct 31 12:53:15.094211 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:15.071011    5060 patch_prober.go:29] interesting pod/e2e-inf-a-e98df8-openshift-default-779d98479-vkrj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body=
Oct 31 12:53:15.094211 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:15.071069    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e98df8-openshift-default-779d98479-vkrj6" podUID="0d62f8e4-b062-4dd4-a23d-0f66566c83da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused"
Oct 31 12:53:15.343634 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:15.342152    5060 patch_prober.go:29] interesting pod/e2e-inf-a-4e8eb9-openshift-default-768d574d85-b9pn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" start-of-body=
Oct 31 12:53:15.343634 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:15.342216    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-b9pn6" podUID="41d07641-1c30-4e59-b402-fd7505e781ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused"
Oct 31 12:53:15.462522 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:15.451854    5060 patch_prober.go:29] interesting pod/e2e-inf-a-4e8eb9-openshift-default-768d574d85-zg66r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body=
Oct 31 12:53:15.462522 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:15.451918    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-zg66r" podUID="ea38d319-4637-47c0-bf7a-125b61dac2fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused"
Oct 31 12:53:15.565512 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:15.535842    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01233c4a-9626-42cc-b7fb-e58e8f1c0278" path="/var/lib/kubelet/pods/01233c4a-9626-42cc-b7fb-e58e8f1c0278/volumes"
Oct 31 12:53:15.607674 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:15.605738    5060 patch_prober.go:29] interesting pod/e2e-inf-a-4e8eb9-openshift-default-768d574d85-ds5zz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" start-of-body=
Oct 31 12:53:15.607674 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:15.605837    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-ds5zz" podUID="0070ea70-06e9-4733-8d11-ecb4cb993ff7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused"
Oct 31 12:53:15.967150 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:15.943851    5060 patch_prober.go:29] interesting pod/e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-vqcct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" start-of-body=
Oct 31 12:53:15.967150 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:15.943910    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-vqcct" podUID="b2705ec7-400e-4265-b213-5b58ba42cd4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused"
Oct 31 12:53:15.967150 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:15.945393    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-vqcct"
Oct 31 12:53:16.104961 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:15.983274    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/simulator-w4-c67de2-kserve-868679cf9b-gqjbx" podUID="e104d7da-1fb2-4c99-a480-b4ae84c4f273" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:53:16.242511 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:15.991588    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-2-w5-b25d3b-kserve-76f798d59f-vvzhg" podUID="394ed76c-84ac-4612-9acb-a7e36cb68d89" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:53:16.243239 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:16.008733    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x2qzv" podUID="c247ed2a-aa90-424b-9b7e-4f2936d67e3f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:53:16.269053 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:16.008774    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/premium-w2-e6c79d-kserve-6468bf46d4-h66f8" podUID="cd6a266f-8d12-4089-83ee-c75c716bf26c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Oct 31 12:53:16.269690 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:16.111008    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-e98df8-openshift-default-779d98479-vkrj6"
Oct 31 12:53:16.297881 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:16.140937    5060 patch_prober.go:29] interesting pod/e2e-inf-a-e98df8-openshift-default-779d98479-vkrj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body=
Oct 31 12:53:16.313347 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:16.313305    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e98df8-openshift-default-779d98479-vkrj6" podUID="0d62f8e4-b062-4dd4-a23d-0f66566c83da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused"
Oct 31 12:53:16.338994 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:16.338959    5060 patch_prober.go:29] interesting pod/e2e-inf-a-4e8eb9-openshift-default-768d574d85-b9pn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" start-of-body=
Oct 31 12:53:16.372831 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:16.372786    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-b9pn6" podUID="41d07641-1c30-4e59-b402-fd7505e781ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused"
Oct 31 12:53:16.576872 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:16.574235    5060 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-2-w5-b25d3b-kserve-76f798d59f-vvzhg" podUID="394ed76c-84ac-4612-9acb-a7e36cb68d89" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:53:16.691533 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:16.663940    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-ds5zz"
Oct 31 12:53:16.691533 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:16.664885    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-zg66r"
Oct 31 12:53:16.795004 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:16.728972    5060 patch_prober.go:29] interesting pod/e2e-inf-a-4e8eb9-openshift-default-768d574d85-ds5zz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" start-of-body=
Oct 31 12:53:16.795004 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:16.729035    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-ds5zz" podUID="0070ea70-06e9-4733-8d11-ecb4cb993ff7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused"
Oct 31 12:53:16.795004 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:16.788281    5060 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-c67de2/simulator-w4-c67de2-kserve-868679cf9b-gqjbx" podUID="e104d7da-1fb2-4c99-a480-b4ae84c4f273" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:53:16.854951 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:16.854865    5060 patch_prober.go:29] interesting pod/e2e-inf-a-4e8eb9-openshift-default-768d574d85-zg66r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body=
Oct 31 12:53:16.855191 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:16.855164    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-zg66r" podUID="ea38d319-4637-47c0-bf7a-125b61dac2fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused"
Oct 31 12:53:16.993104 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:16.927673    5060 patch_prober.go:29] interesting pod/e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-vqcct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" start-of-body=
Oct 31 12:53:16.993104 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:16.927753    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-vqcct" podUID="b2705ec7-400e-4265-b213-5b58ba42cd4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused"
Oct 31 12:53:16.993104 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:16.990018    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e6c79d/premium-w2-e6c79d-kserve-6468bf46d4-h66f8"
Oct 31 12:53:17.191520 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:17.158171    5060 patch_prober.go:29] interesting pod/e2e-inf-a-e98df8-openshift-default-779d98479-vkrj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body=
Oct 31 12:53:17.191520 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:17.158257    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e98df8-openshift-default-779d98479-vkrj6" podUID="0d62f8e4-b062-4dd4-a23d-0f66566c83da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused"
Oct 31 12:53:17.301545 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:17.293070    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-2-w5-b25d3b-kserve-76f798d59f-vvzhg" podUID="394ed76c-84ac-4612-9acb-a7e36cb68d89" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:53:17.455680 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:17.454372    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/simulator-w4-c67de2-kserve-868679cf9b-gqjbx" podUID="e104d7da-1fb2-4c99-a480-b4ae84c4f273" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded"
Oct 31 12:53:17.520524 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:17.513631    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x2qzv" podUID="c247ed2a-aa90-424b-9b7e-4f2936d67e3f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded"
Oct 31 12:53:17.563527 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:17.562756    5060 patch_prober.go:29] interesting pod/e2e-inf-a-4e8eb9-openshift-default-768d574d85-b9pn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" start-of-body=
Oct 31 12:53:17.563527 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:17.562836    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-b9pn6" podUID="41d07641-1c30-4e59-b402-fd7505e781ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused"
Oct 31 12:53:17.690755 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:17.677732    5060 patch_prober.go:29] interesting pod/e2e-inf-a-4e8eb9-openshift-default-768d574d85-ds5zz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" start-of-body=
Oct 31 12:53:17.690755 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:17.677805    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-ds5zz" podUID="0070ea70-06e9-4733-8d11-ecb4cb993ff7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused"
Oct 31 12:53:17.935515 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:17.931581    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d176a5-57f6cd6df5-tzc5c"]
Oct 31 12:53:17.935515 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:17.931789    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1a7d0e-748bf567d5-j7bb7"
Oct 31 12:53:17.942621 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:17.939573    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-b9pn6"
Oct 31 12:53:18.167557 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:18.143225    5060 patch_prober.go:29] interesting pod/e2e-inf-a-e98df8-openshift-default-779d98479-vkrj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body=
Oct 31 12:53:18.270772 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:18.270720    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e98df8-openshift-default-779d98479-vkrj6" podUID="0d62f8e4-b062-4dd4-a23d-0f66566c83da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused"
Oct 31 12:53:18.333511 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:18.309568    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vmzbr" podUID="7049f597-e139-4f0e-afbd-ec6ea0effab9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:53:18.385597 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:18.369656    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d176a5-57f6cd6df5-tzc5c" event={"ID":"40133e45-b954-420a-aa2c-8e5b9c873ef4","Type":"ContainerStarted","Data":"10e04fda01a3c95d435e192e8ce58ac94f335dd629be14e01f4e09b4ab3c90c7"}
Oct 31 12:53:18.439588 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:18.439543    5060 patch_prober.go:29] interesting pod/e2e-inf-a-4e8eb9-openshift-default-768d574d85-zg66r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body=
Oct 31 12:53:18.439869 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:18.439838    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-zg66r" podUID="ea38d319-4637-47c0-bf7a-125b61dac2fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused"
Oct 31 12:53:18.448890 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:18.448852    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-2-w5-b25d3b-kserve-76f798d59f-vvzhg" podUID="394ed76c-84ac-4612-9acb-a7e36cb68d89" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": context deadline exceeded"
Oct 31 12:53:18.575703 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:18.564673    5060 patch_prober.go:29] interesting pod/e2e-inf-a-4e8eb9-openshift-default-768d574d85-b9pn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" start-of-body=
Oct 31 12:53:18.979027 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:18.978983    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-b9pn6" podUID="41d07641-1c30-4e59-b402-fd7505e781ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused"
Oct 31 12:53:19.047709 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:53:19.047667    5060 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod330523c8_06c9_42a2_ac3f_9dd4ad62c392.slice/crio-conmon-d190e7520eebe52b0c749ad77fb372857ff3f35218eb3dc8fea952d63280f3d9.scope\": RecentStats: unable to find data in memory cache]"
Oct 31 12:53:19.191515 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:18.642028    5060 patch_prober.go:29] interesting pod/e2e-inf-a-4e8eb9-openshift-default-768d574d85-zg66r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body=
Oct 31 12:53:19.191515 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:19.149637    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-zg66r" podUID="ea38d319-4637-47c0-bf7a-125b61dac2fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused"
Oct 31 12:53:19.191515 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:18.931517    5060 patch_prober.go:29] interesting pod/e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-vqcct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" start-of-body=
Oct 31 12:53:19.191515 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:19.149715    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-vqcct" podUID="b2705ec7-400e-4265-b213-5b58ba42cd4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused"
Oct 31 12:53:19.228518 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:19.203617    5060 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-e6c79d/premium-w2-e6c79d-kserve-6468bf46d4-h66f8" podUID="cd6a266f-8d12-4089-83ee-c75c716bf26c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": context deadline exceeded"
Oct 31 12:53:19.323507 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:19.206011    5060 patch_prober.go:29] interesting pod/e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-vqcct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" start-of-body=
Oct 31 12:53:19.323507 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:19.234132    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-vqcct" podUID="b2705ec7-400e-4265-b213-5b58ba42cd4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused"
Oct 31 12:53:19.323507 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:19.215576    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/premium-w2-e6c79d-kserve-6468bf46d4-h66f8" podUID="cd6a266f-8d12-4089-83ee-c75c716bf26c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:53:19.349515 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:53:19.207271    5060 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40133e45_b954_420a_aa2c_8e5b9c873ef4.slice/crio-conmon-1e156eee88d8e158f3ae297081e63eee06fef2459d71a4f8ade2c3ec2e726d39.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod330523c8_06c9_42a2_ac3f_9dd4ad62c392.slice/crio-d190e7520eebe52b0c749ad77fb372857ff3f35218eb3dc8fea952d63280f3d9.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod330523c8_06c9_42a2_ac3f_9dd4ad62c392.slice/crio-conmon-d190e7520eebe52b0c749ad77fb372857ff3f35218eb3dc8fea952d63280f3d9.scope\": RecentStats: unable to find data in memory cache]"
Oct 31 12:53:19.441723 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:19.216185    5060 patch_prober.go:29] interesting pod/e2e-inf-a-4e8eb9-openshift-default-768d574d85-ds5zz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" start-of-body=
Oct 31 12:53:19.441723 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:19.371669    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-ds5zz" podUID="0070ea70-06e9-4733-8d11-ecb4cb993ff7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused"
Oct 31 12:53:19.456576 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:19.452573    5060 patch_prober.go:29] interesting pod/e2e-inf-a-e98df8-openshift-default-779d98479-vkrj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body=
Oct 31 12:53:19.500772 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:19.500725    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-2-w5-b25d3b-kserve-76f798d59f-vvzhg" podUID="394ed76c-84ac-4612-9acb-a7e36cb68d89" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": context deadline exceeded"
Oct 31 12:53:19.605578 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:19.600322    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x2qzv" podUID="c247ed2a-aa90-424b-9b7e-4f2936d67e3f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded"
Oct 31 12:53:19.642424 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:19.641365    5060 patch_prober.go:29] interesting pod/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-dm6jq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" start-of-body=
Oct 31 12:53:19.701520 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:19.679821    5060 patch_prober.go:29] interesting pod/e2e-inf-a-4e8eb9-openshift-default-768d574d85-zg66r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body=
Oct 31 12:53:19.741523 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:19.682243    5060 patch_prober.go:29] interesting pod/e2e-inf-a-4e8eb9-openshift-default-768d574d85-b9pn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" start-of-body=
Oct 31 12:53:19.741523 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:19.725155    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-b9pn6" podUID="41d07641-1c30-4e59-b402-fd7505e781ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused"
Oct 31 12:53:19.799525 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:19.795721    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-zg66r" podUID="ea38d319-4637-47c0-bf7a-125b61dac2fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused"
Oct 31 12:53:20.003560 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:19.460566    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e98df8-openshift-default-779d98479-vkrj6" podUID="0d62f8e4-b062-4dd4-a23d-0f66566c83da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused"
Oct 31 12:53:20.118637 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:19.682260    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-dm6jq" podUID="6750d8d2-c91b-4d08-9553-2fe15afb061a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused"
Oct 31 12:53:20.159514 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:19.693807    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/simulator-w4-c67de2-kserve-868679cf9b-gqjbx" podUID="e104d7da-1fb2-4c99-a480-b4ae84c4f273" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:53:20.198782 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:20.190912    5060 patch_prober.go:29] interesting pod/e2e-inf-a-4e8eb9-openshift-default-768d574d85-ds5zz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" start-of-body=
Oct 31 12:53:20.198782 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:20.191012    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-ds5zz" podUID="0070ea70-06e9-4733-8d11-ecb4cb993ff7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused"
Oct 31 12:53:20.241522 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:20.236798    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/premium-w2-e6c79d-kserve-6468bf46d4-h66f8" podUID="cd6a266f-8d12-4089-83ee-c75c716bf26c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:53:20.369062 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:20.369006    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-rm7cl" podUID="8292eaa8-0a02-46be-8654-3c4136ff604f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:53:20.521856 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:20.501479    5060 patch_prober.go:29] interesting pod/e2e-inf-a-e98df8-openshift-default-779d98479-vkrj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body=
Oct 31 12:53:20.521856 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:20.501558    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e98df8-openshift-default-779d98479-vkrj6" podUID="0d62f8e4-b062-4dd4-a23d-0f66566c83da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused"
Oct 31 12:53:20.523741 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:20.523695    5060 patch_prober.go:29] interesting pod/e2e-inf-a-4e8eb9-openshift-default-768d574d85-zg66r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body=
Oct 31 12:53:20.523885 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:20.523757    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-zg66r" podUID="ea38d319-4637-47c0-bf7a-125b61dac2fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused"
Oct 31 12:53:20.523966 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:20.523941    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-rbn8q" podUID="dea929f7-61c3-4921-9bdb-106f763e067e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:53:20.528613 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:20.528571    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-2-w5-b25d3b-kserve-76f798d59f-vvzhg" podUID="394ed76c-84ac-4612-9acb-a7e36cb68d89" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:53:20.541714 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:20.536658    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vmzbr" podUID="7049f597-e139-4f0e-afbd-ec6ea0effab9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:53:20.579513 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:20.554891    5060 patch_prober.go:29] interesting pod/e2e-inf-a-4e8eb9-openshift-default-768d574d85-ds5zz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" start-of-body=
Oct 31 12:53:20.579513 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:20.554943    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-ds5zz" podUID="0070ea70-06e9-4733-8d11-ecb4cb993ff7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused"
Oct 31 12:53:20.655818 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:20.621083    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x2qzv" podUID="c247ed2a-aa90-424b-9b7e-4f2936d67e3f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:53:20.711423 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:20.711389    5060 generic.go:356] "Generic (PLEG): container finished" podID="330523c8-06c9-42a2-ac3f-9dd4ad62c392" containerID="d190e7520eebe52b0c749ad77fb372857ff3f35218eb3dc8fea952d63280f3d9" exitCode=0
Oct 31 12:53:20.761515 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:20.739629    5060 patch_prober.go:29] interesting pod/e2e-inf-a-4e8eb9-openshift-default-768d574d85-b9pn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" start-of-body=
Oct 31 12:53:20.761515 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:20.739688    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-b9pn6" podUID="41d07641-1c30-4e59-b402-fd7505e781ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused"
Oct 31 12:53:20.798753 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:53:20.798716    5060 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.756s"
Oct 31 12:53:20.911570 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:20.818973    5060 generic.go:356] "Generic (PLEG): container finished" podID="6750d8d2-c91b-4d08-9553-2fe15afb061a" containerID="4a4fa34a0917cc911ea4ce7ecf3411bc3464d8a28cbdc5a67b9fc4e5b2ac6fa9" exitCode=0
Oct 31 12:53:20.911570 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:20.908453    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1a7d0e-bsjb6"]
Oct 31 12:53:21.064732 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:21.064698    5060 patch_prober.go:29] interesting pod/e2e-inf-a-e98df8-openshift-default-779d98479-vkrj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body=
Oct 31 12:53:21.140693 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:21.140650    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e98df8-openshift-default-779d98479-vkrj6" podUID="0d62f8e4-b062-4dd4-a23d-0f66566c83da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused"
Oct 31 12:53:21.179299 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:21.084508    5060 patch_prober.go:29] interesting pod/e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-vqcct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 12:53:21.179666 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:21.179638    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-vqcct" podUID="b2705ec7-400e-4265-b213-5b58ba42cd4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:53:21.206520 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:21.179569    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/simulator-w4-c67de2-kserve-868679cf9b-gqjbx" podUID="e104d7da-1fb2-4c99-a480-b4ae84c4f273" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:53:21.206520 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:21.092934    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d176a5-57f6cd6df5-tzc5c" podStartSLOduration=13.092913749 podStartE2EDuration="13.092913749s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:53:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:53:20.894169154 +0000 UTC m=+1563.997236868" watchObservedRunningTime="2026-10-31 12:53:21.092913749 +0000 UTC m=+1564.195981463"
Oct 31 12:53:21.262513 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:21.191671    5060 patch_prober.go:29] interesting pod/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wnh97 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body=
Oct 31 12:53:21.262513 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:21.241592    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wnh97" podUID="330523c8-06c9-42a2-ac3f-9dd4ad62c392" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused"
Oct 31 12:53:21.262513 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:21.241996    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/premium-w2-e6c79d-kserve-6468bf46d4-h66f8" podUID="cd6a266f-8d12-4089-83ee-c75c716bf26c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:53:21.369513 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:21.362710    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1a7d0e-bsjb6"]
Oct 31 12:53:21.369513 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:21.362808    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e6c79d/premium-w2-e6c79d-kserve-6468bf46d4-h66f8"
Oct 31 12:53:21.369513 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:21.362914    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d176a5-57f6cd6df5-tzc5c"
Oct 31 12:53:21.370919 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:21.370888    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1a7d0e-bsjb6"
Oct 31 12:53:21.383995 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:21.383955    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 12:53:21.385379 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:21.385350    5060 patch_prober.go:29] interesting pod/e2e-inf-a-4e8eb9-openshift-default-768d574d85-b9pn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" start-of-body=
Oct 31 12:53:21.385615 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:21.385589    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-b9pn6" podUID="41d07641-1c30-4e59-b402-fd7505e781ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused"
Oct 31 12:53:21.506617 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:21.486607    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9q7l\" (UniqueName: \"kubernetes.io/projected/fca0927c-616b-4766-8cb2-cae87d082f01-kube-api-access-d9q7l\") pod \"maas-api-revoke-keys-e2e-inf-b-1a7d0e-bsjb6\" (UID: \"fca0927c-616b-4766-8cb2-cae87d082f01\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1a7d0e-bsjb6"
Oct 31 12:53:21.506617 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:21.486661    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fca0927c-616b-4766-8cb2-cae87d082f01-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-1a7d0e-bsjb6\" (UID: \"fca0927c-616b-4766-8cb2-cae87d082f01\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1a7d0e-bsjb6"
Oct 31 12:53:21.506617 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:21.486880    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fca0927c-616b-4766-8cb2-cae87d082f01-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-1a7d0e-bsjb6\" (UID: \"fca0927c-616b-4766-8cb2-cae87d082f01\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1a7d0e-bsjb6"
Oct 31 12:53:21.566987 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:21.563608    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-2-w5-b25d3b-kserve-76f798d59f-vvzhg" podUID="394ed76c-84ac-4612-9acb-a7e36cb68d89" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:53:21.602084 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:21.591407    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9q7l\" (UniqueName: \"kubernetes.io/projected/fca0927c-616b-4766-8cb2-cae87d082f01-kube-api-access-d9q7l\") pod \"maas-api-revoke-keys-e2e-inf-b-1a7d0e-bsjb6\" (UID: \"fca0927c-616b-4766-8cb2-cae87d082f01\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1a7d0e-bsjb6"
Oct 31 12:53:21.602084 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:21.591540    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fca0927c-616b-4766-8cb2-cae87d082f01-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-1a7d0e-bsjb6\" (UID: \"fca0927c-616b-4766-8cb2-cae87d082f01\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1a7d0e-bsjb6"
Oct 31 12:53:21.602084 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:21.591729    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fca0927c-616b-4766-8cb2-cae87d082f01-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-1a7d0e-bsjb6\" (UID: \"fca0927c-616b-4766-8cb2-cae87d082f01\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1a7d0e-bsjb6"
Oct 31 12:53:21.602084 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:21.595077    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fca0927c-616b-4766-8cb2-cae87d082f01-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-1a7d0e-bsjb6\" (UID: \"fca0927c-616b-4766-8cb2-cae87d082f01\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1a7d0e-bsjb6"
Oct 31 12:53:21.602084 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:21.595297    5060 patch_prober.go:29] interesting pod/e2e-inf-a-4e8eb9-openshift-default-768d574d85-zg66r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body=
Oct 31 12:53:21.602084 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:21.595378    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-zg66r" podUID="ea38d319-4637-47c0-bf7a-125b61dac2fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused"
Oct 31 12:53:21.662328 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:21.662296    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7wsmb\""
Oct 31 12:53:21.835576 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:21.799482    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-vqcct"
Oct 31 12:53:21.835576 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:21.799797    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wnh97" event={"ID":"330523c8-06c9-42a2-ac3f-9dd4ad62c392","Type":"ContainerDied","Data":"d190e7520eebe52b0c749ad77fb372857ff3f35218eb3dc8fea952d63280f3d9"}
Oct 31 12:53:21.835576 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:21.799835    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d176a5-57f6cd6df5-tzc5c" event={"ID":"40133e45-b954-420a-aa2c-8e5b9c873ef4","Type":"ContainerStarted","Data":"1e156eee88d8e158f3ae297081e63eee06fef2459d71a4f8ade2c3ec2e726d39"}
Oct 31 12:53:21.835576 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:21.799891    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x2qzv"
Oct 31 12:53:21.835576 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:21.799909    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-dm6jq" event={"ID":"6750d8d2-c91b-4d08-9553-2fe15afb061a","Type":"ContainerDied","Data":"4a4fa34a0917cc911ea4ce7ecf3411bc3464d8a28cbdc5a67b9fc4e5b2ac6fa9"}
Oct 31 12:53:21.835576 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:21.801518    5060 scope.go:122] "RemoveContainer" containerID="c81f960cbd8694792bbef5bd454b1dcd679f0054ccc4a589d9e6a72eb1ed5557"
Oct 31 12:53:21.835576 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:21.802709    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-vqcct"
Oct 31 12:53:21.950156 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:21.950113    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-2-w5-b25d3b-kserve-76f798d59f-vvzhg"
Oct 31 12:53:22.144561 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:22.133232    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9q7l\" (UniqueName: \"kubernetes.io/projected/fca0927c-616b-4766-8cb2-cae87d082f01-kube-api-access-d9q7l\") pod \"maas-api-revoke-keys-e2e-inf-b-1a7d0e-bsjb6\" (UID: \"fca0927c-616b-4766-8cb2-cae87d082f01\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1a7d0e-bsjb6"
Oct 31 12:53:22.144561 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:22.134586    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fca0927c-616b-4766-8cb2-cae87d082f01-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-1a7d0e-bsjb6\" (UID: \"fca0927c-616b-4766-8cb2-cae87d082f01\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1a7d0e-bsjb6"
Oct 31 12:53:22.214576 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:22.187425    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1a7d0e-bsjb6"
Oct 31 12:53:22.214576 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:22.210113    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c67de2/simulator-w4-c67de2-kserve-868679cf9b-gqjbx" podUID="e104d7da-1fb2-4c99-a480-b4ae84c4f273" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:53:22.317713 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:22.256952    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/premium-w2-e6c79d-kserve-6468bf46d4-h66f8" podUID="cd6a266f-8d12-4089-83ee-c75c716bf26c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:53:22.359743 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:22.358917    5060 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x2qzv" podUID="c247ed2a-aa90-424b-9b7e-4f2936d67e3f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:53:22.525531 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:22.525038    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x2qzv" podUID="c247ed2a-aa90-424b-9b7e-4f2936d67e3f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:53:22.560588 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:22.560556    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-e98df8-openshift-default-779d98479-vkrj6"
Oct 31 12:53:22.569888 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:22.569610    5060 patch_prober.go:29] interesting pod/e2e-inf-a-4e8eb9-openshift-default-768d574d85-ds5zz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.217:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 12:53:22.569888 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:22.569670    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-ds5zz" podUID="0070ea70-06e9-4733-8d11-ecb4cb993ff7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.217:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:53:22.570462 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:22.570440    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-e98df8-openshift-default-779d98479-vkrj6"
Oct 31 12:53:22.575141 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:22.575116    5060 generic.go:356] "Generic (PLEG): container finished" podID="e1e6c7f5-d872-4549-b950-1e751b6374ab" containerID="15eb93212f666238582935bcfb7fb66c4776d563bb22273bc4eb80596e90659b" exitCode=0
Oct 31 12:53:22.575401 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:22.575278    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-xmr62" event={"ID":"e1e6c7f5-d872-4549-b950-1e751b6374ab","Type":"ContainerDied","Data":"15eb93212f666238582935bcfb7fb66c4776d563bb22273bc4eb80596e90659b"}
Oct 31 12:53:22.601004 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:22.598106    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c67de2/simulator-w4-c67de2-kserve-868679cf9b-gqjbx"
Oct 31 12:53:22.680380 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:22.679582    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e6c79d/premium-w2-e6c79d-kserve-6468bf46d4-h66f8"
Oct 31 12:53:22.747360 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:22.738734    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-vxzjg"]
Oct 31 12:53:22.747360 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:22.746836    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-vxzjg" podUID="20fa8b33-c9e6-4fb0-9f3c-c8254dece2f6" containerName="istio-proxy" containerID="cri-o://6c60c3bdad21a71ead4fe8545b4a414aae5df9b36a2afad4d6cc8b5c09170636" gracePeriod=30
Oct 31 12:53:22.747360 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:22.746968    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-zg66r"]
Oct 31 12:53:22.747360 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:22.747161    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-zg66r" podUID="ea38d319-4637-47c0-bf7a-125b61dac2fe" containerName="istio-proxy" containerID="cri-o://7852c16adc99dafa80d1c9727e019f8620e0370ced0b974d88f7a36f05a969b9" gracePeriod=30
Oct 31 12:53:22.776092 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:22.776009    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-ds5zz"]
Oct 31 12:53:22.776540 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:22.776505    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-ds5zz" podUID="0070ea70-06e9-4733-8d11-ecb4cb993ff7" containerName="istio-proxy" containerID="cri-o://854af3eb3ad954edf26df321b78fd4ad65ec92e0b1d06ab91d690d848515f96b" gracePeriod=30
Oct 31 12:53:22.789989 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:22.789954    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-b9pn6"]
Oct 31 12:53:22.790844 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:22.790713    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-b9pn6" podUID="41d07641-1c30-4e59-b402-fd7505e781ba" containerName="istio-proxy" containerID="cri-o://dfb44bb00e52e91a04fb972e54c41471fb0f804fc6f3769d653cbd1212f00dea" gracePeriod=30
Oct 31 12:53:22.797681 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:22.797644    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-b9pn6"
Oct 31 12:53:22.860459 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:22.860354    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wnh97"
Oct 31 12:53:22.927382 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:22.927321    5060 patch_prober.go:29] interesting pod/e2e-inf-a-4e8eb9-openshift-default-768d574d85-ds5zz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 12:53:22.927924 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:22.927701    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-ds5zz" podUID="0070ea70-06e9-4733-8d11-ecb4cb993ff7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 12:53:23.015726 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:23.002333    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/330523c8-06c9-42a2-ac3f-9dd4ad62c392-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/330523c8-06c9-42a2-ac3f-9dd4ad62c392-credential-socket\") pod \"330523c8-06c9-42a2-ac3f-9dd4ad62c392\" (UID: \"330523c8-06c9-42a2-ac3f-9dd4ad62c392\") "
Oct 31 12:53:23.015726 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:23.002445    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/330523c8-06c9-42a2-ac3f-9dd4ad62c392-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/330523c8-06c9-42a2-ac3f-9dd4ad62c392-workload-socket\") pod \"330523c8-06c9-42a2-ac3f-9dd4ad62c392\" (UID: \"330523c8-06c9-42a2-ac3f-9dd4ad62c392\") "
Oct 31 12:53:23.015726 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:23.002480    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/330523c8-06c9-42a2-ac3f-9dd4ad62c392-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/330523c8-06c9-42a2-ac3f-9dd4ad62c392-istio-envoy\") pod \"330523c8-06c9-42a2-ac3f-9dd4ad62c392\" (UID: \"330523c8-06c9-42a2-ac3f-9dd4ad62c392\") "
Oct 31 12:53:23.015726 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:23.002564    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/330523c8-06c9-42a2-ac3f-9dd4ad62c392-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/330523c8-06c9-42a2-ac3f-9dd4ad62c392-istio-data\") pod \"330523c8-06c9-42a2-ac3f-9dd4ad62c392\" (UID: \"330523c8-06c9-42a2-ac3f-9dd4ad62c392\") "
Oct 31 12:53:23.015726 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:23.002603    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/330523c8-06c9-42a2-ac3f-9dd4ad62c392-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/330523c8-06c9-42a2-ac3f-9dd4ad62c392-workload-certs\") pod \"330523c8-06c9-42a2-ac3f-9dd4ad62c392\" (UID: \"330523c8-06c9-42a2-ac3f-9dd4ad62c392\") "
Oct 31 12:53:23.015726 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:23.002729    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/330523c8-06c9-42a2-ac3f-9dd4ad62c392-istio-token\" (UniqueName: \"kubernetes.io/projected/330523c8-06c9-42a2-ac3f-9dd4ad62c392-istio-token\") pod \"330523c8-06c9-42a2-ac3f-9dd4ad62c392\" (UID: \"330523c8-06c9-42a2-ac3f-9dd4ad62c392\") "
Oct 31 12:53:23.015726 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:23.002862    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/330523c8-06c9-42a2-ac3f-9dd4ad62c392-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/330523c8-06c9-42a2-ac3f-9dd4ad62c392-istiod-ca-cert\") pod \"330523c8-06c9-42a2-ac3f-9dd4ad62c392\" (UID: \"330523c8-06c9-42a2-ac3f-9dd4ad62c392\") "
Oct 31 12:53:23.015726 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:23.002927    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/330523c8-06c9-42a2-ac3f-9dd4ad62c392-kube-api-access-sm4q5\" (UniqueName: \"kubernetes.io/projected/330523c8-06c9-42a2-ac3f-9dd4ad62c392-kube-api-access-sm4q5\") pod \"330523c8-06c9-42a2-ac3f-9dd4ad62c392\" (UID: \"330523c8-06c9-42a2-ac3f-9dd4ad62c392\") "
Oct 31 12:53:23.015726 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:23.002980    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/330523c8-06c9-42a2-ac3f-9dd4ad62c392-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/330523c8-06c9-42a2-ac3f-9dd4ad62c392-istio-podinfo\") pod \"330523c8-06c9-42a2-ac3f-9dd4ad62c392\" (UID: \"330523c8-06c9-42a2-ac3f-9dd4ad62c392\") "
Oct 31 12:53:23.015726 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:23.005023    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/330523c8-06c9-42a2-ac3f-9dd4ad62c392-istio-data" pod "330523c8-06c9-42a2-ac3f-9dd4ad62c392" (UID: "330523c8-06c9-42a2-ac3f-9dd4ad62c392"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:53:23.015726 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:23.005292    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/330523c8-06c9-42a2-ac3f-9dd4ad62c392-credential-socket" pod "330523c8-06c9-42a2-ac3f-9dd4ad62c392" (UID: "330523c8-06c9-42a2-ac3f-9dd4ad62c392"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:53:23.015726 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:23.010370    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/330523c8-06c9-42a2-ac3f-9dd4ad62c392-istiod-ca-cert" pod "330523c8-06c9-42a2-ac3f-9dd4ad62c392" (UID: "330523c8-06c9-42a2-ac3f-9dd4ad62c392"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:53:23.015726 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:23.010443    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/330523c8-06c9-42a2-ac3f-9dd4ad62c392-istio-podinfo" pod "330523c8-06c9-42a2-ac3f-9dd4ad62c392" (UID: "330523c8-06c9-42a2-ac3f-9dd4ad62c392"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 12:53:23.015726 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:23.011481    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/330523c8-06c9-42a2-ac3f-9dd4ad62c392-workload-certs" pod "330523c8-06c9-42a2-ac3f-9dd4ad62c392" (UID: "330523c8-06c9-42a2-ac3f-9dd4ad62c392"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:53:23.015726 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:23.014172    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/330523c8-06c9-42a2-ac3f-9dd4ad62c392-istio-envoy" pod "330523c8-06c9-42a2-ac3f-9dd4ad62c392" (UID: "330523c8-06c9-42a2-ac3f-9dd4ad62c392"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:53:23.015726 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:23.014354    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/330523c8-06c9-42a2-ac3f-9dd4ad62c392-kube-api-access-sm4q5" pod "330523c8-06c9-42a2-ac3f-9dd4ad62c392" (UID: "330523c8-06c9-42a2-ac3f-9dd4ad62c392"). InnerVolumeSpecName "kube-api-access-sm4q5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:53:23.017050 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:23.016880    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/330523c8-06c9-42a2-ac3f-9dd4ad62c392-istio-token" pod "330523c8-06c9-42a2-ac3f-9dd4ad62c392" (UID: "330523c8-06c9-42a2-ac3f-9dd4ad62c392"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:53:23.017050 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:23.016950    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/330523c8-06c9-42a2-ac3f-9dd4ad62c392-workload-socket" pod "330523c8-06c9-42a2-ac3f-9dd4ad62c392" (UID: "330523c8-06c9-42a2-ac3f-9dd4ad62c392"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:53:23.025625 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:23.025535    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-dm6jq"
Oct 31 12:53:23.035206 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:23.035105    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-xmr62"
Oct 31 12:53:23.085074 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:23.085038    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-zg66r"
Oct 31 12:53:23.133012 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:23.132944    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6750d8d2-c91b-4d08-9553-2fe15afb061a-istio-token\" (UniqueName: \"kubernetes.io/projected/6750d8d2-c91b-4d08-9553-2fe15afb061a-istio-token\") pod \"6750d8d2-c91b-4d08-9553-2fe15afb061a\" (UID: \"6750d8d2-c91b-4d08-9553-2fe15afb061a\") "
Oct 31 12:53:23.133363 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:23.133314    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6750d8d2-c91b-4d08-9553-2fe15afb061a-kube-api-access-8zjq9\" (UniqueName: \"kubernetes.io/projected/6750d8d2-c91b-4d08-9553-2fe15afb061a-kube-api-access-8zjq9\") pod \"6750d8d2-c91b-4d08-9553-2fe15afb061a\" (UID: \"6750d8d2-c91b-4d08-9553-2fe15afb061a\") "
Oct 31 12:53:23.133609 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:23.133578    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6750d8d2-c91b-4d08-9553-2fe15afb061a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6750d8d2-c91b-4d08-9553-2fe15afb061a-istiod-ca-cert\") pod \"6750d8d2-c91b-4d08-9553-2fe15afb061a\" (UID: \"6750d8d2-c91b-4d08-9553-2fe15afb061a\") "
Oct 31 12:53:23.133814 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:23.133787    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e1e6c7f5-d872-4549-b950-1e751b6374ab-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1e6c7f5-d872-4549-b950-1e751b6374ab-istio-podinfo\") pod \"e1e6c7f5-d872-4549-b950-1e751b6374ab\" (UID: \"e1e6c7f5-d872-4549-b950-1e751b6374ab\") "
Oct 31 12:53:23.134093 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:23.134064    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6750d8d2-c91b-4d08-9553-2fe15afb061a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6750d8d2-c91b-4d08-9553-2fe15afb061a-workload-certs\") pod \"6750d8d2-c91b-4d08-9553-2fe15afb061a\" (UID: \"6750d8d2-c91b-4d08-9553-2fe15afb061a\") "
Oct 31 12:53:23.134247 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:23.134230    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1e6c7f5-d872-4549-b950-1e751b6374ab-istio-token\" (UniqueName: \"kubernetes.io/projected/e1e6c7f5-d872-4549-b950-1e751b6374ab-istio-token\") pod \"e1e6c7f5-d872-4549-b950-1e751b6374ab\" (UID: \"e1e6c7f5-d872-4549-b950-1e751b6374ab\") "
Oct 31 12:53:23.134376 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:23.134360    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6750d8d2-c91b-4d08-9553-2fe15afb061a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6750d8d2-c91b-4d08-9553-2fe15afb061a-istio-data\") pod \"6750d8d2-c91b-4d08-9553-2fe15afb061a\" (UID: \"6750d8d2-c91b-4d08-9553-2fe15afb061a\") "
Oct 31 12:53:23.134518 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:23.134474    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6750d8d2-c91b-4d08-9553-2fe15afb061a-istiod-ca-cert" pod "6750d8d2-c91b-4d08-9553-2fe15afb061a" (UID: "6750d8d2-c91b-4d08-9553-2fe15afb061a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:53:23.134629 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:23.134611    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1e6c7f5-d872-4549-b950-1e751b6374ab-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1e6c7f5-d872-4549-b950-1e751b6374ab-workload-certs\") pod \"e1e6c7f5-d872-4549-b950-1e751b6374ab\" (UID: \"e1e6c7f5-d872-4549-b950-1e751b6374ab\") "
Oct 31 12:53:23.134744 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:23.134721    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6750d8d2-c91b-4d08-9553-2fe15afb061a-workload-certs" pod "6750d8d2-c91b-4d08-9553-2fe15afb061a" (UID: "6750d8d2-c91b-4d08-9553-2fe15afb061a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:53:23.134868 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:23.134849    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1e6c7f5-d872-4549-b950-1e751b6374ab-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1e6c7f5-d872-4549-b950-1e751b6374ab-istiod-ca-cert\") pod \"e1e6c7f5-d872-4549-b950-1e751b6374ab\" (UID: \"e1e6c7f5-d872-4549-b950-1e751b6374ab\") "
Oct 31 12:53:23.135022 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:23.135005    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1e6c7f5-d872-4549-b950-1e751b6374ab-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1e6c7f5-d872-4549-b950-1e751b6374ab-credential-socket\") pod \"e1e6c7f5-d872-4549-b950-1e751b6374ab\" (UID: \"e1e6c7f5-d872-4549-b950-1e751b6374ab\") "
Oct 31 12:53:23.135174 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:23.135156    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1e6c7f5-d872-4549-b950-1e751b6374ab-kube-api-access-kc5gq\" (UniqueName: \"kubernetes.io/projected/e1e6c7f5-d872-4549-b950-1e751b6374ab-kube-api-access-kc5gq\") pod \"e1e6c7f5-d872-4549-b950-1e751b6374ab\" (UID: \"e1e6c7f5-d872-4549-b950-1e751b6374ab\") "
Oct 31 12:53:23.135342 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:23.135323    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6750d8d2-c91b-4d08-9553-2fe15afb061a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6750d8d2-c91b-4d08-9553-2fe15afb061a-istio-envoy\") pod \"6750d8d2-c91b-4d08-9553-2fe15afb061a\" (UID: \"6750d8d2-c91b-4d08-9553-2fe15afb061a\") "
Oct 31 12:53:23.135468 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:23.134904    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6750d8d2-c91b-4d08-9553-2fe15afb061a-istio-data" pod "6750d8d2-c91b-4d08-9553-2fe15afb061a" (UID: "6750d8d2-c91b-4d08-9553-2fe15afb061a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:53:23.136186 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:23.135042    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1e6c7f5-d872-4549-b950-1e751b6374ab-workload-certs" pod "e1e6c7f5-d872-4549-b950-1e751b6374ab" (UID: "e1e6c7f5-d872-4549-b950-1e751b6374ab"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:53:23.136186 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:23.135220    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1e6c7f5-d872-4549-b950-1e751b6374ab-istiod-ca-cert" pod "e1e6c7f5-d872-4549-b950-1e751b6374ab" (UID: "e1e6c7f5-d872-4549-b950-1e751b6374ab"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:53:23.136186 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:23.135439    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1e6c7f5-d872-4549-b950-1e751b6374ab-credential-socket" pod "e1e6c7f5-d872-4549-b950-1e751b6374ab" (UID: "e1e6c7f5-d872-4549-b950-1e751b6374ab"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:53:23.136186 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:23.135459    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1e6c7f5-d872-4549-b950-1e751b6374ab-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1e6c7f5-d872-4549-b950-1e751b6374ab-istio-data\") pod \"e1e6c7f5-d872-4549-b950-1e751b6374ab\" (UID: \"e1e6c7f5-d872-4549-b950-1e751b6374ab\") "
Oct 31 12:53:23.136186 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:23.135694    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6750d8d2-c91b-4d08-9553-2fe15afb061a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6750d8d2-c91b-4d08-9553-2fe15afb061a-workload-socket\") pod \"6750d8d2-c91b-4d08-9553-2fe15afb061a\" (UID: \"6750d8d2-c91b-4d08-9553-2fe15afb061a\") "
Oct 31 12:53:23.136186 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:23.135731    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1e6c7f5-d872-4549-b950-1e751b6374ab-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1e6c7f5-d872-4549-b950-1e751b6374ab-istio-envoy\") pod \"e1e6c7f5-d872-4549-b950-1e751b6374ab\" (UID: \"e1e6c7f5-d872-4549-b950-1e751b6374ab\") "
Oct 31 12:53:23.136186 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:23.135785    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6750d8d2-c91b-4d08-9553-2fe15afb061a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6750d8d2-c91b-4d08-9553-2fe15afb061a-credential-socket\") pod \"6750d8d2-c91b-4d08-9553-2fe15afb061a\" (UID: \"6750d8d2-c91b-4d08-9553-2fe15afb061a\") "
Oct 31 12:53:23.136186 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:23.135880    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1e6c7f5-d872-4549-b950-1e751b6374ab-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1e6c7f5-d872-4549-b950-1e751b6374ab-workload-socket\") pod \"e1e6c7f5-d872-4549-b950-1e751b6374ab\" (UID: \"e1e6c7f5-d872-4549-b950-1e751b6374ab\") "
Oct 31 12:53:23.136186 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:23.135934    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6750d8d2-c91b-4d08-9553-2fe15afb061a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6750d8d2-c91b-4d08-9553-2fe15afb061a-istio-podinfo\") pod \"6750d8d2-c91b-4d08-9553-2fe15afb061a\" (UID: \"6750d8d2-c91b-4d08-9553-2fe15afb061a\") "
Oct 31 12:53:23.136729 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:23.136709    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6750d8d2-c91b-4d08-9553-2fe15afb061a-workload-socket" pod "6750d8d2-c91b-4d08-9553-2fe15afb061a" (UID: "6750d8d2-c91b-4d08-9553-2fe15afb061a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:53:23.137201 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:23.137176    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1e6c7f5-d872-4549-b950-1e751b6374ab-istio-data" pod "e1e6c7f5-d872-4549-b950-1e751b6374ab" (UID: "e1e6c7f5-d872-4549-b950-1e751b6374ab"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:53:23.137881 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:23.137860    5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/330523c8-06c9-42a2-ac3f-9dd4ad62c392-istio-token\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:23.138015 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:23.138000    5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6750d8d2-c91b-4d08-9553-2fe15afb061a-istiod-ca-cert\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:23.138125 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:23.138112    5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/330523c8-06c9-42a2-ac3f-9dd4ad62c392-istiod-ca-cert\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:23.138221 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:23.138209    5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6750d8d2-c91b-4d08-9553-2fe15afb061a-workload-certs\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:23.138306 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:23.138293    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sm4q5\" (UniqueName: \"kubernetes.io/projected/330523c8-06c9-42a2-ac3f-9dd4ad62c392-kube-api-access-sm4q5\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:23.138388 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:23.138377    5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6750d8d2-c91b-4d08-9553-2fe15afb061a-istio-data\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:23.138468 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:23.138455    5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1e6c7f5-d872-4549-b950-1e751b6374ab-workload-certs\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:23.138591 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:23.138577    5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/330523c8-06c9-42a2-ac3f-9dd4ad62c392-istio-podinfo\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:23.138682 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:23.138669    5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1e6c7f5-d872-4549-b950-1e751b6374ab-istiod-ca-cert\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:23.138769 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:23.138757    5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1e6c7f5-d872-4549-b950-1e751b6374ab-credential-socket\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:23.138863 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:23.138851    5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1e6c7f5-d872-4549-b950-1e751b6374ab-istio-data\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:23.138955 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:23.138943    5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6750d8d2-c91b-4d08-9553-2fe15afb061a-workload-socket\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:23.139073 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:23.139057    5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/330523c8-06c9-42a2-ac3f-9dd4ad62c392-credential-socket\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:23.139897 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:23.139139    5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/330523c8-06c9-42a2-ac3f-9dd4ad62c392-workload-socket\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:23.139897 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:23.139154    5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/330523c8-06c9-42a2-ac3f-9dd4ad62c392-istio-envoy\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:23.139897 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:23.139167    5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/330523c8-06c9-42a2-ac3f-9dd4ad62c392-istio-data\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:23.139897 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:23.139180    5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/330523c8-06c9-42a2-ac3f-9dd4ad62c392-workload-certs\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:23.139897 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:23.139549    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6750d8d2-c91b-4d08-9553-2fe15afb061a-istio-envoy" pod "6750d8d2-c91b-4d08-9553-2fe15afb061a" (UID: "6750d8d2-c91b-4d08-9553-2fe15afb061a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:53:23.142400 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:23.140783    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1e6c7f5-d872-4549-b950-1e751b6374ab-workload-socket" pod "e1e6c7f5-d872-4549-b950-1e751b6374ab" (UID: "e1e6c7f5-d872-4549-b950-1e751b6374ab"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:53:23.142400 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:23.140816    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6750d8d2-c91b-4d08-9553-2fe15afb061a-credential-socket" pod "6750d8d2-c91b-4d08-9553-2fe15afb061a" (UID: "6750d8d2-c91b-4d08-9553-2fe15afb061a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:53:23.142400 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:23.141763    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6750d8d2-c91b-4d08-9553-2fe15afb061a-kube-api-access-8zjq9" pod "6750d8d2-c91b-4d08-9553-2fe15afb061a" (UID: "6750d8d2-c91b-4d08-9553-2fe15afb061a"). InnerVolumeSpecName "kube-api-access-8zjq9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:53:23.142400 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:23.141902    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1e6c7f5-d872-4549-b950-1e751b6374ab-istio-token" pod "e1e6c7f5-d872-4549-b950-1e751b6374ab" (UID: "e1e6c7f5-d872-4549-b950-1e751b6374ab"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:53:23.148419 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:23.148356    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1e6c7f5-d872-4549-b950-1e751b6374ab-istio-envoy" pod "e1e6c7f5-d872-4549-b950-1e751b6374ab" (UID: "e1e6c7f5-d872-4549-b950-1e751b6374ab"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:53:23.148620 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:23.148533    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e1e6c7f5-d872-4549-b950-1e751b6374ab-istio-podinfo" pod "e1e6c7f5-d872-4549-b950-1e751b6374ab" (UID: "e1e6c7f5-d872-4549-b950-1e751b6374ab"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 12:53:23.149186 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:23.148676    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6750d8d2-c91b-4d08-9553-2fe15afb061a-istio-token" pod "6750d8d2-c91b-4d08-9553-2fe15afb061a" (UID: "6750d8d2-c91b-4d08-9553-2fe15afb061a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:53:23.149186 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:23.148750    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6750d8d2-c91b-4d08-9553-2fe15afb061a-istio-podinfo" pod "6750d8d2-c91b-4d08-9553-2fe15afb061a" (UID: "6750d8d2-c91b-4d08-9553-2fe15afb061a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 12:53:23.149186 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:23.149016    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1e6c7f5-d872-4549-b950-1e751b6374ab-kube-api-access-kc5gq" pod "e1e6c7f5-d872-4549-b950-1e751b6374ab" (UID: "e1e6c7f5-d872-4549-b950-1e751b6374ab"). InnerVolumeSpecName "kube-api-access-kc5gq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:53:23.244901 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:23.244864    5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6750d8d2-c91b-4d08-9553-2fe15afb061a-istio-token\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:23.244901 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:23.244906    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zjq9\" (UniqueName: \"kubernetes.io/projected/6750d8d2-c91b-4d08-9553-2fe15afb061a-kube-api-access-8zjq9\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:23.245323 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:23.244923    5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1e6c7f5-d872-4549-b950-1e751b6374ab-istio-podinfo\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:23.245323 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:23.244937    5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1e6c7f5-d872-4549-b950-1e751b6374ab-istio-token\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:23.245323 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:23.244952    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kc5gq\" (UniqueName: \"kubernetes.io/projected/e1e6c7f5-d872-4549-b950-1e751b6374ab-kube-api-access-kc5gq\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:23.245323 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:23.244966    5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6750d8d2-c91b-4d08-9553-2fe15afb061a-istio-envoy\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:23.245323 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:23.244979    5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1e6c7f5-d872-4549-b950-1e751b6374ab-istio-envoy\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:23.245323 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:23.244993    5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6750d8d2-c91b-4d08-9553-2fe15afb061a-credential-socket\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:23.245323 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:23.245006    5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1e6c7f5-d872-4549-b950-1e751b6374ab-workload-socket\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:23.245323 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:23.245021    5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6750d8d2-c91b-4d08-9553-2fe15afb061a-istio-podinfo\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:23.250396 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:53:23.250363    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfca0927c_616b_4766_8cb2_cae87d082f01.slice/crio-d16cd5d1eb1db0c419ca58cbe1bca2f976a791b45c255b1e1b4ac0da9385b75b WatchSource:0}: Error finding container d16cd5d1eb1db0c419ca58cbe1bca2f976a791b45c255b1e1b4ac0da9385b75b: Status 404 returned error can't find the container with id d16cd5d1eb1db0c419ca58cbe1bca2f976a791b45c255b1e1b4ac0da9385b75b
Oct 31 12:53:23.253720 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:23.253697    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1a7d0e-bsjb6"]
Oct 31 12:53:23.589778 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:23.589738    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1a7d0e-bsjb6" event={"ID":"fca0927c-616b-4766-8cb2-cae87d082f01","Type":"ContainerStarted","Data":"d16cd5d1eb1db0c419ca58cbe1bca2f976a791b45c255b1e1b4ac0da9385b75b"}
Oct 31 12:53:23.591592 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:23.591560    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wnh97" event={"ID":"330523c8-06c9-42a2-ac3f-9dd4ad62c392","Type":"ContainerDied","Data":"6b069694c4651774f5a12305bb7b29c2a2d5e20b54b370ae4d8fe13bc2ca6ad4"}
Oct 31 12:53:23.591750 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:23.591626    5060 scope.go:122] "RemoveContainer" containerID="d190e7520eebe52b0c749ad77fb372857ff3f35218eb3dc8fea952d63280f3d9"
Oct 31 12:53:23.591750 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:23.591574    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wnh97"
Oct 31 12:53:23.593965 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:23.593658    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-dm6jq" event={"ID":"6750d8d2-c91b-4d08-9553-2fe15afb061a","Type":"ContainerDied","Data":"61dcff1427466a44995a9e918eb77dce6ce41bbb1a96ae00c2ddc9246fbd0119"}
Oct 31 12:53:23.593965 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:23.593683    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-dm6jq"
Oct 31 12:53:23.596805 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:23.596763    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-xmr62" event={"ID":"e1e6c7f5-d872-4549-b950-1e751b6374ab","Type":"ContainerDied","Data":"c488f4c2abb95ac89cdde2cb4c444ba6f592255dcea31e1d4eb8f9cebcf29959"}
Oct 31 12:53:23.596968 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:23.596859    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-xmr62"
Oct 31 12:53:23.600584 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:23.600428    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d176a5-57f6cd6df5-tzc5c"
Oct 31 12:53:23.609366 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:23.609344    5060 scope.go:122] "RemoveContainer" containerID="4a4fa34a0917cc911ea4ce7ecf3411bc3464d8a28cbdc5a67b9fc4e5b2ac6fa9"
Oct 31 12:53:23.626865 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:23.626836    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wnh97"]
Oct 31 12:53:23.630030 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:23.629918    5060 scope.go:122] "RemoveContainer" containerID="15eb93212f666238582935bcfb7fb66c4776d563bb22273bc4eb80596e90659b"
Oct 31 12:53:23.633658 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:23.633634    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-wnh97"]
Oct 31 12:53:23.681129 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:23.681094    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-xmr62"]
Oct 31 12:53:23.693473 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:23.693431    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-xmr62"]
Oct 31 12:53:23.708445 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:23.708414    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-dm6jq"]
Oct 31 12:53:23.714899 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:23.714871    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-dm6jq"]
Oct 31 12:53:23.815273 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:23.815238    5060 scope.go:122] "RemoveContainer" containerID="4196ac853c00bbb370884f073952858bd6073a099824e57d1ba818f0a19289a0"
Oct 31 12:53:23.828858 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:23.828833    5060 scope.go:122] "RemoveContainer" containerID="9f840cc0629fb7cbbeecfa77fe1c24e19568541bbfac0a40ad8fec6cbe4b7aac"
Oct 31 12:53:23.840963 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:23.840938    5060 scope.go:122] "RemoveContainer" containerID="b82573bb3be8c333d80ccab70db9a2931ba3f9f3baa15306106cb82bcc215295"
Oct 31 12:53:23.853223 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:23.853029    5060 scope.go:122] "RemoveContainer" containerID="acb276220fae73031ba6385f851e7954286be7110291008ead66dfdf1cfcb663"
Oct 31 12:53:23.933110 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:23.933082    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e98df8-wbs75"]
Oct 31 12:53:23.934546 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:23.934517    5060 cpu_manager.go:414] "containerMap: removing container" podUID="6750d8d2-c91b-4d08-9553-2fe15afb061a" containerName="istio-proxy"
Oct 31 12:53:23.934546 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:23.934545    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6750d8d2-c91b-4d08-9553-2fe15afb061a" containerName="istio-proxy"
Oct 31 12:53:23.934723 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:23.934591    5060 cpu_manager.go:414] "containerMap: removing container" podUID="e1e6c7f5-d872-4549-b950-1e751b6374ab" containerName="istio-proxy"
Oct 31 12:53:23.934723 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:23.934600    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1e6c7f5-d872-4549-b950-1e751b6374ab" containerName="istio-proxy"
Oct 31 12:53:23.934723 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:23.934619    5060 cpu_manager.go:414] "containerMap: removing container" podUID="330523c8-06c9-42a2-ac3f-9dd4ad62c392" containerName="istio-proxy"
Oct 31 12:53:23.934723 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:23.934628    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="330523c8-06c9-42a2-ac3f-9dd4ad62c392" containerName="istio-proxy"
Oct 31 12:53:23.934723 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:23.934651    5060 cpu_manager.go:414] "containerMap: removing container" podUID="330523c8-06c9-42a2-ac3f-9dd4ad62c392" containerName="istio-proxy"
Oct 31 12:53:23.934723 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:23.934658    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="330523c8-06c9-42a2-ac3f-9dd4ad62c392" containerName="istio-proxy"
Oct 31 12:53:23.935041 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:23.934888    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="330523c8-06c9-42a2-ac3f-9dd4ad62c392" containerName="istio-proxy"
Oct 31 12:53:23.935041 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:23.934907    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="330523c8-06c9-42a2-ac3f-9dd4ad62c392" containerName="istio-proxy"
Oct 31 12:53:23.935041 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:23.934932    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="6750d8d2-c91b-4d08-9553-2fe15afb061a" containerName="istio-proxy"
Oct 31 12:53:23.935041 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:23.934948    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1e6c7f5-d872-4549-b950-1e751b6374ab" containerName="istio-proxy"
Oct 31 12:53:23.941408 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:23.941386    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e98df8-wbs75"
Oct 31 12:53:23.959034 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:23.958897    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e98df8-wbs75"]
Oct 31 12:53:24.292931 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:24.292884    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6ndb\" (UniqueName: \"kubernetes.io/projected/53231544-cdae-4031-a635-1260aab4def3-kube-api-access-p6ndb\") pod \"maas-api-revoke-keys-e2e-inf-a-e98df8-wbs75\" (UID: \"53231544-cdae-4031-a635-1260aab4def3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e98df8-wbs75"
Oct 31 12:53:24.293673 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:24.293002    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53231544-cdae-4031-a635-1260aab4def3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-e98df8-wbs75\" (UID: \"53231544-cdae-4031-a635-1260aab4def3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e98df8-wbs75"
Oct 31 12:53:24.293673 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:24.293190    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53231544-cdae-4031-a635-1260aab4def3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-e98df8-wbs75\" (UID: \"53231544-cdae-4031-a635-1260aab4def3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e98df8-wbs75"
Oct 31 12:53:24.399085 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:24.399045    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53231544-cdae-4031-a635-1260aab4def3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-e98df8-wbs75\" (UID: \"53231544-cdae-4031-a635-1260aab4def3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e98df8-wbs75"
Oct 31 12:53:24.399894 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:24.399864    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6ndb\" (UniqueName: \"kubernetes.io/projected/53231544-cdae-4031-a635-1260aab4def3-kube-api-access-p6ndb\") pod \"maas-api-revoke-keys-e2e-inf-a-e98df8-wbs75\" (UID: \"53231544-cdae-4031-a635-1260aab4def3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e98df8-wbs75"
Oct 31 12:53:24.400128 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:24.400113    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53231544-cdae-4031-a635-1260aab4def3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-e98df8-wbs75\" (UID: \"53231544-cdae-4031-a635-1260aab4def3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e98df8-wbs75"
Oct 31 12:53:24.400974 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:24.400948    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53231544-cdae-4031-a635-1260aab4def3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-e98df8-wbs75\" (UID: \"53231544-cdae-4031-a635-1260aab4def3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e98df8-wbs75"
Oct 31 12:53:24.409859 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:24.409830    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53231544-cdae-4031-a635-1260aab4def3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-e98df8-wbs75\" (UID: \"53231544-cdae-4031-a635-1260aab4def3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e98df8-wbs75"
Oct 31 12:53:24.425181 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:24.425146    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6ndb\" (UniqueName: \"kubernetes.io/projected/53231544-cdae-4031-a635-1260aab4def3-kube-api-access-p6ndb\") pod \"maas-api-revoke-keys-e2e-inf-a-e98df8-wbs75\" (UID: \"53231544-cdae-4031-a635-1260aab4def3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e98df8-wbs75"
Oct 31 12:53:24.519107 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:24.489515    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e98df8-wbs75"
Oct 31 12:53:24.742644 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:24.742601    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1a7d0e-bsjb6" event={"ID":"fca0927c-616b-4766-8cb2-cae87d082f01","Type":"ContainerStarted","Data":"5b3e1587d6c3e6ff23958a705b523269aaf7f555ce70a9ce9ad5343ce3d0c431"}
Oct 31 12:53:24.769355 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:24.769289    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1a7d0e-bsjb6" podStartSLOduration=5.76927131 podStartE2EDuration="5.76927131s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:53:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:53:24.765319215 +0000 UTC m=+1567.868386926" watchObservedRunningTime="2026-10-31 12:53:24.76927131 +0000 UTC m=+1567.872339023"
Oct 31 12:53:25.055573 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:25.055535    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="330523c8-06c9-42a2-ac3f-9dd4ad62c392" path="/var/lib/kubelet/pods/330523c8-06c9-42a2-ac3f-9dd4ad62c392/volumes"
Oct 31 12:53:25.058540 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:25.056285    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6750d8d2-c91b-4d08-9553-2fe15afb061a" path="/var/lib/kubelet/pods/6750d8d2-c91b-4d08-9553-2fe15afb061a/volumes"
Oct 31 12:53:25.058540 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:25.057195    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1e6c7f5-d872-4549-b950-1e751b6374ab" path="/var/lib/kubelet/pods/e1e6c7f5-d872-4549-b950-1e751b6374ab/volumes"
Oct 31 12:53:25.058540 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:25.058457    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e98df8-wbs75"]
Oct 31 12:53:25.058767 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:53:25.058705    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53231544_cdae_4031_a635_1260aab4def3.slice/crio-63e70957d777e94acc067a72ad51f6fe7d81355fa65ef603dccf4aa2388a1b93 WatchSource:0}: Error finding container 63e70957d777e94acc067a72ad51f6fe7d81355fa65ef603dccf4aa2388a1b93: Status 404 returned error can't find the container with id 63e70957d777e94acc067a72ad51f6fe7d81355fa65ef603dccf4aa2388a1b93
Oct 31 12:53:25.104623 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:25.104588    5060 patch_prober.go:29] interesting pod/e2e-inf-a-e43d8b-openshift-default-5578dcfc7f-5f9ss container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 12:53:25.104776 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:25.104645    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-e43d8b-openshift-default-5578dcfc7f-5f9ss" podUID="a0ba8ab3-65d2-4132-9896-80f1fb46b5c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:53:25.761547 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:25.761414    5060 generic.go:356] "Generic (PLEG): container finished" podID="fca0927c-616b-4766-8cb2-cae87d082f01" containerID="5b3e1587d6c3e6ff23958a705b523269aaf7f555ce70a9ce9ad5343ce3d0c431" exitCode=0
Oct 31 12:53:25.761976 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:25.761577    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1a7d0e-bsjb6" event={"ID":"fca0927c-616b-4766-8cb2-cae87d082f01","Type":"ContainerDied","Data":"5b3e1587d6c3e6ff23958a705b523269aaf7f555ce70a9ce9ad5343ce3d0c431"}
Oct 31 12:53:25.763729 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:25.763699    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e98df8-wbs75" event={"ID":"53231544-cdae-4031-a635-1260aab4def3","Type":"ContainerStarted","Data":"e3f476e9efd100c5f7f824ca329b93c37eb01200542f98d9f9907991b4aa8045"}
Oct 31 12:53:25.763880 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:25.763748    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e98df8-wbs75" event={"ID":"53231544-cdae-4031-a635-1260aab4def3","Type":"ContainerStarted","Data":"63e70957d777e94acc067a72ad51f6fe7d81355fa65ef603dccf4aa2388a1b93"}
Oct 31 12:53:25.813980 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:25.813915    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e98df8-wbs75" podStartSLOduration=2.813895379 podStartE2EDuration="2.813895379s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:53:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:53:25.808129518 +0000 UTC m=+1568.911197230" watchObservedRunningTime="2026-10-31 12:53:25.813895379 +0000 UTC m=+1568.916963091"
Oct 31 12:53:26.777042 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:26.777004    5060 generic.go:356] "Generic (PLEG): container finished" podID="53231544-cdae-4031-a635-1260aab4def3" containerID="e3f476e9efd100c5f7f824ca329b93c37eb01200542f98d9f9907991b4aa8045" exitCode=0
Oct 31 12:53:26.777440 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:26.777081    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e98df8-wbs75" event={"ID":"53231544-cdae-4031-a635-1260aab4def3","Type":"ContainerDied","Data":"e3f476e9efd100c5f7f824ca329b93c37eb01200542f98d9f9907991b4aa8045"}
Oct 31 12:53:27.049077 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:27.049006    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1a7d0e-bsjb6"
Oct 31 12:53:27.127366 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:27.127322    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fca0927c-616b-4766-8cb2-cae87d082f01-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fca0927c-616b-4766-8cb2-cae87d082f01-maas-api-openshift-ca\") pod \"fca0927c-616b-4766-8cb2-cae87d082f01\" (UID: \"fca0927c-616b-4766-8cb2-cae87d082f01\") "
Oct 31 12:53:27.137057 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:27.127441    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fca0927c-616b-4766-8cb2-cae87d082f01-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fca0927c-616b-4766-8cb2-cae87d082f01-maas-api-xks-ca\") pod \"fca0927c-616b-4766-8cb2-cae87d082f01\" (UID: \"fca0927c-616b-4766-8cb2-cae87d082f01\") "
Oct 31 12:53:27.137057 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:27.127739    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fca0927c-616b-4766-8cb2-cae87d082f01-maas-api-openshift-ca" pod "fca0927c-616b-4766-8cb2-cae87d082f01" (UID: "fca0927c-616b-4766-8cb2-cae87d082f01"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:53:27.137057 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:27.128178    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fca0927c-616b-4766-8cb2-cae87d082f01-kube-api-access-d9q7l\" (UniqueName: \"kubernetes.io/projected/fca0927c-616b-4766-8cb2-cae87d082f01-kube-api-access-d9q7l\") pod \"fca0927c-616b-4766-8cb2-cae87d082f01\" (UID: \"fca0927c-616b-4766-8cb2-cae87d082f01\") "
Oct 31 12:53:27.137057 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:27.131168    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fca0927c-616b-4766-8cb2-cae87d082f01-maas-api-openshift-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:27.152262 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:27.152223    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-9b62a0-8445958dbb-99wdf"]
Oct 31 12:53:27.154379 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:27.154354    5060 cpu_manager.go:414] "containerMap: removing container" podUID="fca0927c-616b-4766-8cb2-cae87d082f01" containerName="revoke-keys"
Oct 31 12:53:27.154379 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:27.154378    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fca0927c-616b-4766-8cb2-cae87d082f01" containerName="revoke-keys"
Oct 31 12:53:27.160732 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:27.154724    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="fca0927c-616b-4766-8cb2-cae87d082f01" containerName="revoke-keys"
Oct 31 12:53:27.160732 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:27.157851    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fca0927c-616b-4766-8cb2-cae87d082f01-maas-api-xks-ca" pod "fca0927c-616b-4766-8cb2-cae87d082f01" (UID: "fca0927c-616b-4766-8cb2-cae87d082f01"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:53:27.160732 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:27.159729    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fca0927c-616b-4766-8cb2-cae87d082f01-kube-api-access-d9q7l" pod "fca0927c-616b-4766-8cb2-cae87d082f01" (UID: "fca0927c-616b-4766-8cb2-cae87d082f01"). InnerVolumeSpecName "kube-api-access-d9q7l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:53:27.182834 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:27.178884    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9b62a0-8445958dbb-99wdf"
Oct 31 12:53:27.189950 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:27.189364    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-9b62a0-serving-cert\""
Oct 31 12:53:27.193296 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:27.193275    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-9b62a0-8445958dbb-99wdf"]
Oct 31 12:53:27.233449 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:27.233015    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fca0927c-616b-4766-8cb2-cae87d082f01-maas-api-xks-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:27.233449 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:27.233049    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9q7l\" (UniqueName: \"kubernetes.io/projected/fca0927c-616b-4766-8cb2-cae87d082f01-kube-api-access-d9q7l\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:27.335279 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:27.334698    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9aa65f61-54d4-4e45-8f33-51ea2fe5bc56-metrics-certs\") pod \"maas-api-e2e-inf-a-9b62a0-8445958dbb-99wdf\" (UID: \"9aa65f61-54d4-4e45-8f33-51ea2fe5bc56\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9b62a0-8445958dbb-99wdf"
Oct 31 12:53:27.335279 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:27.334761    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9aa65f61-54d4-4e45-8f33-51ea2fe5bc56-openshift-service-ca\") pod \"maas-api-e2e-inf-a-9b62a0-8445958dbb-99wdf\" (UID: \"9aa65f61-54d4-4e45-8f33-51ea2fe5bc56\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9b62a0-8445958dbb-99wdf"
Oct 31 12:53:27.335279 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:27.334835    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9aa65f61-54d4-4e45-8f33-51ea2fe5bc56-maas-api-tls\") pod \"maas-api-e2e-inf-a-9b62a0-8445958dbb-99wdf\" (UID: \"9aa65f61-54d4-4e45-8f33-51ea2fe5bc56\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9b62a0-8445958dbb-99wdf"
Oct 31 12:53:27.335279 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:27.334991    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8vbn\" (UniqueName: \"kubernetes.io/projected/9aa65f61-54d4-4e45-8f33-51ea2fe5bc56-kube-api-access-x8vbn\") pod \"maas-api-e2e-inf-a-9b62a0-8445958dbb-99wdf\" (UID: \"9aa65f61-54d4-4e45-8f33-51ea2fe5bc56\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9b62a0-8445958dbb-99wdf"
Oct 31 12:53:27.436992 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:27.436951    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9aa65f61-54d4-4e45-8f33-51ea2fe5bc56-metrics-certs\") pod \"maas-api-e2e-inf-a-9b62a0-8445958dbb-99wdf\" (UID: \"9aa65f61-54d4-4e45-8f33-51ea2fe5bc56\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9b62a0-8445958dbb-99wdf"
Oct 31 12:53:27.437195 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:27.437013    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9aa65f61-54d4-4e45-8f33-51ea2fe5bc56-openshift-service-ca\") pod \"maas-api-e2e-inf-a-9b62a0-8445958dbb-99wdf\" (UID: \"9aa65f61-54d4-4e45-8f33-51ea2fe5bc56\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9b62a0-8445958dbb-99wdf"
Oct 31 12:53:27.437195 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:27.437079    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9aa65f61-54d4-4e45-8f33-51ea2fe5bc56-maas-api-tls\") pod \"maas-api-e2e-inf-a-9b62a0-8445958dbb-99wdf\" (UID: \"9aa65f61-54d4-4e45-8f33-51ea2fe5bc56\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9b62a0-8445958dbb-99wdf"
Oct 31 12:53:27.437273 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:27.437217    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8vbn\" (UniqueName: \"kubernetes.io/projected/9aa65f61-54d4-4e45-8f33-51ea2fe5bc56-kube-api-access-x8vbn\") pod \"maas-api-e2e-inf-a-9b62a0-8445958dbb-99wdf\" (UID: \"9aa65f61-54d4-4e45-8f33-51ea2fe5bc56\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9b62a0-8445958dbb-99wdf"
Oct 31 12:53:27.439194 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:27.438817    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9aa65f61-54d4-4e45-8f33-51ea2fe5bc56-openshift-service-ca\") pod \"maas-api-e2e-inf-a-9b62a0-8445958dbb-99wdf\" (UID: \"9aa65f61-54d4-4e45-8f33-51ea2fe5bc56\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9b62a0-8445958dbb-99wdf"
Oct 31 12:53:27.439194 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:53:27.438933    5060 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-9b62a0-serving-cert: secret "maas-api-e2e-inf-a-9b62a0-serving-cert" not found
Oct 31 12:53:27.448274 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:27.448246    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9aa65f61-54d4-4e45-8f33-51ea2fe5bc56-metrics-certs\") pod \"maas-api-e2e-inf-a-9b62a0-8445958dbb-99wdf\" (UID: \"9aa65f61-54d4-4e45-8f33-51ea2fe5bc56\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9b62a0-8445958dbb-99wdf"
Oct 31 12:53:27.456260 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:27.456124    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8vbn\" (UniqueName: \"kubernetes.io/projected/9aa65f61-54d4-4e45-8f33-51ea2fe5bc56-kube-api-access-x8vbn\") pod \"maas-api-e2e-inf-a-9b62a0-8445958dbb-99wdf\" (UID: \"9aa65f61-54d4-4e45-8f33-51ea2fe5bc56\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9b62a0-8445958dbb-99wdf"
Oct 31 12:53:27.472446 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:53:27.472417    5060 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9aa65f61-54d4-4e45-8f33-51ea2fe5bc56-maas-api-tls podName:9aa65f61-54d4-4e45-8f33-51ea2fe5bc56 nodeName:}" failed. No retries permitted until 2026-10-31 12:53:27.962751847 +0000 UTC m=+1571.065819554 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9aa65f61-54d4-4e45-8f33-51ea2fe5bc56-maas-api-tls") pod "maas-api-e2e-inf-a-9b62a0-8445958dbb-99wdf" (UID: "9aa65f61-54d4-4e45-8f33-51ea2fe5bc56") : secret "maas-api-e2e-inf-a-9b62a0-serving-cert" not found
Oct 31 12:53:27.989957 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:27.989833    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9aa65f61-54d4-4e45-8f33-51ea2fe5bc56-maas-api-tls\") pod \"maas-api-e2e-inf-a-9b62a0-8445958dbb-99wdf\" (UID: \"9aa65f61-54d4-4e45-8f33-51ea2fe5bc56\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9b62a0-8445958dbb-99wdf"
Oct 31 12:53:28.017472 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:28.016851    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1a7d0e-bsjb6"
Oct 31 12:53:28.017785 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:28.017694    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1a7d0e-bsjb6" event={"ID":"fca0927c-616b-4766-8cb2-cae87d082f01","Type":"ContainerDied","Data":"d16cd5d1eb1db0c419ca58cbe1bca2f976a791b45c255b1e1b4ac0da9385b75b"}
Oct 31 12:53:28.017785 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:28.017753    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d16cd5d1eb1db0c419ca58cbe1bca2f976a791b45c255b1e1b4ac0da9385b75b"
Oct 31 12:53:28.083764 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:28.083672    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9aa65f61-54d4-4e45-8f33-51ea2fe5bc56-maas-api-tls\") pod \"maas-api-e2e-inf-a-9b62a0-8445958dbb-99wdf\" (UID: \"9aa65f61-54d4-4e45-8f33-51ea2fe5bc56\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9b62a0-8445958dbb-99wdf"
Oct 31 12:53:28.116419 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:28.115812    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9b62a0-8445958dbb-99wdf"
Oct 31 12:53:28.712722 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:28.712687    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-ds5zz"
Oct 31 12:53:28.742718 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:28.741992    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0070ea70-06e9-4733-8d11-ecb4cb993ff7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0070ea70-06e9-4733-8d11-ecb4cb993ff7-istio-data\") pod \"0070ea70-06e9-4733-8d11-ecb4cb993ff7\" (UID: \"0070ea70-06e9-4733-8d11-ecb4cb993ff7\") "
Oct 31 12:53:28.742718 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:28.742090    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0070ea70-06e9-4733-8d11-ecb4cb993ff7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0070ea70-06e9-4733-8d11-ecb4cb993ff7-credential-socket\") pod \"0070ea70-06e9-4733-8d11-ecb4cb993ff7\" (UID: \"0070ea70-06e9-4733-8d11-ecb4cb993ff7\") "
Oct 31 12:53:28.742718 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:28.742132    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0070ea70-06e9-4733-8d11-ecb4cb993ff7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0070ea70-06e9-4733-8d11-ecb4cb993ff7-istio-podinfo\") pod \"0070ea70-06e9-4733-8d11-ecb4cb993ff7\" (UID: \"0070ea70-06e9-4733-8d11-ecb4cb993ff7\") "
Oct 31 12:53:28.742718 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:28.742222    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0070ea70-06e9-4733-8d11-ecb4cb993ff7-kube-api-access-w7l89\" (UniqueName: \"kubernetes.io/projected/0070ea70-06e9-4733-8d11-ecb4cb993ff7-kube-api-access-w7l89\") pod \"0070ea70-06e9-4733-8d11-ecb4cb993ff7\" (UID: \"0070ea70-06e9-4733-8d11-ecb4cb993ff7\") "
Oct 31 12:53:28.742718 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:28.742337    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0070ea70-06e9-4733-8d11-ecb4cb993ff7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0070ea70-06e9-4733-8d11-ecb4cb993ff7-istiod-ca-cert\") pod \"0070ea70-06e9-4733-8d11-ecb4cb993ff7\" (UID: \"0070ea70-06e9-4733-8d11-ecb4cb993ff7\") "
Oct 31 12:53:28.743329 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:28.743279    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0070ea70-06e9-4733-8d11-ecb4cb993ff7-credential-socket" pod "0070ea70-06e9-4733-8d11-ecb4cb993ff7" (UID: "0070ea70-06e9-4733-8d11-ecb4cb993ff7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:53:28.744569 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:28.743509    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0070ea70-06e9-4733-8d11-ecb4cb993ff7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0070ea70-06e9-4733-8d11-ecb4cb993ff7-istio-envoy\") pod \"0070ea70-06e9-4733-8d11-ecb4cb993ff7\" (UID: \"0070ea70-06e9-4733-8d11-ecb4cb993ff7\") "
Oct 31 12:53:28.744569 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:28.743614    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0070ea70-06e9-4733-8d11-ecb4cb993ff7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0070ea70-06e9-4733-8d11-ecb4cb993ff7-workload-socket\") pod \"0070ea70-06e9-4733-8d11-ecb4cb993ff7\" (UID: \"0070ea70-06e9-4733-8d11-ecb4cb993ff7\") "
Oct 31 12:53:28.744569 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:28.743708    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0070ea70-06e9-4733-8d11-ecb4cb993ff7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0070ea70-06e9-4733-8d11-ecb4cb993ff7-workload-certs\") pod \"0070ea70-06e9-4733-8d11-ecb4cb993ff7\" (UID: \"0070ea70-06e9-4733-8d11-ecb4cb993ff7\") "
Oct 31 12:53:28.744569 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:28.743791    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0070ea70-06e9-4733-8d11-ecb4cb993ff7-istio-token\" (UniqueName: \"kubernetes.io/projected/0070ea70-06e9-4733-8d11-ecb4cb993ff7-istio-token\") pod \"0070ea70-06e9-4733-8d11-ecb4cb993ff7\" (UID: \"0070ea70-06e9-4733-8d11-ecb4cb993ff7\") "
Oct 31 12:53:28.745274 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:28.745211    5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0070ea70-06e9-4733-8d11-ecb4cb993ff7-credential-socket\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:28.748604 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:28.747648    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0070ea70-06e9-4733-8d11-ecb4cb993ff7-istio-data" pod "0070ea70-06e9-4733-8d11-ecb4cb993ff7" (UID: "0070ea70-06e9-4733-8d11-ecb4cb993ff7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:53:28.748604 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:28.747886    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0070ea70-06e9-4733-8d11-ecb4cb993ff7-workload-certs" pod "0070ea70-06e9-4733-8d11-ecb4cb993ff7" (UID: "0070ea70-06e9-4733-8d11-ecb4cb993ff7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:53:28.748604 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:28.748568    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0070ea70-06e9-4733-8d11-ecb4cb993ff7-istiod-ca-cert" pod "0070ea70-06e9-4733-8d11-ecb4cb993ff7" (UID: "0070ea70-06e9-4733-8d11-ecb4cb993ff7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:53:28.749316 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:28.749270    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0070ea70-06e9-4733-8d11-ecb4cb993ff7-workload-socket" pod "0070ea70-06e9-4733-8d11-ecb4cb993ff7" (UID: "0070ea70-06e9-4733-8d11-ecb4cb993ff7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:53:28.752707 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:28.752655    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0070ea70-06e9-4733-8d11-ecb4cb993ff7-istio-podinfo" pod "0070ea70-06e9-4733-8d11-ecb4cb993ff7" (UID: "0070ea70-06e9-4733-8d11-ecb4cb993ff7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 12:53:28.764045 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:28.763514    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0070ea70-06e9-4733-8d11-ecb4cb993ff7-kube-api-access-w7l89" pod "0070ea70-06e9-4733-8d11-ecb4cb993ff7" (UID: "0070ea70-06e9-4733-8d11-ecb4cb993ff7"). InnerVolumeSpecName "kube-api-access-w7l89". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:53:28.764045 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:28.763604    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0070ea70-06e9-4733-8d11-ecb4cb993ff7-istio-token" pod "0070ea70-06e9-4733-8d11-ecb4cb993ff7" (UID: "0070ea70-06e9-4733-8d11-ecb4cb993ff7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:53:28.764045 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:28.763658    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0070ea70-06e9-4733-8d11-ecb4cb993ff7-istio-envoy" pod "0070ea70-06e9-4733-8d11-ecb4cb993ff7" (UID: "0070ea70-06e9-4733-8d11-ecb4cb993ff7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:53:28.772712 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:28.772338    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e98df8-wbs75"
Oct 31 12:53:28.923285 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:28.921875    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/53231544-cdae-4031-a635-1260aab4def3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53231544-cdae-4031-a635-1260aab4def3-maas-api-xks-ca\") pod \"53231544-cdae-4031-a635-1260aab4def3\" (UID: \"53231544-cdae-4031-a635-1260aab4def3\") "
Oct 31 12:53:28.923285 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:28.921997    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53231544-cdae-4031-a635-1260aab4def3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53231544-cdae-4031-a635-1260aab4def3-maas-api-openshift-ca\") pod \"53231544-cdae-4031-a635-1260aab4def3\" (UID: \"53231544-cdae-4031-a635-1260aab4def3\") "
Oct 31 12:53:28.923285 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:28.922861    5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0070ea70-06e9-4733-8d11-ecb4cb993ff7-istio-data\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:28.923285 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:28.922892    5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0070ea70-06e9-4733-8d11-ecb4cb993ff7-istio-podinfo\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:28.923285 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:28.922908    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7l89\" (UniqueName: \"kubernetes.io/projected/0070ea70-06e9-4733-8d11-ecb4cb993ff7-kube-api-access-w7l89\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:28.923285 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:28.922925    5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0070ea70-06e9-4733-8d11-ecb4cb993ff7-istiod-ca-cert\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:28.923285 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:28.922940    5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0070ea70-06e9-4733-8d11-ecb4cb993ff7-istio-envoy\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:28.923285 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:28.922955    5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0070ea70-06e9-4733-8d11-ecb4cb993ff7-workload-socket\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:28.923285 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:28.922969    5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0070ea70-06e9-4733-8d11-ecb4cb993ff7-workload-certs\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:28.923285 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:28.922983    5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0070ea70-06e9-4733-8d11-ecb4cb993ff7-istio-token\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:28.931701 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:28.931652    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53231544-cdae-4031-a635-1260aab4def3-maas-api-openshift-ca" pod "53231544-cdae-4031-a635-1260aab4def3" (UID: "53231544-cdae-4031-a635-1260aab4def3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:53:28.939480 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:28.939426    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53231544-cdae-4031-a635-1260aab4def3-maas-api-xks-ca" pod "53231544-cdae-4031-a635-1260aab4def3" (UID: "53231544-cdae-4031-a635-1260aab4def3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:53:28.978504 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:28.978108    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-vxzjg"
Oct 31 12:53:29.024944 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:29.024873    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20fa8b33-c9e6-4fb0-9f3c-c8254dece2f6-istio-data" pod "20fa8b33-c9e6-4fb0-9f3c-c8254dece2f6" (UID: "20fa8b33-c9e6-4fb0-9f3c-c8254dece2f6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:53:29.025666 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:29.025617    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20fa8b33-c9e6-4fb0-9f3c-c8254dece2f6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20fa8b33-c9e6-4fb0-9f3c-c8254dece2f6-istio-data\") pod \"20fa8b33-c9e6-4fb0-9f3c-c8254dece2f6\" (UID: \"20fa8b33-c9e6-4fb0-9f3c-c8254dece2f6\") "
Oct 31 12:53:29.027406 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:29.027368    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20fa8b33-c9e6-4fb0-9f3c-c8254dece2f6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20fa8b33-c9e6-4fb0-9f3c-c8254dece2f6-istio-envoy\") pod \"20fa8b33-c9e6-4fb0-9f3c-c8254dece2f6\" (UID: \"20fa8b33-c9e6-4fb0-9f3c-c8254dece2f6\") "
Oct 31 12:53:29.028588 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:29.028218    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20fa8b33-c9e6-4fb0-9f3c-c8254dece2f6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20fa8b33-c9e6-4fb0-9f3c-c8254dece2f6-credential-socket\") pod \"20fa8b33-c9e6-4fb0-9f3c-c8254dece2f6\" (UID: \"20fa8b33-c9e6-4fb0-9f3c-c8254dece2f6\") "
Oct 31 12:53:29.028588 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:29.028533    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20fa8b33-c9e6-4fb0-9f3c-c8254dece2f6-credential-socket" pod "20fa8b33-c9e6-4fb0-9f3c-c8254dece2f6" (UID: "20fa8b33-c9e6-4fb0-9f3c-c8254dece2f6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:53:29.029400 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:29.028878    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53231544-cdae-4031-a635-1260aab4def3-kube-api-access-p6ndb\" (UniqueName: \"kubernetes.io/projected/53231544-cdae-4031-a635-1260aab4def3-kube-api-access-p6ndb\") pod \"53231544-cdae-4031-a635-1260aab4def3\" (UID: \"53231544-cdae-4031-a635-1260aab4def3\") "
Oct 31 12:53:29.030138 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:29.030094    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20fa8b33-c9e6-4fb0-9f3c-c8254dece2f6-istiod-ca-cert" pod "20fa8b33-c9e6-4fb0-9f3c-c8254dece2f6" (UID: "20fa8b33-c9e6-4fb0-9f3c-c8254dece2f6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:53:29.030791 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:29.030538    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20fa8b33-c9e6-4fb0-9f3c-c8254dece2f6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20fa8b33-c9e6-4fb0-9f3c-c8254dece2f6-istiod-ca-cert\") pod \"20fa8b33-c9e6-4fb0-9f3c-c8254dece2f6\" (UID: \"20fa8b33-c9e6-4fb0-9f3c-c8254dece2f6\") "
Oct 31 12:53:29.031322 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:29.031207    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20fa8b33-c9e6-4fb0-9f3c-c8254dece2f6-kube-api-access-xtllc\" (UniqueName: \"kubernetes.io/projected/20fa8b33-c9e6-4fb0-9f3c-c8254dece2f6-kube-api-access-xtllc\") pod \"20fa8b33-c9e6-4fb0-9f3c-c8254dece2f6\" (UID: \"20fa8b33-c9e6-4fb0-9f3c-c8254dece2f6\") "
Oct 31 12:53:29.031906 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:29.031776    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20fa8b33-c9e6-4fb0-9f3c-c8254dece2f6-istio-token\" (UniqueName: \"kubernetes.io/projected/20fa8b33-c9e6-4fb0-9f3c-c8254dece2f6-istio-token\") pod \"20fa8b33-c9e6-4fb0-9f3c-c8254dece2f6\" (UID: \"20fa8b33-c9e6-4fb0-9f3c-c8254dece2f6\") "
Oct 31 12:53:29.032477 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:29.032440    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20fa8b33-c9e6-4fb0-9f3c-c8254dece2f6-workload-socket" pod "20fa8b33-c9e6-4fb0-9f3c-c8254dece2f6" (UID: "20fa8b33-c9e6-4fb0-9f3c-c8254dece2f6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:53:29.032685 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:29.032627    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20fa8b33-c9e6-4fb0-9f3c-c8254dece2f6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20fa8b33-c9e6-4fb0-9f3c-c8254dece2f6-workload-socket\") pod \"20fa8b33-c9e6-4fb0-9f3c-c8254dece2f6\" (UID: \"20fa8b33-c9e6-4fb0-9f3c-c8254dece2f6\") "
Oct 31 12:53:29.033228 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:29.032962    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20fa8b33-c9e6-4fb0-9f3c-c8254dece2f6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20fa8b33-c9e6-4fb0-9f3c-c8254dece2f6-workload-certs\") pod \"20fa8b33-c9e6-4fb0-9f3c-c8254dece2f6\" (UID: \"20fa8b33-c9e6-4fb0-9f3c-c8254dece2f6\") "
Oct 31 12:53:29.033228 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:29.033192    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20fa8b33-c9e6-4fb0-9f3c-c8254dece2f6-workload-certs" pod "20fa8b33-c9e6-4fb0-9f3c-c8254dece2f6" (UID: "20fa8b33-c9e6-4fb0-9f3c-c8254dece2f6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:53:29.033965 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:29.033728    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/20fa8b33-c9e6-4fb0-9f3c-c8254dece2f6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20fa8b33-c9e6-4fb0-9f3c-c8254dece2f6-istio-podinfo\") pod \"20fa8b33-c9e6-4fb0-9f3c-c8254dece2f6\" (UID: \"20fa8b33-c9e6-4fb0-9f3c-c8254dece2f6\") "
Oct 31 12:53:29.035003 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:29.034971    5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20fa8b33-c9e6-4fb0-9f3c-c8254dece2f6-workload-socket\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:29.035191 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:29.035174    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53231544-cdae-4031-a635-1260aab4def3-maas-api-xks-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:29.035305 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:29.035291    5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20fa8b33-c9e6-4fb0-9f3c-c8254dece2f6-workload-certs\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:29.035432 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:29.035417    5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20fa8b33-c9e6-4fb0-9f3c-c8254dece2f6-istio-data\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:29.035705 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:29.035632    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53231544-cdae-4031-a635-1260aab4def3-maas-api-openshift-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:29.035705 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:29.035660    5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20fa8b33-c9e6-4fb0-9f3c-c8254dece2f6-credential-socket\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:29.035705 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:29.035678    5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20fa8b33-c9e6-4fb0-9f3c-c8254dece2f6-istiod-ca-cert\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:29.102097 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:29.102054    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20fa8b33-c9e6-4fb0-9f3c-c8254dece2f6-istio-token" pod "20fa8b33-c9e6-4fb0-9f3c-c8254dece2f6" (UID: "20fa8b33-c9e6-4fb0-9f3c-c8254dece2f6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:53:29.106365 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:29.104784    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/20fa8b33-c9e6-4fb0-9f3c-c8254dece2f6-istio-podinfo" pod "20fa8b33-c9e6-4fb0-9f3c-c8254dece2f6" (UID: "20fa8b33-c9e6-4fb0-9f3c-c8254dece2f6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 12:53:29.117525 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:29.117348    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53231544-cdae-4031-a635-1260aab4def3-kube-api-access-p6ndb" pod "53231544-cdae-4031-a635-1260aab4def3" (UID: "53231544-cdae-4031-a635-1260aab4def3"). InnerVolumeSpecName "kube-api-access-p6ndb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:53:29.126200 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:29.126110    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20fa8b33-c9e6-4fb0-9f3c-c8254dece2f6-istio-envoy" pod "20fa8b33-c9e6-4fb0-9f3c-c8254dece2f6" (UID: "20fa8b33-c9e6-4fb0-9f3c-c8254dece2f6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:53:29.126699 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:29.126627    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20fa8b33-c9e6-4fb0-9f3c-c8254dece2f6-kube-api-access-xtllc" pod "20fa8b33-c9e6-4fb0-9f3c-c8254dece2f6" (UID: "20fa8b33-c9e6-4fb0-9f3c-c8254dece2f6"). InnerVolumeSpecName "kube-api-access-xtllc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:53:29.145610 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:29.143676    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xtllc\" (UniqueName: \"kubernetes.io/projected/20fa8b33-c9e6-4fb0-9f3c-c8254dece2f6-kube-api-access-xtllc\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:29.145610 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:29.143706    5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20fa8b33-c9e6-4fb0-9f3c-c8254dece2f6-istio-token\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:29.145610 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:29.143721    5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20fa8b33-c9e6-4fb0-9f3c-c8254dece2f6-istio-podinfo\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:29.145610 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:29.143735    5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20fa8b33-c9e6-4fb0-9f3c-c8254dece2f6-istio-envoy\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:29.145610 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:29.143749    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6ndb\" (UniqueName: \"kubernetes.io/projected/53231544-cdae-4031-a635-1260aab4def3-kube-api-access-p6ndb\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:29.145938 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:29.145871    5060 generic.go:356] "Generic (PLEG): container finished" podID="41d07641-1c30-4e59-b402-fd7505e781ba" containerID="dfb44bb00e52e91a04fb972e54c41471fb0f804fc6f3769d653cbd1212f00dea" exitCode=0
Oct 31 12:53:29.146257 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:29.146166    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-b9pn6" event={"ID":"41d07641-1c30-4e59-b402-fd7505e781ba","Type":"ContainerDied","Data":"dfb44bb00e52e91a04fb972e54c41471fb0f804fc6f3769d653cbd1212f00dea"}
Oct 31 12:53:29.148524 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:29.148058    5060 generic.go:356] "Generic (PLEG): container finished" podID="0070ea70-06e9-4733-8d11-ecb4cb993ff7" containerID="854af3eb3ad954edf26df321b78fd4ad65ec92e0b1d06ab91d690d848515f96b" exitCode=0
Oct 31 12:53:29.148524 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:29.148200    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-ds5zz" event={"ID":"0070ea70-06e9-4733-8d11-ecb4cb993ff7","Type":"ContainerDied","Data":"854af3eb3ad954edf26df321b78fd4ad65ec92e0b1d06ab91d690d848515f96b"}
Oct 31 12:53:29.148524 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:29.148232    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-ds5zz" event={"ID":"0070ea70-06e9-4733-8d11-ecb4cb993ff7","Type":"ContainerDied","Data":"1f61ced7df877c2828cf38dfb610fe4f1b1fd81be04e13a04f5d6084d5426bed"}
Oct 31 12:53:29.148524 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:29.148253    5060 scope.go:122] "RemoveContainer" containerID="854af3eb3ad954edf26df321b78fd4ad65ec92e0b1d06ab91d690d848515f96b"
Oct 31 12:53:29.148814 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:29.148733    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-ds5zz"
Oct 31 12:53:29.185135 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:29.168929    5060 generic.go:356] "Generic (PLEG): container finished" podID="20fa8b33-c9e6-4fb0-9f3c-c8254dece2f6" containerID="6c60c3bdad21a71ead4fe8545b4a414aae5df9b36a2afad4d6cc8b5c09170636" exitCode=0
Oct 31 12:53:29.185135 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:29.169031    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-vxzjg" event={"ID":"20fa8b33-c9e6-4fb0-9f3c-c8254dece2f6","Type":"ContainerDied","Data":"6c60c3bdad21a71ead4fe8545b4a414aae5df9b36a2afad4d6cc8b5c09170636"}
Oct 31 12:53:29.185135 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:29.169089    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-vxzjg" event={"ID":"20fa8b33-c9e6-4fb0-9f3c-c8254dece2f6","Type":"ContainerDied","Data":"7eee1d352b0d46eb4f360488182eb6f14acce1becd45890a1dea9a0b5e31c334"}
Oct 31 12:53:29.185135 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:29.169169    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-vxzjg"
Oct 31 12:53:29.185135 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:29.173223    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e98df8-wbs75" event={"ID":"53231544-cdae-4031-a635-1260aab4def3","Type":"ContainerDied","Data":"63e70957d777e94acc067a72ad51f6fe7d81355fa65ef603dccf4aa2388a1b93"}
Oct 31 12:53:29.185135 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:29.173272    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="63e70957d777e94acc067a72ad51f6fe7d81355fa65ef603dccf4aa2388a1b93"
Oct 31 12:53:29.185135 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:29.173343    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e98df8-wbs75"
Oct 31 12:53:29.185135 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:29.182144    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-zg66r"
Oct 31 12:53:29.185135 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:29.183356    5060 generic.go:356] "Generic (PLEG): container finished" podID="ea38d319-4637-47c0-bf7a-125b61dac2fe" containerID="7852c16adc99dafa80d1c9727e019f8620e0370ced0b974d88f7a36f05a969b9" exitCode=0
Oct 31 12:53:29.185135 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:29.183408    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-zg66r" event={"ID":"ea38d319-4637-47c0-bf7a-125b61dac2fe","Type":"ContainerDied","Data":"7852c16adc99dafa80d1c9727e019f8620e0370ced0b974d88f7a36f05a969b9"}
Oct 31 12:53:29.185135 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:29.184552    5060 scope.go:122] "RemoveContainer" containerID="854af3eb3ad954edf26df321b78fd4ad65ec92e0b1d06ab91d690d848515f96b"
Oct 31 12:53:29.185135 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:53:29.184851    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"854af3eb3ad954edf26df321b78fd4ad65ec92e0b1d06ab91d690d848515f96b\": container with ID starting with 854af3eb3ad954edf26df321b78fd4ad65ec92e0b1d06ab91d690d848515f96b not found: ID does not exist" containerID="854af3eb3ad954edf26df321b78fd4ad65ec92e0b1d06ab91d690d848515f96b"
Oct 31 12:53:29.185135 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:29.184875    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"854af3eb3ad954edf26df321b78fd4ad65ec92e0b1d06ab91d690d848515f96b"} err="failed to get container status \"854af3eb3ad954edf26df321b78fd4ad65ec92e0b1d06ab91d690d848515f96b\": rpc error: code = NotFound desc = could not find container \"854af3eb3ad954edf26df321b78fd4ad65ec92e0b1d06ab91d690d848515f96b\": container with ID starting with 854af3eb3ad954edf26df321b78fd4ad65ec92e0b1d06ab91d690d848515f96b not found: ID does not exist"
Oct 31 12:53:29.185135 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:29.184897    5060 scope.go:122] "RemoveContainer" containerID="6c60c3bdad21a71ead4fe8545b4a414aae5df9b36a2afad4d6cc8b5c09170636"
Oct 31 12:53:29.214674 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:29.214389    5060 scope.go:122] "RemoveContainer" containerID="cec778106ab298c2c59e82c8abfe4bb666ce3948bf26f0b090cf4e8426f6b03a"
Oct 31 12:53:29.219407 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:29.219363    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-vxzjg"]
Oct 31 12:53:29.224674 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:29.224643    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-vxzjg"]
Oct 31 12:53:29.244833 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:29.243139    5060 scope.go:122] "RemoveContainer" containerID="6c60c3bdad21a71ead4fe8545b4a414aae5df9b36a2afad4d6cc8b5c09170636"
Oct 31 12:53:29.253703 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:53:29.253670    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c60c3bdad21a71ead4fe8545b4a414aae5df9b36a2afad4d6cc8b5c09170636\": container with ID starting with 6c60c3bdad21a71ead4fe8545b4a414aae5df9b36a2afad4d6cc8b5c09170636 not found: ID does not exist" containerID="6c60c3bdad21a71ead4fe8545b4a414aae5df9b36a2afad4d6cc8b5c09170636"
Oct 31 12:53:29.254062 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:29.254024    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c60c3bdad21a71ead4fe8545b4a414aae5df9b36a2afad4d6cc8b5c09170636"} err="failed to get container status \"6c60c3bdad21a71ead4fe8545b4a414aae5df9b36a2afad4d6cc8b5c09170636\": rpc error: code = NotFound desc = could not find container \"6c60c3bdad21a71ead4fe8545b4a414aae5df9b36a2afad4d6cc8b5c09170636\": container with ID starting with 6c60c3bdad21a71ead4fe8545b4a414aae5df9b36a2afad4d6cc8b5c09170636 not found: ID does not exist"
Oct 31 12:53:29.256055 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:29.256035    5060 scope.go:122] "RemoveContainer" containerID="cec778106ab298c2c59e82c8abfe4bb666ce3948bf26f0b090cf4e8426f6b03a"
Oct 31 12:53:29.256188 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:29.256160    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea38d319-4637-47c0-bf7a-125b61dac2fe-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea38d319-4637-47c0-bf7a-125b61dac2fe-istiod-ca-cert\") pod \"ea38d319-4637-47c0-bf7a-125b61dac2fe\" (UID: \"ea38d319-4637-47c0-bf7a-125b61dac2fe\") "
Oct 31 12:53:29.256261 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:29.256234    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea38d319-4637-47c0-bf7a-125b61dac2fe-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea38d319-4637-47c0-bf7a-125b61dac2fe-workload-certs\") pod \"ea38d319-4637-47c0-bf7a-125b61dac2fe\" (UID: \"ea38d319-4637-47c0-bf7a-125b61dac2fe\") "
Oct 31 12:53:29.256338 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:29.256322    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea38d319-4637-47c0-bf7a-125b61dac2fe-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea38d319-4637-47c0-bf7a-125b61dac2fe-istio-data\") pod \"ea38d319-4637-47c0-bf7a-125b61dac2fe\" (UID: \"ea38d319-4637-47c0-bf7a-125b61dac2fe\") "
Oct 31 12:53:29.256638 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:29.256379    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea38d319-4637-47c0-bf7a-125b61dac2fe-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea38d319-4637-47c0-bf7a-125b61dac2fe-workload-socket\") pod \"ea38d319-4637-47c0-bf7a-125b61dac2fe\" (UID: \"ea38d319-4637-47c0-bf7a-125b61dac2fe\") "
Oct 31 12:53:29.256638 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:29.256421    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea38d319-4637-47c0-bf7a-125b61dac2fe-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea38d319-4637-47c0-bf7a-125b61dac2fe-istio-envoy\") pod \"ea38d319-4637-47c0-bf7a-125b61dac2fe\" (UID: \"ea38d319-4637-47c0-bf7a-125b61dac2fe\") "
Oct 31 12:53:29.256638 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:29.256593    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea38d319-4637-47c0-bf7a-125b61dac2fe-istio-token\" (UniqueName: \"kubernetes.io/projected/ea38d319-4637-47c0-bf7a-125b61dac2fe-istio-token\") pod \"ea38d319-4637-47c0-bf7a-125b61dac2fe\" (UID: \"ea38d319-4637-47c0-bf7a-125b61dac2fe\") "
Oct 31 12:53:29.256828 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:29.256637    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea38d319-4637-47c0-bf7a-125b61dac2fe-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea38d319-4637-47c0-bf7a-125b61dac2fe-credential-socket\") pod \"ea38d319-4637-47c0-bf7a-125b61dac2fe\" (UID: \"ea38d319-4637-47c0-bf7a-125b61dac2fe\") "
Oct 31 12:53:29.256828 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:29.256681    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea38d319-4637-47c0-bf7a-125b61dac2fe-kube-api-access-v4rz4\" (UniqueName: \"kubernetes.io/projected/ea38d319-4637-47c0-bf7a-125b61dac2fe-kube-api-access-v4rz4\") pod \"ea38d319-4637-47c0-bf7a-125b61dac2fe\" (UID: \"ea38d319-4637-47c0-bf7a-125b61dac2fe\") "
Oct 31 12:53:29.256964 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:29.256826    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ea38d319-4637-47c0-bf7a-125b61dac2fe-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea38d319-4637-47c0-bf7a-125b61dac2fe-istio-podinfo\") pod \"ea38d319-4637-47c0-bf7a-125b61dac2fe\" (UID: \"ea38d319-4637-47c0-bf7a-125b61dac2fe\") "
Oct 31 12:53:29.257430 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:29.257408    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea38d319-4637-47c0-bf7a-125b61dac2fe-workload-socket" pod "ea38d319-4637-47c0-bf7a-125b61dac2fe" (UID: "ea38d319-4637-47c0-bf7a-125b61dac2fe"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:53:29.258395 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:29.258370    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea38d319-4637-47c0-bf7a-125b61dac2fe-istiod-ca-cert" pod "ea38d319-4637-47c0-bf7a-125b61dac2fe" (UID: "ea38d319-4637-47c0-bf7a-125b61dac2fe"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:53:29.258785 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:29.258764    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea38d319-4637-47c0-bf7a-125b61dac2fe-workload-certs" pod "ea38d319-4637-47c0-bf7a-125b61dac2fe" (UID: "ea38d319-4637-47c0-bf7a-125b61dac2fe"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:53:29.259074 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:29.259053    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea38d319-4637-47c0-bf7a-125b61dac2fe-istio-data" pod "ea38d319-4637-47c0-bf7a-125b61dac2fe" (UID: "ea38d319-4637-47c0-bf7a-125b61dac2fe"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:53:29.259193 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:53:29.259173    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cec778106ab298c2c59e82c8abfe4bb666ce3948bf26f0b090cf4e8426f6b03a\": container with ID starting with cec778106ab298c2c59e82c8abfe4bb666ce3948bf26f0b090cf4e8426f6b03a not found: ID does not exist" containerID="cec778106ab298c2c59e82c8abfe4bb666ce3948bf26f0b090cf4e8426f6b03a"
Oct 31 12:53:29.259302 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:29.259278    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cec778106ab298c2c59e82c8abfe4bb666ce3948bf26f0b090cf4e8426f6b03a"} err="failed to get container status \"cec778106ab298c2c59e82c8abfe4bb666ce3948bf26f0b090cf4e8426f6b03a\": rpc error: code = NotFound desc = could not find container \"cec778106ab298c2c59e82c8abfe4bb666ce3948bf26f0b090cf4e8426f6b03a\": container with ID starting with cec778106ab298c2c59e82c8abfe4bb666ce3948bf26f0b090cf4e8426f6b03a not found: ID does not exist"
Oct 31 12:53:29.259383 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:29.259305    5060 scope.go:122] "RemoveContainer" containerID="7852c16adc99dafa80d1c9727e019f8620e0370ced0b974d88f7a36f05a969b9"
Oct 31 12:53:29.262543 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:29.262476    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ea38d319-4637-47c0-bf7a-125b61dac2fe-istio-podinfo" pod "ea38d319-4637-47c0-bf7a-125b61dac2fe" (UID: "ea38d319-4637-47c0-bf7a-125b61dac2fe"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 12:53:29.262698 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:29.262655    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea38d319-4637-47c0-bf7a-125b61dac2fe-istio-token" pod "ea38d319-4637-47c0-bf7a-125b61dac2fe" (UID: "ea38d319-4637-47c0-bf7a-125b61dac2fe"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:53:29.262831 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:29.262807    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea38d319-4637-47c0-bf7a-125b61dac2fe-credential-socket" pod "ea38d319-4637-47c0-bf7a-125b61dac2fe" (UID: "ea38d319-4637-47c0-bf7a-125b61dac2fe"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:53:29.265868 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:29.265847    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea38d319-4637-47c0-bf7a-125b61dac2fe-kube-api-access-v4rz4" pod "ea38d319-4637-47c0-bf7a-125b61dac2fe" (UID: "ea38d319-4637-47c0-bf7a-125b61dac2fe"). InnerVolumeSpecName "kube-api-access-v4rz4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:53:29.266001 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:29.265935    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea38d319-4637-47c0-bf7a-125b61dac2fe-istio-envoy" pod "ea38d319-4637-47c0-bf7a-125b61dac2fe" (UID: "ea38d319-4637-47c0-bf7a-125b61dac2fe"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:53:29.286643 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:53:29.286599    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9aa65f61_54d4_4e45_8f33_51ea2fe5bc56.slice/crio-662c02e3a518a464c7902ab0551f07e2bf9ab4ffd61e1e960b511d4ee6418d60 WatchSource:0}: Error finding container 662c02e3a518a464c7902ab0551f07e2bf9ab4ffd61e1e960b511d4ee6418d60: Status 404 returned error can't find the container with id 662c02e3a518a464c7902ab0551f07e2bf9ab4ffd61e1e960b511d4ee6418d60
Oct 31 12:53:29.312203 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:29.311782    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-b9pn6"
Oct 31 12:53:29.312203 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:29.312135    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-9b62a0-8445958dbb-99wdf"]
Oct 31 12:53:29.350241 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:29.350205    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-ds5zz"]
Oct 31 12:53:29.350722 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:29.350702    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-ds5zz"]
Oct 31 12:53:29.365702 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:29.365656    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41d07641-1c30-4e59-b402-fd7505e781ba-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41d07641-1c30-4e59-b402-fd7505e781ba-credential-socket\") pod \"41d07641-1c30-4e59-b402-fd7505e781ba\" (UID: \"41d07641-1c30-4e59-b402-fd7505e781ba\") "
Oct 31 12:53:29.369547 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:29.366031    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41d07641-1c30-4e59-b402-fd7505e781ba-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41d07641-1c30-4e59-b402-fd7505e781ba-workload-certs\") pod \"41d07641-1c30-4e59-b402-fd7505e781ba\" (UID: \"41d07641-1c30-4e59-b402-fd7505e781ba\") "
Oct 31 12:53:29.369855 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:29.369830    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41d07641-1c30-4e59-b402-fd7505e781ba-kube-api-access-42dl6\" (UniqueName: \"kubernetes.io/projected/41d07641-1c30-4e59-b402-fd7505e781ba-kube-api-access-42dl6\") pod \"41d07641-1c30-4e59-b402-fd7505e781ba\" (UID: \"41d07641-1c30-4e59-b402-fd7505e781ba\") "
Oct 31 12:53:29.370085 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:29.370046    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41d07641-1c30-4e59-b402-fd7505e781ba-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41d07641-1c30-4e59-b402-fd7505e781ba-istiod-ca-cert\") pod \"41d07641-1c30-4e59-b402-fd7505e781ba\" (UID: \"41d07641-1c30-4e59-b402-fd7505e781ba\") "
Oct 31 12:53:29.376239 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:29.376027    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41d07641-1c30-4e59-b402-fd7505e781ba-credential-socket" pod "41d07641-1c30-4e59-b402-fd7505e781ba" (UID: "41d07641-1c30-4e59-b402-fd7505e781ba"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:53:29.378178 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:29.376520    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41d07641-1c30-4e59-b402-fd7505e781ba-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41d07641-1c30-4e59-b402-fd7505e781ba-istio-data\") pod \"41d07641-1c30-4e59-b402-fd7505e781ba\" (UID: \"41d07641-1c30-4e59-b402-fd7505e781ba\") "
Oct 31 12:53:29.404436 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:29.404406    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41d07641-1c30-4e59-b402-fd7505e781ba-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41d07641-1c30-4e59-b402-fd7505e781ba-workload-socket\") pod \"41d07641-1c30-4e59-b402-fd7505e781ba\" (UID: \"41d07641-1c30-4e59-b402-fd7505e781ba\") "
Oct 31 12:53:29.426458 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:29.376774    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41d07641-1c30-4e59-b402-fd7505e781ba-workload-certs" pod "41d07641-1c30-4e59-b402-fd7505e781ba" (UID: "41d07641-1c30-4e59-b402-fd7505e781ba"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:53:29.426458 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:29.385636    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41d07641-1c30-4e59-b402-fd7505e781ba-istio-data" pod "41d07641-1c30-4e59-b402-fd7505e781ba" (UID: "41d07641-1c30-4e59-b402-fd7505e781ba"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:53:29.426458 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:29.425633    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41d07641-1c30-4e59-b402-fd7505e781ba-workload-socket" pod "41d07641-1c30-4e59-b402-fd7505e781ba" (UID: "41d07641-1c30-4e59-b402-fd7505e781ba"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:53:29.427829 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:29.427208    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41d07641-1c30-4e59-b402-fd7505e781ba-istio-token\" (UniqueName: \"kubernetes.io/projected/41d07641-1c30-4e59-b402-fd7505e781ba-istio-token\") pod \"41d07641-1c30-4e59-b402-fd7505e781ba\" (UID: \"41d07641-1c30-4e59-b402-fd7505e781ba\") "
Oct 31 12:53:29.428294 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:29.428265    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41d07641-1c30-4e59-b402-fd7505e781ba-istiod-ca-cert" pod "41d07641-1c30-4e59-b402-fd7505e781ba" (UID: "41d07641-1c30-4e59-b402-fd7505e781ba"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:53:29.428517 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:29.428436    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41d07641-1c30-4e59-b402-fd7505e781ba-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41d07641-1c30-4e59-b402-fd7505e781ba-istio-envoy\") pod \"41d07641-1c30-4e59-b402-fd7505e781ba\" (UID: \"41d07641-1c30-4e59-b402-fd7505e781ba\") "
Oct 31 12:53:29.428678 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:29.428662    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/41d07641-1c30-4e59-b402-fd7505e781ba-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41d07641-1c30-4e59-b402-fd7505e781ba-istio-podinfo\") pod \"41d07641-1c30-4e59-b402-fd7505e781ba\" (UID: \"41d07641-1c30-4e59-b402-fd7505e781ba\") "
Oct 31 12:53:29.430582 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:29.430550    5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea38d319-4637-47c0-bf7a-125b61dac2fe-credential-socket\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:29.447960 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:29.430721    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4rz4\" (UniqueName: \"kubernetes.io/projected/ea38d319-4637-47c0-bf7a-125b61dac2fe-kube-api-access-v4rz4\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:29.447960 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:29.447606    5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41d07641-1c30-4e59-b402-fd7505e781ba-istiod-ca-cert\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:29.447960 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:29.447632    5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41d07641-1c30-4e59-b402-fd7505e781ba-istio-data\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:29.447960 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:29.447647    5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41d07641-1c30-4e59-b402-fd7505e781ba-workload-socket\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:29.447960 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:29.447662    5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea38d319-4637-47c0-bf7a-125b61dac2fe-istio-podinfo\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:29.447960 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:29.447676    5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea38d319-4637-47c0-bf7a-125b61dac2fe-istiod-ca-cert\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:29.447960 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:29.447690    5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea38d319-4637-47c0-bf7a-125b61dac2fe-workload-certs\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:29.447960 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:29.447709    5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea38d319-4637-47c0-bf7a-125b61dac2fe-istio-data\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:29.447960 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:29.447723    5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea38d319-4637-47c0-bf7a-125b61dac2fe-workload-socket\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:29.447960 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:29.447738    5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea38d319-4637-47c0-bf7a-125b61dac2fe-istio-envoy\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:29.447960 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:29.447754    5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41d07641-1c30-4e59-b402-fd7505e781ba-credential-socket\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:29.447960 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:29.447768    5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea38d319-4637-47c0-bf7a-125b61dac2fe-istio-token\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:29.447960 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:29.447782    5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41d07641-1c30-4e59-b402-fd7505e781ba-workload-certs\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:29.462420 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:29.461642    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/41d07641-1c30-4e59-b402-fd7505e781ba-istio-podinfo" pod "41d07641-1c30-4e59-b402-fd7505e781ba" (UID: "41d07641-1c30-4e59-b402-fd7505e781ba"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 12:53:29.487910 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:29.464036    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41d07641-1c30-4e59-b402-fd7505e781ba-istio-envoy" pod "41d07641-1c30-4e59-b402-fd7505e781ba" (UID: "41d07641-1c30-4e59-b402-fd7505e781ba"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:53:29.487910 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:29.466075    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41d07641-1c30-4e59-b402-fd7505e781ba-kube-api-access-42dl6" pod "41d07641-1c30-4e59-b402-fd7505e781ba" (UID: "41d07641-1c30-4e59-b402-fd7505e781ba"). InnerVolumeSpecName "kube-api-access-42dl6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:53:29.487910 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:29.468855    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41d07641-1c30-4e59-b402-fd7505e781ba-istio-token" pod "41d07641-1c30-4e59-b402-fd7505e781ba" (UID: "41d07641-1c30-4e59-b402-fd7505e781ba"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:53:29.550018 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:29.549978    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42dl6\" (UniqueName: \"kubernetes.io/projected/41d07641-1c30-4e59-b402-fd7505e781ba-kube-api-access-42dl6\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:29.550018 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:29.550020    5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41d07641-1c30-4e59-b402-fd7505e781ba-istio-token\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:29.550208 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:29.550034    5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41d07641-1c30-4e59-b402-fd7505e781ba-istio-envoy\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:29.550208 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:29.550048    5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41d07641-1c30-4e59-b402-fd7505e781ba-istio-podinfo\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:30.217774 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:53:30.217727    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7phb" containerName="extractor"
Oct 31 12:53:30.233726 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:30.233684    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-b9pn6" event={"ID":"41d07641-1c30-4e59-b402-fd7505e781ba","Type":"ContainerDied","Data":"9cd5c391d8f2cb2f6b53ce112e11f43d8ae53ed19c51b8c9b83e057598cac628"}
Oct 31 12:53:30.234089 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:30.234049    5060 scope.go:122] "RemoveContainer" containerID="dfb44bb00e52e91a04fb972e54c41471fb0f804fc6f3769d653cbd1212f00dea"
Oct 31 12:53:30.234458 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:30.234429    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-b9pn6"
Oct 31 12:53:30.236613 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:53:30.236570    5060 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod330523c8_06c9_42a2_ac3f_9dd4ad62c392.slice/crio-d190e7520eebe52b0c749ad77fb372857ff3f35218eb3dc8fea952d63280f3d9.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1e6c7f5_d872_4549_b950_1e751b6374ab.slice/crio-15eb93212f666238582935bcfb7fb66c4776d563bb22273bc4eb80596e90659b.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1e6c7f5_d872_4549_b950_1e751b6374ab.slice/crio-conmon-15eb93212f666238582935bcfb7fb66c4776d563bb22273bc4eb80596e90659b.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod330523c8_06c9_42a2_ac3f_9dd4ad62c392.slice/crio-conmon-d190e7520eebe52b0c749ad77fb372857ff3f35218eb3dc8fea952d63280f3d9.scope\": RecentStats: unable to find data in memory cache]"
Oct 31 12:53:30.245569 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:30.245016    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9b62a0-8445958dbb-99wdf" event={"ID":"9aa65f61-54d4-4e45-8f33-51ea2fe5bc56","Type":"ContainerStarted","Data":"662c02e3a518a464c7902ab0551f07e2bf9ab4ffd61e1e960b511d4ee6418d60"}
Oct 31 12:53:30.249126 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:30.249006    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-zg66r" event={"ID":"ea38d319-4637-47c0-bf7a-125b61dac2fe","Type":"ContainerDied","Data":"cbd9f0525d16e5a919d517fb91c92cd7b9618dad5c913f647af6ecca2905f2b6"}
Oct 31 12:53:30.249382 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:30.249274    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-zg66r"
Oct 31 12:53:30.282760 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:30.282721    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-b9pn6"]
Oct 31 12:53:30.301766 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:30.301636    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-b9pn6"]
Oct 31 12:53:30.311121 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:30.311091    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-zg66r"]
Oct 31 12:53:30.388719 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:30.388456    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-zg66r"]
Oct 31 12:53:31.091616 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:31.091583    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0070ea70-06e9-4733-8d11-ecb4cb993ff7" path="/var/lib/kubelet/pods/0070ea70-06e9-4733-8d11-ecb4cb993ff7/volumes"
Oct 31 12:53:31.092184 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:31.092168    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20fa8b33-c9e6-4fb0-9f3c-c8254dece2f6" path="/var/lib/kubelet/pods/20fa8b33-c9e6-4fb0-9f3c-c8254dece2f6/volumes"
Oct 31 12:53:31.092888 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:31.092866    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41d07641-1c30-4e59-b402-fd7505e781ba" path="/var/lib/kubelet/pods/41d07641-1c30-4e59-b402-fd7505e781ba/volumes"
Oct 31 12:53:31.093665 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:31.093648    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea38d319-4637-47c0-bf7a-125b61dac2fe" path="/var/lib/kubelet/pods/ea38d319-4637-47c0-bf7a-125b61dac2fe/volumes"
Oct 31 12:53:31.262470 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:31.262434    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9b62a0-8445958dbb-99wdf" event={"ID":"9aa65f61-54d4-4e45-8f33-51ea2fe5bc56","Type":"ContainerStarted","Data":"ad65c2b0b7be6499f4f7a5ce027fdf7cff5ae574e08b2031866792e3c6e77401"}
Oct 31 12:53:31.262996 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:31.262651    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9b62a0-8445958dbb-99wdf"
Oct 31 12:53:31.290359 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:31.290298    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9b62a0-8445958dbb-99wdf" podStartSLOduration=3.17175759 podStartE2EDuration="4.290280193s" totalImagesPullingTime="1.118522603s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:53:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:53:31.288729322 +0000 UTC m=+1574.391797036" watchObservedRunningTime="2026-10-31 12:53:31.290280193 +0000 UTC m=+1574.393347906"
Oct 31 12:53:31.576432 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:53:31.576374    5060 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod330523c8_06c9_42a2_ac3f_9dd4ad62c392.slice/crio-conmon-d190e7520eebe52b0c749ad77fb372857ff3f35218eb3dc8fea952d63280f3d9.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1e6c7f5_d872_4549_b950_1e751b6374ab.slice/crio-15eb93212f666238582935bcfb7fb66c4776d563bb22273bc4eb80596e90659b.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod330523c8_06c9_42a2_ac3f_9dd4ad62c392.slice/crio-d190e7520eebe52b0c749ad77fb372857ff3f35218eb3dc8fea952d63280f3d9.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1e6c7f5_d872_4549_b950_1e751b6374ab.slice/crio-conmon-15eb93212f666238582935bcfb7fb66c4776d563bb22273bc4eb80596e90659b.scope\": RecentStats: unable to find data in memory cache]"
Oct 31 12:53:31.578954 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:53:31.578902    5060 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod330523c8_06c9_42a2_ac3f_9dd4ad62c392.slice/crio-d190e7520eebe52b0c749ad77fb372857ff3f35218eb3dc8fea952d63280f3d9.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1e6c7f5_d872_4549_b950_1e751b6374ab.slice/crio-conmon-15eb93212f666238582935bcfb7fb66c4776d563bb22273bc4eb80596e90659b.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1e6c7f5_d872_4549_b950_1e751b6374ab.slice/crio-15eb93212f666238582935bcfb7fb66c4776d563bb22273bc4eb80596e90659b.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod330523c8_06c9_42a2_ac3f_9dd4ad62c392.slice/crio-conmon-d190e7520eebe52b0c749ad77fb372857ff3f35218eb3dc8fea952d63280f3d9.scope\": RecentStats: unable to find data in memory cache]"
Oct 31 12:53:32.363019 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:53:32.362973    5060 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1e6c7f5_d872_4549_b950_1e751b6374ab.slice/crio-conmon-15eb93212f666238582935bcfb7fb66c4776d563bb22273bc4eb80596e90659b.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod330523c8_06c9_42a2_ac3f_9dd4ad62c392.slice/crio-conmon-d190e7520eebe52b0c749ad77fb372857ff3f35218eb3dc8fea952d63280f3d9.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod330523c8_06c9_42a2_ac3f_9dd4ad62c392.slice/crio-d190e7520eebe52b0c749ad77fb372857ff3f35218eb3dc8fea952d63280f3d9.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1e6c7f5_d872_4549_b950_1e751b6374ab.slice/crio-15eb93212f666238582935bcfb7fb66c4776d563bb22273bc4eb80596e90659b.scope\": RecentStats: unable to find data in memory cache]"
Oct 31 12:53:32.513956 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:32.513917    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d086f3-58b5756f6f-m4jx8"]
Oct 31 12:53:32.514740 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:32.514722    5060 cpu_manager.go:414] "containerMap: removing container" podUID="0070ea70-06e9-4733-8d11-ecb4cb993ff7" containerName="istio-proxy"
Oct 31 12:53:32.514740 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:32.514739    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0070ea70-06e9-4733-8d11-ecb4cb993ff7" containerName="istio-proxy"
Oct 31 12:53:32.514866 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:32.514765    5060 cpu_manager.go:414] "containerMap: removing container" podUID="20fa8b33-c9e6-4fb0-9f3c-c8254dece2f6" containerName="istio-proxy"
Oct 31 12:53:32.514866 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:32.514770    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20fa8b33-c9e6-4fb0-9f3c-c8254dece2f6" containerName="istio-proxy"
Oct 31 12:53:32.514866 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:32.514786    5060 cpu_manager.go:414] "containerMap: removing container" podUID="ea38d319-4637-47c0-bf7a-125b61dac2fe" containerName="istio-proxy"
Oct 31 12:53:32.514866 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:32.514791    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea38d319-4637-47c0-bf7a-125b61dac2fe" containerName="istio-proxy"
Oct 31 12:53:32.514866 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:32.514805    5060 cpu_manager.go:414] "containerMap: removing container" podUID="20fa8b33-c9e6-4fb0-9f3c-c8254dece2f6" containerName="istio-proxy"
Oct 31 12:53:32.514866 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:32.514811    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20fa8b33-c9e6-4fb0-9f3c-c8254dece2f6" containerName="istio-proxy"
Oct 31 12:53:32.514866 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:32.514822    5060 cpu_manager.go:414] "containerMap: removing container" podUID="41d07641-1c30-4e59-b402-fd7505e781ba" containerName="istio-proxy"
Oct 31 12:53:32.514866 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:32.514827    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41d07641-1c30-4e59-b402-fd7505e781ba" containerName="istio-proxy"
Oct 31 12:53:32.514866 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:32.514851    5060 cpu_manager.go:414] "containerMap: removing container" podUID="53231544-cdae-4031-a635-1260aab4def3" containerName="revoke-keys"
Oct 31 12:53:32.514866 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:32.514856    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53231544-cdae-4031-a635-1260aab4def3" containerName="revoke-keys"
Oct 31 12:53:32.515903 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:32.514958    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="53231544-cdae-4031-a635-1260aab4def3" containerName="revoke-keys"
Oct 31 12:53:32.515903 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:32.514968    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="0070ea70-06e9-4733-8d11-ecb4cb993ff7" containerName="istio-proxy"
Oct 31 12:53:32.515903 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:32.514977    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="20fa8b33-c9e6-4fb0-9f3c-c8254dece2f6" containerName="istio-proxy"
Oct 31 12:53:32.515903 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:32.514985    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea38d319-4637-47c0-bf7a-125b61dac2fe" containerName="istio-proxy"
Oct 31 12:53:32.515903 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:32.514996    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="20fa8b33-c9e6-4fb0-9f3c-c8254dece2f6" containerName="istio-proxy"
Oct 31 12:53:32.515903 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:32.515007    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="41d07641-1c30-4e59-b402-fd7505e781ba" containerName="istio-proxy"
Oct 31 12:53:32.518615 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:32.518598    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d086f3-58b5756f6f-m4jx8"
Oct 31 12:53:32.527623 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:32.527598    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-d086f3-serving-cert\""
Oct 31 12:53:32.535936 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:32.535904    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7971b7e-6267-4ce1-ac72-38be9977a7b4-openshift-service-ca\") pod \"maas-api-e2e-inf-a-d086f3-58b5756f6f-m4jx8\" (UID: \"e7971b7e-6267-4ce1-ac72-38be9977a7b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d086f3-58b5756f6f-m4jx8"
Oct 31 12:53:32.536064 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:32.536046    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e7971b7e-6267-4ce1-ac72-38be9977a7b4-maas-api-tls\") pod \"maas-api-e2e-inf-a-d086f3-58b5756f6f-m4jx8\" (UID: \"e7971b7e-6267-4ce1-ac72-38be9977a7b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d086f3-58b5756f6f-m4jx8"
Oct 31 12:53:32.536135 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:32.536122    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dr926\" (UniqueName: \"kubernetes.io/projected/e7971b7e-6267-4ce1-ac72-38be9977a7b4-kube-api-access-dr926\") pod \"maas-api-e2e-inf-a-d086f3-58b5756f6f-m4jx8\" (UID: \"e7971b7e-6267-4ce1-ac72-38be9977a7b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d086f3-58b5756f6f-m4jx8"
Oct 31 12:53:32.536279 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:32.536254    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7971b7e-6267-4ce1-ac72-38be9977a7b4-metrics-certs\") pod \"maas-api-e2e-inf-a-d086f3-58b5756f6f-m4jx8\" (UID: \"e7971b7e-6267-4ce1-ac72-38be9977a7b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d086f3-58b5756f6f-m4jx8"
Oct 31 12:53:32.545324 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:32.545298    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d086f3-58b5756f6f-m4jx8"]
Oct 31 12:53:32.759609 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:32.757526    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7971b7e-6267-4ce1-ac72-38be9977a7b4-openshift-service-ca\") pod \"maas-api-e2e-inf-a-d086f3-58b5756f6f-m4jx8\" (UID: \"e7971b7e-6267-4ce1-ac72-38be9977a7b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d086f3-58b5756f6f-m4jx8"
Oct 31 12:53:32.759609 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:32.757678    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e7971b7e-6267-4ce1-ac72-38be9977a7b4-maas-api-tls\") pod \"maas-api-e2e-inf-a-d086f3-58b5756f6f-m4jx8\" (UID: \"e7971b7e-6267-4ce1-ac72-38be9977a7b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d086f3-58b5756f6f-m4jx8"
Oct 31 12:53:32.759609 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:32.757787    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dr926\" (UniqueName: \"kubernetes.io/projected/e7971b7e-6267-4ce1-ac72-38be9977a7b4-kube-api-access-dr926\") pod \"maas-api-e2e-inf-a-d086f3-58b5756f6f-m4jx8\" (UID: \"e7971b7e-6267-4ce1-ac72-38be9977a7b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d086f3-58b5756f6f-m4jx8"
Oct 31 12:53:32.783423 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:32.782669    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7971b7e-6267-4ce1-ac72-38be9977a7b4-metrics-certs\") pod \"maas-api-e2e-inf-a-d086f3-58b5756f6f-m4jx8\" (UID: \"e7971b7e-6267-4ce1-ac72-38be9977a7b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d086f3-58b5756f6f-m4jx8"
Oct 31 12:53:32.796045 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:32.795073    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7971b7e-6267-4ce1-ac72-38be9977a7b4-openshift-service-ca\") pod \"maas-api-e2e-inf-a-d086f3-58b5756f6f-m4jx8\" (UID: \"e7971b7e-6267-4ce1-ac72-38be9977a7b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d086f3-58b5756f6f-m4jx8"
Oct 31 12:53:32.809224 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:32.809166    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e7971b7e-6267-4ce1-ac72-38be9977a7b4-maas-api-tls\") pod \"maas-api-e2e-inf-a-d086f3-58b5756f6f-m4jx8\" (UID: \"e7971b7e-6267-4ce1-ac72-38be9977a7b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d086f3-58b5756f6f-m4jx8"
Oct 31 12:53:32.823144 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:32.823061    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dr926\" (UniqueName: \"kubernetes.io/projected/e7971b7e-6267-4ce1-ac72-38be9977a7b4-kube-api-access-dr926\") pod \"maas-api-e2e-inf-a-d086f3-58b5756f6f-m4jx8\" (UID: \"e7971b7e-6267-4ce1-ac72-38be9977a7b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d086f3-58b5756f6f-m4jx8"
Oct 31 12:53:32.824003 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:32.823974    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7971b7e-6267-4ce1-ac72-38be9977a7b4-metrics-certs\") pod \"maas-api-e2e-inf-a-d086f3-58b5756f6f-m4jx8\" (UID: \"e7971b7e-6267-4ce1-ac72-38be9977a7b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d086f3-58b5756f6f-m4jx8"
Oct 31 12:53:32.830552 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:32.830523    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-2a0ea4-584584b785-hpjz7"]
Oct 31 12:53:32.897884 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:32.897782    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d086f3-58b5756f6f-m4jx8"
Oct 31 12:53:32.898119 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:32.898095    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e43d8b-openshift-default-5578dcfc7f-5f9ss"
Oct 31 12:53:32.991525 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:32.989141    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0ba8ab3-65d2-4132-9896-80f1fb46b5c5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0ba8ab3-65d2-4132-9896-80f1fb46b5c5-workload-certs\") pod \"a0ba8ab3-65d2-4132-9896-80f1fb46b5c5\" (UID: \"a0ba8ab3-65d2-4132-9896-80f1fb46b5c5\") "
Oct 31 12:53:32.991525 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:32.989561    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0ba8ab3-65d2-4132-9896-80f1fb46b5c5-workload-certs" pod "a0ba8ab3-65d2-4132-9896-80f1fb46b5c5" (UID: "a0ba8ab3-65d2-4132-9896-80f1fb46b5c5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:53:32.991525 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:32.989849    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0ba8ab3-65d2-4132-9896-80f1fb46b5c5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0ba8ab3-65d2-4132-9896-80f1fb46b5c5-workload-socket\") pod \"a0ba8ab3-65d2-4132-9896-80f1fb46b5c5\" (UID: \"a0ba8ab3-65d2-4132-9896-80f1fb46b5c5\") "
Oct 31 12:53:32.991525 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:32.989990    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0ba8ab3-65d2-4132-9896-80f1fb46b5c5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0ba8ab3-65d2-4132-9896-80f1fb46b5c5-istio-data\") pod \"a0ba8ab3-65d2-4132-9896-80f1fb46b5c5\" (UID: \"a0ba8ab3-65d2-4132-9896-80f1fb46b5c5\") "
Oct 31 12:53:32.991525 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:32.990060    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0ba8ab3-65d2-4132-9896-80f1fb46b5c5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0ba8ab3-65d2-4132-9896-80f1fb46b5c5-credential-socket\") pod \"a0ba8ab3-65d2-4132-9896-80f1fb46b5c5\" (UID: \"a0ba8ab3-65d2-4132-9896-80f1fb46b5c5\") "
Oct 31 12:53:32.991525 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:32.990110    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0ba8ab3-65d2-4132-9896-80f1fb46b5c5-istio-token\" (UniqueName: \"kubernetes.io/projected/a0ba8ab3-65d2-4132-9896-80f1fb46b5c5-istio-token\") pod \"a0ba8ab3-65d2-4132-9896-80f1fb46b5c5\" (UID: \"a0ba8ab3-65d2-4132-9896-80f1fb46b5c5\") "
Oct 31 12:53:32.991525 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:32.990235    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a0ba8ab3-65d2-4132-9896-80f1fb46b5c5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0ba8ab3-65d2-4132-9896-80f1fb46b5c5-istio-podinfo\") pod \"a0ba8ab3-65d2-4132-9896-80f1fb46b5c5\" (UID: \"a0ba8ab3-65d2-4132-9896-80f1fb46b5c5\") "
Oct 31 12:53:32.991525 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:32.990275    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0ba8ab3-65d2-4132-9896-80f1fb46b5c5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0ba8ab3-65d2-4132-9896-80f1fb46b5c5-istiod-ca-cert\") pod \"a0ba8ab3-65d2-4132-9896-80f1fb46b5c5\" (UID: \"a0ba8ab3-65d2-4132-9896-80f1fb46b5c5\") "
Oct 31 12:53:32.991525 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:32.990321    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0ba8ab3-65d2-4132-9896-80f1fb46b5c5-kube-api-access-cg6vs\" (UniqueName: \"kubernetes.io/projected/a0ba8ab3-65d2-4132-9896-80f1fb46b5c5-kube-api-access-cg6vs\") pod \"a0ba8ab3-65d2-4132-9896-80f1fb46b5c5\" (UID: \"a0ba8ab3-65d2-4132-9896-80f1fb46b5c5\") "
Oct 31 12:53:32.991525 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:32.990362    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0ba8ab3-65d2-4132-9896-80f1fb46b5c5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0ba8ab3-65d2-4132-9896-80f1fb46b5c5-istio-envoy\") pod \"a0ba8ab3-65d2-4132-9896-80f1fb46b5c5\" (UID: \"a0ba8ab3-65d2-4132-9896-80f1fb46b5c5\") "
Oct 31 12:53:32.991525 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:32.991136    5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0ba8ab3-65d2-4132-9896-80f1fb46b5c5-workload-certs\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:33.000622 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:33.000166    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0ba8ab3-65d2-4132-9896-80f1fb46b5c5-kube-api-access-cg6vs" pod "a0ba8ab3-65d2-4132-9896-80f1fb46b5c5" (UID: "a0ba8ab3-65d2-4132-9896-80f1fb46b5c5"). InnerVolumeSpecName "kube-api-access-cg6vs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:53:33.002272 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:33.002241    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0ba8ab3-65d2-4132-9896-80f1fb46b5c5-istiod-ca-cert" pod "a0ba8ab3-65d2-4132-9896-80f1fb46b5c5" (UID: "a0ba8ab3-65d2-4132-9896-80f1fb46b5c5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:53:33.002573 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:33.002548    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0ba8ab3-65d2-4132-9896-80f1fb46b5c5-istio-data" pod "a0ba8ab3-65d2-4132-9896-80f1fb46b5c5" (UID: "a0ba8ab3-65d2-4132-9896-80f1fb46b5c5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:53:33.003086 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:33.003061    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0ba8ab3-65d2-4132-9896-80f1fb46b5c5-workload-socket" pod "a0ba8ab3-65d2-4132-9896-80f1fb46b5c5" (UID: "a0ba8ab3-65d2-4132-9896-80f1fb46b5c5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:53:33.003182 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:33.003162    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0ba8ab3-65d2-4132-9896-80f1fb46b5c5-credential-socket" pod "a0ba8ab3-65d2-4132-9896-80f1fb46b5c5" (UID: "a0ba8ab3-65d2-4132-9896-80f1fb46b5c5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:53:33.004038 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:33.004001    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0ba8ab3-65d2-4132-9896-80f1fb46b5c5-istio-envoy" pod "a0ba8ab3-65d2-4132-9896-80f1fb46b5c5" (UID: "a0ba8ab3-65d2-4132-9896-80f1fb46b5c5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:53:33.004577 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:33.004543    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a0ba8ab3-65d2-4132-9896-80f1fb46b5c5-istio-podinfo" pod "a0ba8ab3-65d2-4132-9896-80f1fb46b5c5" (UID: "a0ba8ab3-65d2-4132-9896-80f1fb46b5c5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 12:53:33.006567 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:33.005921    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0ba8ab3-65d2-4132-9896-80f1fb46b5c5-istio-token" pod "a0ba8ab3-65d2-4132-9896-80f1fb46b5c5" (UID: "a0ba8ab3-65d2-4132-9896-80f1fb46b5c5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:53:33.092396 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:33.092364    5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0ba8ab3-65d2-4132-9896-80f1fb46b5c5-istio-podinfo\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:33.092396 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:33.092395    5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0ba8ab3-65d2-4132-9896-80f1fb46b5c5-istiod-ca-cert\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:33.092649 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:33.092406    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cg6vs\" (UniqueName: \"kubernetes.io/projected/a0ba8ab3-65d2-4132-9896-80f1fb46b5c5-kube-api-access-cg6vs\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:33.092649 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:33.092416    5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0ba8ab3-65d2-4132-9896-80f1fb46b5c5-istio-envoy\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:33.092649 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:33.092425    5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0ba8ab3-65d2-4132-9896-80f1fb46b5c5-workload-socket\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:33.092649 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:33.092434    5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0ba8ab3-65d2-4132-9896-80f1fb46b5c5-istio-data\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:33.092649 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:33.092443    5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0ba8ab3-65d2-4132-9896-80f1fb46b5c5-credential-socket\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:33.092649 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:33.092452    5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0ba8ab3-65d2-4132-9896-80f1fb46b5c5-istio-token\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:33.255747 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:33.248166    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d086f3-58b5756f6f-m4jx8"]
Oct 31 12:53:33.359514 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:53:33.351698    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7971b7e_6267_4ce1_ac72_38be9977a7b4.slice/crio-91c6cc8bce0e150c82ee97f0e25e45f1467fa52263a0af4fdca5de14666d2043 WatchSource:0}: Error finding container 91c6cc8bce0e150c82ee97f0e25e45f1467fa52263a0af4fdca5de14666d2043: Status 404 returned error can't find the container with id 91c6cc8bce0e150c82ee97f0e25e45f1467fa52263a0af4fdca5de14666d2043
Oct 31 12:53:33.385513 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:33.379288    5060 generic.go:356] "Generic (PLEG): container finished" podID="a0ba8ab3-65d2-4132-9896-80f1fb46b5c5" containerID="643c50baeec7939f03f59f571525c956c954acc4db8b464eb73930c894d69b75" exitCode=0
Oct 31 12:53:33.385513 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:33.379381    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e43d8b-openshift-default-5578dcfc7f-5f9ss" event={"ID":"a0ba8ab3-65d2-4132-9896-80f1fb46b5c5","Type":"ContainerDied","Data":"643c50baeec7939f03f59f571525c956c954acc4db8b464eb73930c894d69b75"}
Oct 31 12:53:33.385513 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:33.379434    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e43d8b-openshift-default-5578dcfc7f-5f9ss" event={"ID":"a0ba8ab3-65d2-4132-9896-80f1fb46b5c5","Type":"ContainerDied","Data":"2b6b4095ec442fb7975d5b4fb0c6e4e415f999034ec793b89ea631ebaaf571d1"}
Oct 31 12:53:33.385513 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:33.379463    5060 scope.go:122] "RemoveContainer" containerID="643c50baeec7939f03f59f571525c956c954acc4db8b464eb73930c894d69b75"
Oct 31 12:53:33.385513 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:33.379663    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e43d8b-openshift-default-5578dcfc7f-5f9ss"
Oct 31 12:53:33.573202 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:33.573164    5060 scope.go:122] "RemoveContainer" containerID="643c50baeec7939f03f59f571525c956c954acc4db8b464eb73930c894d69b75"
Oct 31 12:53:33.573631 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:33.573317    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-e43d8b-openshift-default-5578dcfc7f-5f9ss"]
Oct 31 12:53:33.574662 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:53:33.574183    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"643c50baeec7939f03f59f571525c956c954acc4db8b464eb73930c894d69b75\": container with ID starting with 643c50baeec7939f03f59f571525c956c954acc4db8b464eb73930c894d69b75 not found: ID does not exist" containerID="643c50baeec7939f03f59f571525c956c954acc4db8b464eb73930c894d69b75"
Oct 31 12:53:33.574662 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:33.574222    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"643c50baeec7939f03f59f571525c956c954acc4db8b464eb73930c894d69b75"} err="failed to get container status \"643c50baeec7939f03f59f571525c956c954acc4db8b464eb73930c894d69b75\": rpc error: code = NotFound desc = could not find container \"643c50baeec7939f03f59f571525c956c954acc4db8b464eb73930c894d69b75\": container with ID starting with 643c50baeec7939f03f59f571525c956c954acc4db8b464eb73930c894d69b75 not found: ID does not exist"
Oct 31 12:53:33.581002 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:33.580970    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-e43d8b-openshift-default-5578dcfc7f-5f9ss"]
Oct 31 12:53:34.392002 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:34.391949    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d086f3-58b5756f6f-m4jx8" event={"ID":"e7971b7e-6267-4ce1-ac72-38be9977a7b4","Type":"ContainerStarted","Data":"810ebd8e03ac0a6c7cf27d578dd6cfc5e90a378312a9ea1517ecd99441b26882"}
Oct 31 12:53:34.392559 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:34.392063    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d086f3-58b5756f6f-m4jx8"
Oct 31 12:53:34.392559 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:34.392087    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d086f3-58b5756f6f-m4jx8" event={"ID":"e7971b7e-6267-4ce1-ac72-38be9977a7b4","Type":"ContainerStarted","Data":"91c6cc8bce0e150c82ee97f0e25e45f1467fa52263a0af4fdca5de14666d2043"}
Oct 31 12:53:34.435513 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:34.425815    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d086f3-58b5756f6f-m4jx8" podStartSLOduration=1.9816099870000001 podStartE2EDuration="2.425794936s" totalImagesPullingTime="444.184949ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:53:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:53:34.422454982 +0000 UTC m=+1577.525522696" watchObservedRunningTime="2026-10-31 12:53:34.425794936 +0000 UTC m=+1577.528862650"
Oct 31 12:53:35.194525 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:35.188311    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0ba8ab3-65d2-4132-9896-80f1fb46b5c5" path="/var/lib/kubelet/pods/a0ba8ab3-65d2-4132-9896-80f1fb46b5c5/volumes"
Oct 31 12:53:36.071940 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:36.071901    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-1a7d0e-69d4d6fb9f-7r75d"]
Oct 31 12:53:37.285839 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:37.285807    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9b62a0-8445958dbb-99wdf"
Oct 31 12:53:37.831421 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:37.831365    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-2a0ea4-584584b785-hpjz7" podUID="ecb5bb94-76fd-43a2-a49e-b1b0c8320cb2" containerName="payload-processing" containerID="cri-o://c5eb8feef072ffdf4535618d202840acdaa52297d180229a81cf5a4c19036d8f" gracePeriod=25
Oct 31 12:53:38.226763 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:38.226730    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-2a0ea4-584584b785-hpjz7"
Oct 31 12:53:38.369855 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:38.369818    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ecb5bb94-76fd-43a2-a49e-b1b0c8320cb2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecb5bb94-76fd-43a2-a49e-b1b0c8320cb2-plugins-config-volume\") pod \"ecb5bb94-76fd-43a2-a49e-b1b0c8320cb2\" (UID: \"ecb5bb94-76fd-43a2-a49e-b1b0c8320cb2\") "
Oct 31 12:53:38.370232 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:38.370060    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ecb5bb94-76fd-43a2-a49e-b1b0c8320cb2-kube-api-access-wcbzm\" (UniqueName: \"kubernetes.io/projected/ecb5bb94-76fd-43a2-a49e-b1b0c8320cb2-kube-api-access-wcbzm\") pod \"ecb5bb94-76fd-43a2-a49e-b1b0c8320cb2\" (UID: \"ecb5bb94-76fd-43a2-a49e-b1b0c8320cb2\") "
Oct 31 12:53:38.370298 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:38.370276    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecb5bb94-76fd-43a2-a49e-b1b0c8320cb2-plugins-config-volume" pod "ecb5bb94-76fd-43a2-a49e-b1b0c8320cb2" (UID: "ecb5bb94-76fd-43a2-a49e-b1b0c8320cb2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:53:38.370787 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:38.370764    5060 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecb5bb94-76fd-43a2-a49e-b1b0c8320cb2-plugins-config-volume\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:38.372422 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:38.372396    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecb5bb94-76fd-43a2-a49e-b1b0c8320cb2-kube-api-access-wcbzm" pod "ecb5bb94-76fd-43a2-a49e-b1b0c8320cb2" (UID: "ecb5bb94-76fd-43a2-a49e-b1b0c8320cb2"). InnerVolumeSpecName "kube-api-access-wcbzm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:53:38.472300 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:38.472227    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wcbzm\" (UniqueName: \"kubernetes.io/projected/ecb5bb94-76fd-43a2-a49e-b1b0c8320cb2-kube-api-access-wcbzm\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:38.609485 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:38.609451    5060 generic.go:356] "Generic (PLEG): container finished" podID="ecb5bb94-76fd-43a2-a49e-b1b0c8320cb2" containerID="c5eb8feef072ffdf4535618d202840acdaa52297d180229a81cf5a4c19036d8f" exitCode=0
Oct 31 12:53:38.609695 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:38.609506    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-2a0ea4-584584b785-hpjz7"
Oct 31 12:53:38.609695 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:38.609552    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-2a0ea4-584584b785-hpjz7" event={"ID":"ecb5bb94-76fd-43a2-a49e-b1b0c8320cb2","Type":"ContainerDied","Data":"c5eb8feef072ffdf4535618d202840acdaa52297d180229a81cf5a4c19036d8f"}
Oct 31 12:53:38.609695 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:38.609614    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-2a0ea4-584584b785-hpjz7" event={"ID":"ecb5bb94-76fd-43a2-a49e-b1b0c8320cb2","Type":"ContainerDied","Data":"c120f5b32341ad44668c6b16e5cb27515a24bcf1541a4d1b1ed591840e76b2fd"}
Oct 31 12:53:38.609695 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:38.609645    5060 scope.go:122] "RemoveContainer" containerID="c5eb8feef072ffdf4535618d202840acdaa52297d180229a81cf5a4c19036d8f"
Oct 31 12:53:38.622450 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:38.622434    5060 scope.go:122] "RemoveContainer" containerID="c5eb8feef072ffdf4535618d202840acdaa52297d180229a81cf5a4c19036d8f"
Oct 31 12:53:38.622725 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:53:38.622707    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5eb8feef072ffdf4535618d202840acdaa52297d180229a81cf5a4c19036d8f\": container with ID starting with c5eb8feef072ffdf4535618d202840acdaa52297d180229a81cf5a4c19036d8f not found: ID does not exist" containerID="c5eb8feef072ffdf4535618d202840acdaa52297d180229a81cf5a4c19036d8f"
Oct 31 12:53:38.622784 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:38.622732    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5eb8feef072ffdf4535618d202840acdaa52297d180229a81cf5a4c19036d8f"} err="failed to get container status \"c5eb8feef072ffdf4535618d202840acdaa52297d180229a81cf5a4c19036d8f\": rpc error: code = NotFound desc = could not find container \"c5eb8feef072ffdf4535618d202840acdaa52297d180229a81cf5a4c19036d8f\": container with ID starting with c5eb8feef072ffdf4535618d202840acdaa52297d180229a81cf5a4c19036d8f not found: ID does not exist"
Oct 31 12:53:38.647231 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:38.647196    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-2a0ea4-584584b785-hpjz7"]
Oct 31 12:53:38.657720 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:38.657688    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-2a0ea4-584584b785-hpjz7"]
Oct 31 12:53:39.050075 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:39.050038    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ecb5bb94-76fd-43a2-a49e-b1b0c8320cb2" path="/var/lib/kubelet/pods/ecb5bb94-76fd-43a2-a49e-b1b0c8320cb2/volumes"
Oct 31 12:53:40.344527 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:53:40.344463    5060 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1e6c7f5_d872_4549_b950_1e751b6374ab.slice/crio-conmon-15eb93212f666238582935bcfb7fb66c4776d563bb22273bc4eb80596e90659b.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod330523c8_06c9_42a2_ac3f_9dd4ad62c392.slice/crio-d190e7520eebe52b0c749ad77fb372857ff3f35218eb3dc8fea952d63280f3d9.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1e6c7f5_d872_4549_b950_1e751b6374ab.slice/crio-15eb93212f666238582935bcfb7fb66c4776d563bb22273bc4eb80596e90659b.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod330523c8_06c9_42a2_ac3f_9dd4ad62c392.slice/crio-conmon-d190e7520eebe52b0c749ad77fb372857ff3f35218eb3dc8fea952d63280f3d9.scope\": RecentStats: unable to find data in memory cache]"
Oct 31 12:53:40.524262 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:40.524228    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d086f3-58b5756f6f-m4jx8"
Oct 31 12:53:41.073186 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:41.073144    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-1a7d0e-69d4d6fb9f-7r75d" podUID="218f45e9-a5c5-4a5d-b4a6-365fa52b66ba" containerName="payload-processing" containerID="cri-o://a0b2c52b819b82b53933d0bdfacb36ab0288b4dad1019578aa73aec6059c43d5" gracePeriod=25
Oct 31 12:53:41.422334 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:41.422300    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-1a7d0e-69d4d6fb9f-7r75d"
Oct 31 12:53:41.593562 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:41.592559    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/218f45e9-a5c5-4a5d-b4a6-365fa52b66ba-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/218f45e9-a5c5-4a5d-b4a6-365fa52b66ba-plugins-config-volume\") pod \"218f45e9-a5c5-4a5d-b4a6-365fa52b66ba\" (UID: \"218f45e9-a5c5-4a5d-b4a6-365fa52b66ba\") "
Oct 31 12:53:41.593562 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:41.592724    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/218f45e9-a5c5-4a5d-b4a6-365fa52b66ba-kube-api-access-nd58k\" (UniqueName: \"kubernetes.io/projected/218f45e9-a5c5-4a5d-b4a6-365fa52b66ba-kube-api-access-nd58k\") pod \"218f45e9-a5c5-4a5d-b4a6-365fa52b66ba\" (UID: \"218f45e9-a5c5-4a5d-b4a6-365fa52b66ba\") "
Oct 31 12:53:41.601208 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:41.598181    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/218f45e9-a5c5-4a5d-b4a6-365fa52b66ba-plugins-config-volume" pod "218f45e9-a5c5-4a5d-b4a6-365fa52b66ba" (UID: "218f45e9-a5c5-4a5d-b4a6-365fa52b66ba"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:53:41.620514 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:41.611084    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/218f45e9-a5c5-4a5d-b4a6-365fa52b66ba-kube-api-access-nd58k" pod "218f45e9-a5c5-4a5d-b4a6-365fa52b66ba" (UID: "218f45e9-a5c5-4a5d-b4a6-365fa52b66ba"). InnerVolumeSpecName "kube-api-access-nd58k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:53:41.658575 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:41.658389    5060 generic.go:356] "Generic (PLEG): container finished" podID="218f45e9-a5c5-4a5d-b4a6-365fa52b66ba" containerID="a0b2c52b819b82b53933d0bdfacb36ab0288b4dad1019578aa73aec6059c43d5" exitCode=0
Oct 31 12:53:41.658785 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:41.658562    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-1a7d0e-69d4d6fb9f-7r75d"
Oct 31 12:53:41.658935 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:41.658613    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-1a7d0e-69d4d6fb9f-7r75d" event={"ID":"218f45e9-a5c5-4a5d-b4a6-365fa52b66ba","Type":"ContainerDied","Data":"a0b2c52b819b82b53933d0bdfacb36ab0288b4dad1019578aa73aec6059c43d5"}
Oct 31 12:53:41.659098 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:41.659074    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-1a7d0e-69d4d6fb9f-7r75d" event={"ID":"218f45e9-a5c5-4a5d-b4a6-365fa52b66ba","Type":"ContainerDied","Data":"0f3624adfa8763c2d916e97c2e8a9fcf91a533ed13e9cd6e0a8cd3dbf0b85b84"}
Oct 31 12:53:41.659731 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:41.659194    5060 scope.go:122] "RemoveContainer" containerID="a0b2c52b819b82b53933d0bdfacb36ab0288b4dad1019578aa73aec6059c43d5"
Oct 31 12:53:41.675360 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:41.675339    5060 scope.go:122] "RemoveContainer" containerID="a0b2c52b819b82b53933d0bdfacb36ab0288b4dad1019578aa73aec6059c43d5"
Oct 31 12:53:41.675730 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:53:41.675708    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0b2c52b819b82b53933d0bdfacb36ab0288b4dad1019578aa73aec6059c43d5\": container with ID starting with a0b2c52b819b82b53933d0bdfacb36ab0288b4dad1019578aa73aec6059c43d5 not found: ID does not exist" containerID="a0b2c52b819b82b53933d0bdfacb36ab0288b4dad1019578aa73aec6059c43d5"
Oct 31 12:53:41.675802 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:41.675740    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0b2c52b819b82b53933d0bdfacb36ab0288b4dad1019578aa73aec6059c43d5"} err="failed to get container status \"a0b2c52b819b82b53933d0bdfacb36ab0288b4dad1019578aa73aec6059c43d5\": rpc error: code = NotFound desc = could not find container \"a0b2c52b819b82b53933d0bdfacb36ab0288b4dad1019578aa73aec6059c43d5\": container with ID starting with a0b2c52b819b82b53933d0bdfacb36ab0288b4dad1019578aa73aec6059c43d5 not found: ID does not exist"
Oct 31 12:53:41.693557 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:41.693523    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-1a7d0e-69d4d6fb9f-7r75d"]
Oct 31 12:53:41.695828 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:41.695800    5060 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/218f45e9-a5c5-4a5d-b4a6-365fa52b66ba-plugins-config-volume\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:41.695968 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:41.695831    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nd58k\" (UniqueName: \"kubernetes.io/projected/218f45e9-a5c5-4a5d-b4a6-365fa52b66ba-kube-api-access-nd58k\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:41.698636 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:41.698612    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-1a7d0e-69d4d6fb9f-7r75d"]
Oct 31 12:53:43.054088 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:43.054045    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="218f45e9-a5c5-4a5d-b4a6-365fa52b66ba" path="/var/lib/kubelet/pods/218f45e9-a5c5-4a5d-b4a6-365fa52b66ba/volumes"
Oct 31 12:53:43.313636 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:43.313545    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e98df8-wbs75"]
Oct 31 12:53:43.334269 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:43.334234    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e98df8-wbs75"]
Oct 31 12:53:44.354204 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:44.354162    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-85ce08-vl7tk"]
Oct 31 12:53:44.355415 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:44.355389    5060 cpu_manager.go:414] "containerMap: removing container" podUID="ecb5bb94-76fd-43a2-a49e-b1b0c8320cb2" containerName="payload-processing"
Oct 31 12:53:44.355537 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:44.355419    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecb5bb94-76fd-43a2-a49e-b1b0c8320cb2" containerName="payload-processing"
Oct 31 12:53:44.355537 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:44.355449    5060 cpu_manager.go:414] "containerMap: removing container" podUID="218f45e9-a5c5-4a5d-b4a6-365fa52b66ba" containerName="payload-processing"
Oct 31 12:53:44.355537 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:44.355458    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="218f45e9-a5c5-4a5d-b4a6-365fa52b66ba" containerName="payload-processing"
Oct 31 12:53:44.355537 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:44.355519    5060 cpu_manager.go:414] "containerMap: removing container" podUID="a0ba8ab3-65d2-4132-9896-80f1fb46b5c5" containerName="istio-proxy"
Oct 31 12:53:44.355537 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:44.355528    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0ba8ab3-65d2-4132-9896-80f1fb46b5c5" containerName="istio-proxy"
Oct 31 12:53:44.355732 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:44.355718    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="218f45e9-a5c5-4a5d-b4a6-365fa52b66ba" containerName="payload-processing"
Oct 31 12:53:44.355769 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:44.355742    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0ba8ab3-65d2-4132-9896-80f1fb46b5c5" containerName="istio-proxy"
Oct 31 12:53:44.355769 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:44.355765    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="ecb5bb94-76fd-43a2-a49e-b1b0c8320cb2" containerName="payload-processing"
Oct 31 12:53:44.364313 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:44.364278    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-85ce08-vl7tk"
Oct 31 12:53:44.367003 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:44.366973    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-85ce08-vl7tk"]
Oct 31 12:53:44.370513 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:44.370474    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 12:53:44.388119 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:44.382928    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7wsmb\""
Oct 31 12:53:44.433373 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:44.433338    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5017d07-e40d-45e5-8a08-1d1866bc0d54-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-85ce08-vl7tk\" (UID: \"b5017d07-e40d-45e5-8a08-1d1866bc0d54\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-85ce08-vl7tk"
Oct 31 12:53:44.433671 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:44.433645    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5017d07-e40d-45e5-8a08-1d1866bc0d54-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-85ce08-vl7tk\" (UID: \"b5017d07-e40d-45e5-8a08-1d1866bc0d54\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-85ce08-vl7tk"
Oct 31 12:53:44.433791 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:44.433769    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6slx\" (UniqueName: \"kubernetes.io/projected/b5017d07-e40d-45e5-8a08-1d1866bc0d54-kube-api-access-c6slx\") pod \"maas-api-revoke-keys-e2e-worker-w1-85ce08-vl7tk\" (UID: \"b5017d07-e40d-45e5-8a08-1d1866bc0d54\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-85ce08-vl7tk"
Oct 31 12:53:44.535191 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:44.535155    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6slx\" (UniqueName: \"kubernetes.io/projected/b5017d07-e40d-45e5-8a08-1d1866bc0d54-kube-api-access-c6slx\") pod \"maas-api-revoke-keys-e2e-worker-w1-85ce08-vl7tk\" (UID: \"b5017d07-e40d-45e5-8a08-1d1866bc0d54\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-85ce08-vl7tk"
Oct 31 12:53:44.535388 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:44.535286    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5017d07-e40d-45e5-8a08-1d1866bc0d54-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-85ce08-vl7tk\" (UID: \"b5017d07-e40d-45e5-8a08-1d1866bc0d54\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-85ce08-vl7tk"
Oct 31 12:53:44.535470 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:44.535430    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5017d07-e40d-45e5-8a08-1d1866bc0d54-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-85ce08-vl7tk\" (UID: \"b5017d07-e40d-45e5-8a08-1d1866bc0d54\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-85ce08-vl7tk"
Oct 31 12:53:44.536175 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:44.536140    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5017d07-e40d-45e5-8a08-1d1866bc0d54-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-85ce08-vl7tk\" (UID: \"b5017d07-e40d-45e5-8a08-1d1866bc0d54\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-85ce08-vl7tk"
Oct 31 12:53:44.538161 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:44.538130    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5017d07-e40d-45e5-8a08-1d1866bc0d54-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-85ce08-vl7tk\" (UID: \"b5017d07-e40d-45e5-8a08-1d1866bc0d54\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-85ce08-vl7tk"
Oct 31 12:53:44.554396 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:44.554371    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6slx\" (UniqueName: \"kubernetes.io/projected/b5017d07-e40d-45e5-8a08-1d1866bc0d54-kube-api-access-c6slx\") pod \"maas-api-revoke-keys-e2e-worker-w1-85ce08-vl7tk\" (UID: \"b5017d07-e40d-45e5-8a08-1d1866bc0d54\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-85ce08-vl7tk"
Oct 31 12:53:44.577715 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:44.577678    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-1a7d0e-748bf567d5-j7bb7"]
Oct 31 12:53:44.578366 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:44.577963    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1a7d0e-748bf567d5-j7bb7" podUID="bcb6aa33-17fe-45ac-93a5-ff86419c2246" containerName="maas-api" containerID="cri-o://464aa43494598df7326debe86e667e43f5ad090d5bf196f25672da303e988785" gracePeriod=30
Oct 31 12:53:44.693620 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:44.693585    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-85ce08-vl7tk"
Oct 31 12:53:45.177538 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:45.177431    5060 generic.go:356] "Generic (PLEG): container finished" podID="bcb6aa33-17fe-45ac-93a5-ff86419c2246" containerID="464aa43494598df7326debe86e667e43f5ad090d5bf196f25672da303e988785" exitCode=0
Oct 31 12:53:45.197544 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:45.197473    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53231544-cdae-4031-a635-1260aab4def3" path="/var/lib/kubelet/pods/53231544-cdae-4031-a635-1260aab4def3/volumes"
Oct 31 12:53:45.198353 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:45.198268    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1a7d0e-bsjb6"]
Oct 31 12:53:45.198591 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:45.198576    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1a7d0e-bsjb6"]
Oct 31 12:53:45.198756 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:45.198736    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1a7d0e-748bf567d5-j7bb7" event={"ID":"bcb6aa33-17fe-45ac-93a5-ff86419c2246","Type":"ContainerDied","Data":"464aa43494598df7326debe86e667e43f5ad090d5bf196f25672da303e988785"}
Oct 31 12:53:45.424643 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:53:45.424612    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5017d07_e40d_45e5_8a08_1d1866bc0d54.slice/crio-b8401e886c3577b954b062c1a1f7938ab38aebe19b93ea868bc78a88d42d5d81 WatchSource:0}: Error finding container b8401e886c3577b954b062c1a1f7938ab38aebe19b93ea868bc78a88d42d5d81: Status 404 returned error can't find the container with id b8401e886c3577b954b062c1a1f7938ab38aebe19b93ea868bc78a88d42d5d81
Oct 31 12:53:45.425772 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:45.425748    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1a7d0e-748bf567d5-j7bb7"
Oct 31 12:53:45.429254 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:45.429228    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-85ce08-vl7tk"]
Oct 31 12:53:45.540142 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:45.540103    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bcb6aa33-17fe-45ac-93a5-ff86419c2246-kube-api-access-6xccp\" (UniqueName: \"kubernetes.io/projected/bcb6aa33-17fe-45ac-93a5-ff86419c2246-kube-api-access-6xccp\") pod \"bcb6aa33-17fe-45ac-93a5-ff86419c2246\" (UID: \"bcb6aa33-17fe-45ac-93a5-ff86419c2246\") "
Oct 31 12:53:45.540372 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:45.540323    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bcb6aa33-17fe-45ac-93a5-ff86419c2246-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bcb6aa33-17fe-45ac-93a5-ff86419c2246-maas-api-tls\") pod \"bcb6aa33-17fe-45ac-93a5-ff86419c2246\" (UID: \"bcb6aa33-17fe-45ac-93a5-ff86419c2246\") "
Oct 31 12:53:45.540541 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:45.540513    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bcb6aa33-17fe-45ac-93a5-ff86419c2246-metrics-certs\" (UniqueName: \"kubernetes.io/secret/bcb6aa33-17fe-45ac-93a5-ff86419c2246-metrics-certs\") pod \"bcb6aa33-17fe-45ac-93a5-ff86419c2246\" (UID: \"bcb6aa33-17fe-45ac-93a5-ff86419c2246\") "
Oct 31 12:53:45.540686 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:45.540668    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bcb6aa33-17fe-45ac-93a5-ff86419c2246-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bcb6aa33-17fe-45ac-93a5-ff86419c2246-openshift-service-ca\") pod \"bcb6aa33-17fe-45ac-93a5-ff86419c2246\" (UID: \"bcb6aa33-17fe-45ac-93a5-ff86419c2246\") "
Oct 31 12:53:45.541146 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:45.541116    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bcb6aa33-17fe-45ac-93a5-ff86419c2246-openshift-service-ca" pod "bcb6aa33-17fe-45ac-93a5-ff86419c2246" (UID: "bcb6aa33-17fe-45ac-93a5-ff86419c2246"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:53:45.541450 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:45.541424    5060 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bcb6aa33-17fe-45ac-93a5-ff86419c2246-openshift-service-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:45.542666 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:45.542633    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcb6aa33-17fe-45ac-93a5-ff86419c2246-kube-api-access-6xccp" pod "bcb6aa33-17fe-45ac-93a5-ff86419c2246" (UID: "bcb6aa33-17fe-45ac-93a5-ff86419c2246"). InnerVolumeSpecName "kube-api-access-6xccp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:53:45.542741 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:45.542714    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bcb6aa33-17fe-45ac-93a5-ff86419c2246-metrics-certs" pod "bcb6aa33-17fe-45ac-93a5-ff86419c2246" (UID: "bcb6aa33-17fe-45ac-93a5-ff86419c2246"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:53:45.543393 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:45.543371    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bcb6aa33-17fe-45ac-93a5-ff86419c2246-maas-api-tls" pod "bcb6aa33-17fe-45ac-93a5-ff86419c2246" (UID: "bcb6aa33-17fe-45ac-93a5-ff86419c2246"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:53:45.644000 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:45.643959    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xccp\" (UniqueName: \"kubernetes.io/projected/bcb6aa33-17fe-45ac-93a5-ff86419c2246-kube-api-access-6xccp\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:45.666507 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:45.644232    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bcb6aa33-17fe-45ac-93a5-ff86419c2246-maas-api-tls\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:45.666776 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:45.666747    5060 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bcb6aa33-17fe-45ac-93a5-ff86419c2246-metrics-certs\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:46.208189 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:46.208149    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-85ce08-vl7tk" event={"ID":"b5017d07-e40d-45e5-8a08-1d1866bc0d54","Type":"ContainerStarted","Data":"2fba625dcc73a5ede98b5f28994db8578f0f4e7593068688c7c250c220fee059"}
Oct 31 12:53:46.208454 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:46.208436    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-85ce08-vl7tk" event={"ID":"b5017d07-e40d-45e5-8a08-1d1866bc0d54","Type":"ContainerStarted","Data":"b8401e886c3577b954b062c1a1f7938ab38aebe19b93ea868bc78a88d42d5d81"}
Oct 31 12:53:46.211998 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:46.211968    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1a7d0e-748bf567d5-j7bb7" event={"ID":"bcb6aa33-17fe-45ac-93a5-ff86419c2246","Type":"ContainerDied","Data":"15da662a7fc426bcf64aae0262d9d26bd1ded3097c751a3dec6c86b57d2f4aa4"}
Oct 31 12:53:46.212274 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:46.212261    5060 scope.go:122] "RemoveContainer" containerID="464aa43494598df7326debe86e667e43f5ad090d5bf196f25672da303e988785"
Oct 31 12:53:46.212507 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:46.212211    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1a7d0e-748bf567d5-j7bb7"
Oct 31 12:53:46.286644 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:46.286584    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-85ce08-vl7tk" podStartSLOduration=2.286562992 podStartE2EDuration="2.286562992s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:53:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:53:46.242747576 +0000 UTC m=+1589.345815288" watchObservedRunningTime="2026-10-31 12:53:46.286562992 +0000 UTC m=+1589.389630707"
Oct 31 12:53:46.292610 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:46.292570    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-1a7d0e-748bf567d5-j7bb7"]
Oct 31 12:53:46.301451 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:46.301418    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-1a7d0e-748bf567d5-j7bb7"]
Oct 31 12:53:47.052553 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:47.052507    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bcb6aa33-17fe-45ac-93a5-ff86419c2246" path="/var/lib/kubelet/pods/bcb6aa33-17fe-45ac-93a5-ff86419c2246/volumes"
Oct 31 12:53:47.053057 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:47.053040    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fca0927c-616b-4766-8cb2-cae87d082f01" path="/var/lib/kubelet/pods/fca0927c-616b-4766-8cb2-cae87d082f01/volumes"
Oct 31 12:53:47.227147 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:47.227045    5060 generic.go:356] "Generic (PLEG): container finished" podID="b5017d07-e40d-45e5-8a08-1d1866bc0d54" containerID="2fba625dcc73a5ede98b5f28994db8578f0f4e7593068688c7c250c220fee059" exitCode=0
Oct 31 12:53:47.227331 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:47.227134    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-85ce08-vl7tk" event={"ID":"b5017d07-e40d-45e5-8a08-1d1866bc0d54","Type":"ContainerDied","Data":"2fba625dcc73a5ede98b5f28994db8578f0f4e7593068688c7c250c220fee059"}
Oct 31 12:53:47.428218 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:53:47.428176    5060 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod330523c8_06c9_42a2_ac3f_9dd4ad62c392.slice/crio-conmon-d190e7520eebe52b0c749ad77fb372857ff3f35218eb3dc8fea952d63280f3d9.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod330523c8_06c9_42a2_ac3f_9dd4ad62c392.slice/crio-d190e7520eebe52b0c749ad77fb372857ff3f35218eb3dc8fea952d63280f3d9.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1e6c7f5_d872_4549_b950_1e751b6374ab.slice/crio-15eb93212f666238582935bcfb7fb66c4776d563bb22273bc4eb80596e90659b.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1e6c7f5_d872_4549_b950_1e751b6374ab.slice/crio-conmon-15eb93212f666238582935bcfb7fb66c4776d563bb22273bc4eb80596e90659b.scope\": RecentStats: unable to find data in memory cache]"
Oct 31 12:53:48.422090 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:48.422064    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-85ce08-vl7tk"
Oct 31 12:53:48.509880 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:48.509800    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5017d07-e40d-45e5-8a08-1d1866bc0d54-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5017d07-e40d-45e5-8a08-1d1866bc0d54-maas-api-openshift-ca\") pod \"b5017d07-e40d-45e5-8a08-1d1866bc0d54\" (UID: \"b5017d07-e40d-45e5-8a08-1d1866bc0d54\") "
Oct 31 12:53:48.510043 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:48.510026    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5017d07-e40d-45e5-8a08-1d1866bc0d54-kube-api-access-c6slx\" (UniqueName: \"kubernetes.io/projected/b5017d07-e40d-45e5-8a08-1d1866bc0d54-kube-api-access-c6slx\") pod \"b5017d07-e40d-45e5-8a08-1d1866bc0d54\" (UID: \"b5017d07-e40d-45e5-8a08-1d1866bc0d54\") "
Oct 31 12:53:48.510107 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:48.510093    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5017d07-e40d-45e5-8a08-1d1866bc0d54-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5017d07-e40d-45e5-8a08-1d1866bc0d54-maas-api-xks-ca\") pod \"b5017d07-e40d-45e5-8a08-1d1866bc0d54\" (UID: \"b5017d07-e40d-45e5-8a08-1d1866bc0d54\") "
Oct 31 12:53:48.510173 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:48.510147    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5017d07-e40d-45e5-8a08-1d1866bc0d54-maas-api-openshift-ca" pod "b5017d07-e40d-45e5-8a08-1d1866bc0d54" (UID: "b5017d07-e40d-45e5-8a08-1d1866bc0d54"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:53:48.510584 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:48.510562    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5017d07-e40d-45e5-8a08-1d1866bc0d54-maas-api-openshift-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:48.512194 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:48.512165    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5017d07-e40d-45e5-8a08-1d1866bc0d54-kube-api-access-c6slx" pod "b5017d07-e40d-45e5-8a08-1d1866bc0d54" (UID: "b5017d07-e40d-45e5-8a08-1d1866bc0d54"). InnerVolumeSpecName "kube-api-access-c6slx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:53:48.512309 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:48.512165    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5017d07-e40d-45e5-8a08-1d1866bc0d54-maas-api-xks-ca" pod "b5017d07-e40d-45e5-8a08-1d1866bc0d54" (UID: "b5017d07-e40d-45e5-8a08-1d1866bc0d54"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:53:48.615768 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:48.615706    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c6slx\" (UniqueName: \"kubernetes.io/projected/b5017d07-e40d-45e5-8a08-1d1866bc0d54-kube-api-access-c6slx\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:48.615768 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:48.615744    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5017d07-e40d-45e5-8a08-1d1866bc0d54-maas-api-xks-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:49.249355 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:49.249326    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-85ce08-vl7tk"
Oct 31 12:53:49.249579 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:49.249325    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-85ce08-vl7tk" event={"ID":"b5017d07-e40d-45e5-8a08-1d1866bc0d54","Type":"ContainerDied","Data":"b8401e886c3577b954b062c1a1f7938ab38aebe19b93ea868bc78a88d42d5d81"}
Oct 31 12:53:49.249579 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:49.249447    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b8401e886c3577b954b062c1a1f7938ab38aebe19b93ea868bc78a88d42d5d81"
Oct 31 12:53:50.362769 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:53:50.362725    5060 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod330523c8_06c9_42a2_ac3f_9dd4ad62c392.slice/crio-conmon-d190e7520eebe52b0c749ad77fb372857ff3f35218eb3dc8fea952d63280f3d9.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1e6c7f5_d872_4549_b950_1e751b6374ab.slice/crio-conmon-15eb93212f666238582935bcfb7fb66c4776d563bb22273bc4eb80596e90659b.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod330523c8_06c9_42a2_ac3f_9dd4ad62c392.slice/crio-d190e7520eebe52b0c749ad77fb372857ff3f35218eb3dc8fea952d63280f3d9.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1e6c7f5_d872_4549_b950_1e751b6374ab.slice/crio-15eb93212f666238582935bcfb7fb66c4776d563bb22273bc4eb80596e90659b.scope\": RecentStats: unable to find data in memory cache]"
Oct 31 12:53:50.481219 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:50.479716    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-2c393f-openshift-default-6cb59888bf-xrph2"]
Oct 31 12:53:50.482541 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:50.482515    5060 cpu_manager.go:414] "containerMap: removing container" podUID="bcb6aa33-17fe-45ac-93a5-ff86419c2246" containerName="maas-api"
Oct 31 12:53:50.482541 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:50.482542    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bcb6aa33-17fe-45ac-93a5-ff86419c2246" containerName="maas-api"
Oct 31 12:53:50.482738 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:50.482577    5060 cpu_manager.go:414] "containerMap: removing container" podUID="b5017d07-e40d-45e5-8a08-1d1866bc0d54" containerName="revoke-keys"
Oct 31 12:53:50.482738 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:50.482585    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5017d07-e40d-45e5-8a08-1d1866bc0d54" containerName="revoke-keys"
Oct 31 12:53:50.482845 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:50.482782    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="bcb6aa33-17fe-45ac-93a5-ff86419c2246" containerName="maas-api"
Oct 31 12:53:50.482845 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:50.482798    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5017d07-e40d-45e5-8a08-1d1866bc0d54" containerName="revoke-keys"
Oct 31 12:53:50.487701 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:50.487678    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-2c393f-openshift-default-6cb59888bf-xrph2"
Oct 31 12:53:50.502725 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:50.502655    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-2c393f-openshift-default-dockercfg-9b6xg\""
Oct 31 12:53:50.508682 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:50.508655    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-2c393f-openshift-default-6cb59888bf-xrph2"]
Oct 31 12:53:50.575011 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:50.574972    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32a2e0ec-5eaa-4cce-a7c0-87512fe277ed-workload-socket\") pod \"e2e-inf-b-2c393f-openshift-default-6cb59888bf-xrph2\" (UID: \"32a2e0ec-5eaa-4cce-a7c0-87512fe277ed\") " pod="openshift-ingress/e2e-inf-b-2c393f-openshift-default-6cb59888bf-xrph2"
Oct 31 12:53:50.575193 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:50.575020    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32a2e0ec-5eaa-4cce-a7c0-87512fe277ed-workload-certs\") pod \"e2e-inf-b-2c393f-openshift-default-6cb59888bf-xrph2\" (UID: \"32a2e0ec-5eaa-4cce-a7c0-87512fe277ed\") " pod="openshift-ingress/e2e-inf-b-2c393f-openshift-default-6cb59888bf-xrph2"
Oct 31 12:53:50.575193 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:50.575148    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32a2e0ec-5eaa-4cce-a7c0-87512fe277ed-credential-socket\") pod \"e2e-inf-b-2c393f-openshift-default-6cb59888bf-xrph2\" (UID: \"32a2e0ec-5eaa-4cce-a7c0-87512fe277ed\") " pod="openshift-ingress/e2e-inf-b-2c393f-openshift-default-6cb59888bf-xrph2"
Oct 31 12:53:50.575193 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:50.575191    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32a2e0ec-5eaa-4cce-a7c0-87512fe277ed-istio-envoy\") pod \"e2e-inf-b-2c393f-openshift-default-6cb59888bf-xrph2\" (UID: \"32a2e0ec-5eaa-4cce-a7c0-87512fe277ed\") " pod="openshift-ingress/e2e-inf-b-2c393f-openshift-default-6cb59888bf-xrph2"
Oct 31 12:53:50.575373 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:50.575332    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32a2e0ec-5eaa-4cce-a7c0-87512fe277ed-istio-data\") pod \"e2e-inf-b-2c393f-openshift-default-6cb59888bf-xrph2\" (UID: \"32a2e0ec-5eaa-4cce-a7c0-87512fe277ed\") " pod="openshift-ingress/e2e-inf-b-2c393f-openshift-default-6cb59888bf-xrph2"
Oct 31 12:53:50.575434 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:50.575393    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32a2e0ec-5eaa-4cce-a7c0-87512fe277ed-istiod-ca-cert\") pod \"e2e-inf-b-2c393f-openshift-default-6cb59888bf-xrph2\" (UID: \"32a2e0ec-5eaa-4cce-a7c0-87512fe277ed\") " pod="openshift-ingress/e2e-inf-b-2c393f-openshift-default-6cb59888bf-xrph2"
Oct 31 12:53:50.575953 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:50.575564    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32a2e0ec-5eaa-4cce-a7c0-87512fe277ed-istio-token\") pod \"e2e-inf-b-2c393f-openshift-default-6cb59888bf-xrph2\" (UID: \"32a2e0ec-5eaa-4cce-a7c0-87512fe277ed\") " pod="openshift-ingress/e2e-inf-b-2c393f-openshift-default-6cb59888bf-xrph2"
Oct 31 12:53:50.575953 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:50.575591    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32a2e0ec-5eaa-4cce-a7c0-87512fe277ed-istio-podinfo\") pod \"e2e-inf-b-2c393f-openshift-default-6cb59888bf-xrph2\" (UID: \"32a2e0ec-5eaa-4cce-a7c0-87512fe277ed\") " pod="openshift-ingress/e2e-inf-b-2c393f-openshift-default-6cb59888bf-xrph2"
Oct 31 12:53:50.575953 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:50.575606    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27bll\" (UniqueName: \"kubernetes.io/projected/32a2e0ec-5eaa-4cce-a7c0-87512fe277ed-kube-api-access-27bll\") pod \"e2e-inf-b-2c393f-openshift-default-6cb59888bf-xrph2\" (UID: \"32a2e0ec-5eaa-4cce-a7c0-87512fe277ed\") " pod="openshift-ingress/e2e-inf-b-2c393f-openshift-default-6cb59888bf-xrph2"
Oct 31 12:53:50.631222 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:50.627964    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-fd988c-openshift-default-96bd4cd5c-g4dgc"]
Oct 31 12:53:50.633777 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:50.633752    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-fd988c-openshift-default-96bd4cd5c-g4dgc"
Oct 31 12:53:50.653126 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:50.651555    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-fd988c-openshift-default-dockercfg-hz4cr\""
Oct 31 12:53:50.662639 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:50.662611    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-fd988c-openshift-default-96bd4cd5c-g4dgc"]
Oct 31 12:53:50.677368 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:50.677335    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c93ff68-7e63-40b2-882b-8b2aed13e982-credential-socket\") pod \"e2e-inf-b-fd988c-openshift-default-96bd4cd5c-g4dgc\" (UID: \"0c93ff68-7e63-40b2-882b-8b2aed13e982\") " pod="openshift-ingress/e2e-inf-b-fd988c-openshift-default-96bd4cd5c-g4dgc"
Oct 31 12:53:50.677551 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:50.677393    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32a2e0ec-5eaa-4cce-a7c0-87512fe277ed-workload-socket\") pod \"e2e-inf-b-2c393f-openshift-default-6cb59888bf-xrph2\" (UID: \"32a2e0ec-5eaa-4cce-a7c0-87512fe277ed\") " pod="openshift-ingress/e2e-inf-b-2c393f-openshift-default-6cb59888bf-xrph2"
Oct 31 12:53:50.677551 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:50.677424    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32a2e0ec-5eaa-4cce-a7c0-87512fe277ed-workload-certs\") pod \"e2e-inf-b-2c393f-openshift-default-6cb59888bf-xrph2\" (UID: \"32a2e0ec-5eaa-4cce-a7c0-87512fe277ed\") " pod="openshift-ingress/e2e-inf-b-2c393f-openshift-default-6cb59888bf-xrph2"
Oct 31 12:53:50.677704 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:50.677567    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32a2e0ec-5eaa-4cce-a7c0-87512fe277ed-credential-socket\") pod \"e2e-inf-b-2c393f-openshift-default-6cb59888bf-xrph2\" (UID: \"32a2e0ec-5eaa-4cce-a7c0-87512fe277ed\") " pod="openshift-ingress/e2e-inf-b-2c393f-openshift-default-6cb59888bf-xrph2"
Oct 31 12:53:50.677704 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:50.677617    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c93ff68-7e63-40b2-882b-8b2aed13e982-workload-certs\") pod \"e2e-inf-b-fd988c-openshift-default-96bd4cd5c-g4dgc\" (UID: \"0c93ff68-7e63-40b2-882b-8b2aed13e982\") " pod="openshift-ingress/e2e-inf-b-fd988c-openshift-default-96bd4cd5c-g4dgc"
Oct 31 12:53:50.677704 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:50.677668    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32a2e0ec-5eaa-4cce-a7c0-87512fe277ed-istio-envoy\") pod \"e2e-inf-b-2c393f-openshift-default-6cb59888bf-xrph2\" (UID: \"32a2e0ec-5eaa-4cce-a7c0-87512fe277ed\") " pod="openshift-ingress/e2e-inf-b-2c393f-openshift-default-6cb59888bf-xrph2"
Oct 31 12:53:50.677868 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:50.677717    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c93ff68-7e63-40b2-882b-8b2aed13e982-istiod-ca-cert\") pod \"e2e-inf-b-fd988c-openshift-default-96bd4cd5c-g4dgc\" (UID: \"0c93ff68-7e63-40b2-882b-8b2aed13e982\") " pod="openshift-ingress/e2e-inf-b-fd988c-openshift-default-96bd4cd5c-g4dgc"
Oct 31 12:53:50.677868 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:50.677809    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32a2e0ec-5eaa-4cce-a7c0-87512fe277ed-istio-data\") pod \"e2e-inf-b-2c393f-openshift-default-6cb59888bf-xrph2\" (UID: \"32a2e0ec-5eaa-4cce-a7c0-87512fe277ed\") " pod="openshift-ingress/e2e-inf-b-2c393f-openshift-default-6cb59888bf-xrph2"
Oct 31 12:53:50.677868 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:50.677857    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32a2e0ec-5eaa-4cce-a7c0-87512fe277ed-istiod-ca-cert\") pod \"e2e-inf-b-2c393f-openshift-default-6cb59888bf-xrph2\" (UID: \"32a2e0ec-5eaa-4cce-a7c0-87512fe277ed\") " pod="openshift-ingress/e2e-inf-b-2c393f-openshift-default-6cb59888bf-xrph2"
Oct 31 12:53:50.678036 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:50.677899    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c93ff68-7e63-40b2-882b-8b2aed13e982-istio-podinfo\") pod \"e2e-inf-b-fd988c-openshift-default-96bd4cd5c-g4dgc\" (UID: \"0c93ff68-7e63-40b2-882b-8b2aed13e982\") " pod="openshift-ingress/e2e-inf-b-fd988c-openshift-default-96bd4cd5c-g4dgc"
Oct 31 12:53:50.678036 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:50.678027    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32a2e0ec-5eaa-4cce-a7c0-87512fe277ed-workload-socket\") pod \"e2e-inf-b-2c393f-openshift-default-6cb59888bf-xrph2\" (UID: \"32a2e0ec-5eaa-4cce-a7c0-87512fe277ed\") " pod="openshift-ingress/e2e-inf-b-2c393f-openshift-default-6cb59888bf-xrph2"
Oct 31 12:53:50.678149 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:50.678057    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c93ff68-7e63-40b2-882b-8b2aed13e982-istio-envoy\") pod \"e2e-inf-b-fd988c-openshift-default-96bd4cd5c-g4dgc\" (UID: \"0c93ff68-7e63-40b2-882b-8b2aed13e982\") " pod="openshift-ingress/e2e-inf-b-fd988c-openshift-default-96bd4cd5c-g4dgc"
Oct 31 12:53:50.678149 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:50.678086    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c93ff68-7e63-40b2-882b-8b2aed13e982-istio-data\") pod \"e2e-inf-b-fd988c-openshift-default-96bd4cd5c-g4dgc\" (UID: \"0c93ff68-7e63-40b2-882b-8b2aed13e982\") " pod="openshift-ingress/e2e-inf-b-fd988c-openshift-default-96bd4cd5c-g4dgc"
Oct 31 12:53:50.678149 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:50.678115    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c93ff68-7e63-40b2-882b-8b2aed13e982-istio-token\") pod \"e2e-inf-b-fd988c-openshift-default-96bd4cd5c-g4dgc\" (UID: \"0c93ff68-7e63-40b2-882b-8b2aed13e982\") " pod="openshift-ingress/e2e-inf-b-fd988c-openshift-default-96bd4cd5c-g4dgc"
Oct 31 12:53:50.678149 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:50.678141    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82qzw\" (UniqueName: \"kubernetes.io/projected/0c93ff68-7e63-40b2-882b-8b2aed13e982-kube-api-access-82qzw\") pod \"e2e-inf-b-fd988c-openshift-default-96bd4cd5c-g4dgc\" (UID: \"0c93ff68-7e63-40b2-882b-8b2aed13e982\") " pod="openshift-ingress/e2e-inf-b-fd988c-openshift-default-96bd4cd5c-g4dgc"
Oct 31 12:53:50.678368 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:50.678183    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32a2e0ec-5eaa-4cce-a7c0-87512fe277ed-istio-token\") pod \"e2e-inf-b-2c393f-openshift-default-6cb59888bf-xrph2\" (UID: \"32a2e0ec-5eaa-4cce-a7c0-87512fe277ed\") " pod="openshift-ingress/e2e-inf-b-2c393f-openshift-default-6cb59888bf-xrph2"
Oct 31 12:53:50.678368 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:50.678209    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32a2e0ec-5eaa-4cce-a7c0-87512fe277ed-credential-socket\") pod \"e2e-inf-b-2c393f-openshift-default-6cb59888bf-xrph2\" (UID: \"32a2e0ec-5eaa-4cce-a7c0-87512fe277ed\") " pod="openshift-ingress/e2e-inf-b-2c393f-openshift-default-6cb59888bf-xrph2"
Oct 31 12:53:50.678368 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:50.678213    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c93ff68-7e63-40b2-882b-8b2aed13e982-workload-socket\") pod \"e2e-inf-b-fd988c-openshift-default-96bd4cd5c-g4dgc\" (UID: \"0c93ff68-7e63-40b2-882b-8b2aed13e982\") " pod="openshift-ingress/e2e-inf-b-fd988c-openshift-default-96bd4cd5c-g4dgc"
Oct 31 12:53:50.678368 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:50.677856    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32a2e0ec-5eaa-4cce-a7c0-87512fe277ed-workload-certs\") pod \"e2e-inf-b-2c393f-openshift-default-6cb59888bf-xrph2\" (UID: \"32a2e0ec-5eaa-4cce-a7c0-87512fe277ed\") " pod="openshift-ingress/e2e-inf-b-2c393f-openshift-default-6cb59888bf-xrph2"
Oct 31 12:53:50.678368 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:50.678285    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32a2e0ec-5eaa-4cce-a7c0-87512fe277ed-istio-podinfo\") pod \"e2e-inf-b-2c393f-openshift-default-6cb59888bf-xrph2\" (UID: \"32a2e0ec-5eaa-4cce-a7c0-87512fe277ed\") " pod="openshift-ingress/e2e-inf-b-2c393f-openshift-default-6cb59888bf-xrph2"
Oct 31 12:53:50.678368 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:50.678314    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27bll\" (UniqueName: \"kubernetes.io/projected/32a2e0ec-5eaa-4cce-a7c0-87512fe277ed-kube-api-access-27bll\") pod \"e2e-inf-b-2c393f-openshift-default-6cb59888bf-xrph2\" (UID: \"32a2e0ec-5eaa-4cce-a7c0-87512fe277ed\") " pod="openshift-ingress/e2e-inf-b-2c393f-openshift-default-6cb59888bf-xrph2"
Oct 31 12:53:50.679267 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:50.679206    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32a2e0ec-5eaa-4cce-a7c0-87512fe277ed-istio-data\") pod \"e2e-inf-b-2c393f-openshift-default-6cb59888bf-xrph2\" (UID: \"32a2e0ec-5eaa-4cce-a7c0-87512fe277ed\") " pod="openshift-ingress/e2e-inf-b-2c393f-openshift-default-6cb59888bf-xrph2"
Oct 31 12:53:50.679870 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:50.679842    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32a2e0ec-5eaa-4cce-a7c0-87512fe277ed-istiod-ca-cert\") pod \"e2e-inf-b-2c393f-openshift-default-6cb59888bf-xrph2\" (UID: \"32a2e0ec-5eaa-4cce-a7c0-87512fe277ed\") " pod="openshift-ingress/e2e-inf-b-2c393f-openshift-default-6cb59888bf-xrph2"
Oct 31 12:53:50.687514 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:50.681352    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32a2e0ec-5eaa-4cce-a7c0-87512fe277ed-istio-podinfo\") pod \"e2e-inf-b-2c393f-openshift-default-6cb59888bf-xrph2\" (UID: \"32a2e0ec-5eaa-4cce-a7c0-87512fe277ed\") " pod="openshift-ingress/e2e-inf-b-2c393f-openshift-default-6cb59888bf-xrph2"
Oct 31 12:53:50.687514 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:50.681565    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32a2e0ec-5eaa-4cce-a7c0-87512fe277ed-istio-envoy\") pod \"e2e-inf-b-2c393f-openshift-default-6cb59888bf-xrph2\" (UID: \"32a2e0ec-5eaa-4cce-a7c0-87512fe277ed\") " pod="openshift-ingress/e2e-inf-b-2c393f-openshift-default-6cb59888bf-xrph2"
Oct 31 12:53:50.692183 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:50.692154    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27bll\" (UniqueName: \"kubernetes.io/projected/32a2e0ec-5eaa-4cce-a7c0-87512fe277ed-kube-api-access-27bll\") pod \"e2e-inf-b-2c393f-openshift-default-6cb59888bf-xrph2\" (UID: \"32a2e0ec-5eaa-4cce-a7c0-87512fe277ed\") " pod="openshift-ingress/e2e-inf-b-2c393f-openshift-default-6cb59888bf-xrph2"
Oct 31 12:53:50.693298 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:50.693274    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32a2e0ec-5eaa-4cce-a7c0-87512fe277ed-istio-token\") pod \"e2e-inf-b-2c393f-openshift-default-6cb59888bf-xrph2\" (UID: \"32a2e0ec-5eaa-4cce-a7c0-87512fe277ed\") " pod="openshift-ingress/e2e-inf-b-2c393f-openshift-default-6cb59888bf-xrph2"
Oct 31 12:53:50.813513 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:50.807203    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-2c393f-openshift-default-6cb59888bf-xrph2"
Oct 31 12:53:50.851511 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:50.819959    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c93ff68-7e63-40b2-882b-8b2aed13e982-workload-socket\") pod \"e2e-inf-b-fd988c-openshift-default-96bd4cd5c-g4dgc\" (UID: \"0c93ff68-7e63-40b2-882b-8b2aed13e982\") " pod="openshift-ingress/e2e-inf-b-fd988c-openshift-default-96bd4cd5c-g4dgc"
Oct 31 12:53:50.851511 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:50.820102    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c93ff68-7e63-40b2-882b-8b2aed13e982-credential-socket\") pod \"e2e-inf-b-fd988c-openshift-default-96bd4cd5c-g4dgc\" (UID: \"0c93ff68-7e63-40b2-882b-8b2aed13e982\") " pod="openshift-ingress/e2e-inf-b-fd988c-openshift-default-96bd4cd5c-g4dgc"
Oct 31 12:53:50.851511 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:50.820197    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c93ff68-7e63-40b2-882b-8b2aed13e982-workload-certs\") pod \"e2e-inf-b-fd988c-openshift-default-96bd4cd5c-g4dgc\" (UID: \"0c93ff68-7e63-40b2-882b-8b2aed13e982\") " pod="openshift-ingress/e2e-inf-b-fd988c-openshift-default-96bd4cd5c-g4dgc"
Oct 31 12:53:50.851511 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:50.820234    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c93ff68-7e63-40b2-882b-8b2aed13e982-istiod-ca-cert\") pod \"e2e-inf-b-fd988c-openshift-default-96bd4cd5c-g4dgc\" (UID: \"0c93ff68-7e63-40b2-882b-8b2aed13e982\") " pod="openshift-ingress/e2e-inf-b-fd988c-openshift-default-96bd4cd5c-g4dgc"
Oct 31 12:53:50.851511 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:50.820338    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c93ff68-7e63-40b2-882b-8b2aed13e982-istio-podinfo\") pod \"e2e-inf-b-fd988c-openshift-default-96bd4cd5c-g4dgc\" (UID: \"0c93ff68-7e63-40b2-882b-8b2aed13e982\") " pod="openshift-ingress/e2e-inf-b-fd988c-openshift-default-96bd4cd5c-g4dgc"
Oct 31 12:53:50.851511 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:50.820509    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c93ff68-7e63-40b2-882b-8b2aed13e982-istio-envoy\") pod \"e2e-inf-b-fd988c-openshift-default-96bd4cd5c-g4dgc\" (UID: \"0c93ff68-7e63-40b2-882b-8b2aed13e982\") " pod="openshift-ingress/e2e-inf-b-fd988c-openshift-default-96bd4cd5c-g4dgc"
Oct 31 12:53:50.851511 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:50.820537    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c93ff68-7e63-40b2-882b-8b2aed13e982-istio-data\") pod \"e2e-inf-b-fd988c-openshift-default-96bd4cd5c-g4dgc\" (UID: \"0c93ff68-7e63-40b2-882b-8b2aed13e982\") " pod="openshift-ingress/e2e-inf-b-fd988c-openshift-default-96bd4cd5c-g4dgc"
Oct 31 12:53:50.851511 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:50.820566    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c93ff68-7e63-40b2-882b-8b2aed13e982-istio-token\") pod \"e2e-inf-b-fd988c-openshift-default-96bd4cd5c-g4dgc\" (UID: \"0c93ff68-7e63-40b2-882b-8b2aed13e982\") " pod="openshift-ingress/e2e-inf-b-fd988c-openshift-default-96bd4cd5c-g4dgc"
Oct 31 12:53:50.851511 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:50.820593    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82qzw\" (UniqueName: \"kubernetes.io/projected/0c93ff68-7e63-40b2-882b-8b2aed13e982-kube-api-access-82qzw\") pod \"e2e-inf-b-fd988c-openshift-default-96bd4cd5c-g4dgc\" (UID: \"0c93ff68-7e63-40b2-882b-8b2aed13e982\") " pod="openshift-ingress/e2e-inf-b-fd988c-openshift-default-96bd4cd5c-g4dgc"
Oct 31 12:53:50.851511 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:50.821367    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c93ff68-7e63-40b2-882b-8b2aed13e982-workload-socket\") pod \"e2e-inf-b-fd988c-openshift-default-96bd4cd5c-g4dgc\" (UID: \"0c93ff68-7e63-40b2-882b-8b2aed13e982\") " pod="openshift-ingress/e2e-inf-b-fd988c-openshift-default-96bd4cd5c-g4dgc"
Oct 31 12:53:50.851511 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:50.821655    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c93ff68-7e63-40b2-882b-8b2aed13e982-credential-socket\") pod \"e2e-inf-b-fd988c-openshift-default-96bd4cd5c-g4dgc\" (UID: \"0c93ff68-7e63-40b2-882b-8b2aed13e982\") " pod="openshift-ingress/e2e-inf-b-fd988c-openshift-default-96bd4cd5c-g4dgc"
Oct 31 12:53:50.851511 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:50.821893    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c93ff68-7e63-40b2-882b-8b2aed13e982-workload-certs\") pod \"e2e-inf-b-fd988c-openshift-default-96bd4cd5c-g4dgc\" (UID: \"0c93ff68-7e63-40b2-882b-8b2aed13e982\") " pod="openshift-ingress/e2e-inf-b-fd988c-openshift-default-96bd4cd5c-g4dgc"
Oct 31 12:53:50.851511 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:50.832392    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c93ff68-7e63-40b2-882b-8b2aed13e982-istio-data\") pod \"e2e-inf-b-fd988c-openshift-default-96bd4cd5c-g4dgc\" (UID: \"0c93ff68-7e63-40b2-882b-8b2aed13e982\") " pod="openshift-ingress/e2e-inf-b-fd988c-openshift-default-96bd4cd5c-g4dgc"
Oct 31 12:53:50.852328 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:50.851809    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c93ff68-7e63-40b2-882b-8b2aed13e982-istio-envoy\") pod \"e2e-inf-b-fd988c-openshift-default-96bd4cd5c-g4dgc\" (UID: \"0c93ff68-7e63-40b2-882b-8b2aed13e982\") " pod="openshift-ingress/e2e-inf-b-fd988c-openshift-default-96bd4cd5c-g4dgc"
Oct 31 12:53:50.852646 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:50.852590    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c93ff68-7e63-40b2-882b-8b2aed13e982-istiod-ca-cert\") pod \"e2e-inf-b-fd988c-openshift-default-96bd4cd5c-g4dgc\" (UID: \"0c93ff68-7e63-40b2-882b-8b2aed13e982\") " pod="openshift-ingress/e2e-inf-b-fd988c-openshift-default-96bd4cd5c-g4dgc"
Oct 31 12:53:50.863694 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:50.862775    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c93ff68-7e63-40b2-882b-8b2aed13e982-istio-token\") pod \"e2e-inf-b-fd988c-openshift-default-96bd4cd5c-g4dgc\" (UID: \"0c93ff68-7e63-40b2-882b-8b2aed13e982\") " pod="openshift-ingress/e2e-inf-b-fd988c-openshift-default-96bd4cd5c-g4dgc"
Oct 31 12:53:50.871376 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:50.871347    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c93ff68-7e63-40b2-882b-8b2aed13e982-istio-podinfo\") pod \"e2e-inf-b-fd988c-openshift-default-96bd4cd5c-g4dgc\" (UID: \"0c93ff68-7e63-40b2-882b-8b2aed13e982\") " pod="openshift-ingress/e2e-inf-b-fd988c-openshift-default-96bd4cd5c-g4dgc"
Oct 31 12:53:50.891649 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:50.872169    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82qzw\" (UniqueName: \"kubernetes.io/projected/0c93ff68-7e63-40b2-882b-8b2aed13e982-kube-api-access-82qzw\") pod \"e2e-inf-b-fd988c-openshift-default-96bd4cd5c-g4dgc\" (UID: \"0c93ff68-7e63-40b2-882b-8b2aed13e982\") " pod="openshift-ingress/e2e-inf-b-fd988c-openshift-default-96bd4cd5c-g4dgc"
Oct 31 12:53:50.953091 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:50.952806    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-fd988c-openshift-default-96bd4cd5c-g4dgc"
Oct 31 12:53:51.532946 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:51.531683    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:53:51.532946 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:51.531785    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:53:51.532946 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:51.531829    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:53:51.534948 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:51.534901    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:53:51.535195 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:51.534989    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:53:51.535195 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:51.535029    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:53:51.554231 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:51.554202    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-fd988c-openshift-default-96bd4cd5c-g4dgc"]
Oct 31 12:53:51.564400 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:51.564369    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-2c393f-openshift-default-6cb59888bf-xrph2"]
Oct 31 12:53:52.293195 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:52.293150    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2c393f-openshift-default-6cb59888bf-xrph2" event={"ID":"32a2e0ec-5eaa-4cce-a7c0-87512fe277ed","Type":"ContainerStarted","Data":"7d49e71e687504e7a0bd13babd59f3c3abded0d6427501c69d03fa9f1cdad5e2"}
Oct 31 12:53:52.293407 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:52.293217    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2c393f-openshift-default-6cb59888bf-xrph2" event={"ID":"32a2e0ec-5eaa-4cce-a7c0-87512fe277ed","Type":"ContainerStarted","Data":"5ee2db7fc82a147691cf7870d1dceaecbac3f1b6e5bd9e70fcd6b19cecabac7a"}
Oct 31 12:53:52.295747 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:52.295715    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-fd988c-openshift-default-96bd4cd5c-g4dgc" event={"ID":"0c93ff68-7e63-40b2-882b-8b2aed13e982","Type":"ContainerStarted","Data":"55e1b15a513434f3d2d8350ad52e7342327fa7ab784312e346b4241a14ba93ee"}
Oct 31 12:53:52.295878 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:52.295767    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-fd988c-openshift-default-96bd4cd5c-g4dgc" event={"ID":"0c93ff68-7e63-40b2-882b-8b2aed13e982","Type":"ContainerStarted","Data":"a59d78aff00f2d7375c284ad9b63491a5e74ef8cdfb2b1834d9b3292f91a5dbb"}
Oct 31 12:53:52.336259 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:52.336215    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-2c393f-openshift-default-6cb59888bf-xrph2" podStartSLOduration=2.336199424 podStartE2EDuration="2.336199424s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:53:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:53:52.32895808 +0000 UTC m=+1595.432025791" watchObservedRunningTime="2026-10-31 12:53:52.336199424 +0000 UTC m=+1595.439267193"
Oct 31 12:53:52.378173 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:52.378059    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-fd988c-openshift-default-96bd4cd5c-g4dgc" podStartSLOduration=2.378036975 podStartE2EDuration="2.378036975s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:53:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:53:52.359998757 +0000 UTC m=+1595.463066469" watchObservedRunningTime="2026-10-31 12:53:52.378036975 +0000 UTC m=+1595.481104686"
Oct 31 12:53:52.813039 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:52.813002    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-2c393f-openshift-default-6cb59888bf-xrph2"
Oct 31 12:53:52.819652 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:52.819625    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-2c393f-openshift-default-6cb59888bf-xrph2"
Oct 31 12:53:52.954210 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:52.954171    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-fd988c-openshift-default-96bd4cd5c-g4dgc"
Oct 31 12:53:52.960135 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:52.960109    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-fd988c-openshift-default-96bd4cd5c-g4dgc"
Oct 31 12:53:53.347737 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:53.338147    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-fd988c-openshift-default-96bd4cd5c-g4dgc"
Oct 31 12:53:53.347737 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:53.338207    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-2c393f-openshift-default-6cb59888bf-xrph2"
Oct 31 12:53:53.350324 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:53.350246    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-2c393f-openshift-default-6cb59888bf-xrph2"
Oct 31 12:53:53.358753 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:53.358720    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-fd988c-openshift-default-96bd4cd5c-g4dgc"
Oct 31 12:53:53.977539 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:53.977450    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-e98df8-openshift-default-779d98479-vkrj6"]
Oct 31 12:53:53.977973 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:53.977757    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-e98df8-openshift-default-779d98479-vkrj6" podUID="0d62f8e4-b062-4dd4-a23d-0f66566c83da" containerName="istio-proxy" containerID="cri-o://75f2642303e354ccd77cddad60790638c3b1c480e432f7aad1fb90ca360b1992" gracePeriod=30
Oct 31 12:53:53.982979 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:53.982953    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-e98df8-openshift-default-779d98479-r2jbq"]
Oct 31 12:53:53.983189 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:53.983166    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-e98df8-openshift-default-779d98479-r2jbq" podUID="e08b08ff-4f40-474a-a103-2c499e999942" containerName="istio-proxy" containerID="cri-o://69e50444a6492290d3c0f1f8e259cce1808ee453214910de09e68331867cc777" gracePeriod=30
Oct 31 12:53:54.984231 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:54.984187    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-85ce08-5f656fc7f9-z5g9p"]
Oct 31 12:53:57.157452 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:57.156549    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b218d3-khq4b"]
Oct 31 12:53:57.242510 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:57.234782    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b218d3-khq4b"
Oct 31 12:53:57.251441 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:57.251087    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 12:53:57.273352 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:57.271811    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7wsmb\""
Oct 31 12:53:57.400514 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:57.389838    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7972aee8-dbdd-4c32-b46d-6cdf855aa53d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-b218d3-khq4b\" (UID: \"7972aee8-dbdd-4c32-b46d-6cdf855aa53d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b218d3-khq4b"
Oct 31 12:53:57.400514 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:57.389935    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7972aee8-dbdd-4c32-b46d-6cdf855aa53d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-b218d3-khq4b\" (UID: \"7972aee8-dbdd-4c32-b46d-6cdf855aa53d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b218d3-khq4b"
Oct 31 12:53:57.400514 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:57.390009    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wz7lt\" (UniqueName: \"kubernetes.io/projected/7972aee8-dbdd-4c32-b46d-6cdf855aa53d-kube-api-access-wz7lt\") pod \"maas-api-revoke-keys-e2e-worker-w3-b218d3-khq4b\" (UID: \"7972aee8-dbdd-4c32-b46d-6cdf855aa53d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b218d3-khq4b"
Oct 31 12:53:57.497448 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:57.496611    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wz7lt\" (UniqueName: \"kubernetes.io/projected/7972aee8-dbdd-4c32-b46d-6cdf855aa53d-kube-api-access-wz7lt\") pod \"maas-api-revoke-keys-e2e-worker-w3-b218d3-khq4b\" (UID: \"7972aee8-dbdd-4c32-b46d-6cdf855aa53d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b218d3-khq4b"
Oct 31 12:53:57.497448 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:57.496936    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7972aee8-dbdd-4c32-b46d-6cdf855aa53d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-b218d3-khq4b\" (UID: \"7972aee8-dbdd-4c32-b46d-6cdf855aa53d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b218d3-khq4b"
Oct 31 12:53:57.497448 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:57.497208    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7972aee8-dbdd-4c32-b46d-6cdf855aa53d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-b218d3-khq4b\" (UID: \"7972aee8-dbdd-4c32-b46d-6cdf855aa53d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b218d3-khq4b"
Oct 31 12:53:57.498211 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:57.498170    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7972aee8-dbdd-4c32-b46d-6cdf855aa53d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-b218d3-khq4b\" (UID: \"7972aee8-dbdd-4c32-b46d-6cdf855aa53d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b218d3-khq4b"
Oct 31 12:53:57.588513 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:57.579618    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b218d3-khq4b"]
Oct 31 12:53:57.626408 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:57.626375    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-fb2b1d-r8vf7"]
Oct 31 12:53:57.627632 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:57.627601    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7972aee8-dbdd-4c32-b46d-6cdf855aa53d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-b218d3-khq4b\" (UID: \"7972aee8-dbdd-4c32-b46d-6cdf855aa53d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b218d3-khq4b"
Oct 31 12:53:57.629429 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:57.628572    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wz7lt\" (UniqueName: \"kubernetes.io/projected/7972aee8-dbdd-4c32-b46d-6cdf855aa53d-kube-api-access-wz7lt\") pod \"maas-api-revoke-keys-e2e-worker-w3-b218d3-khq4b\" (UID: \"7972aee8-dbdd-4c32-b46d-6cdf855aa53d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b218d3-khq4b"
Oct 31 12:53:57.632412 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:57.632387    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-fb2b1d-r8vf7"
Oct 31 12:53:57.637090 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:57.637061    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-fb2b1d-r8vf7"]
Oct 31 12:53:57.655389 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:57.647735    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-71bc05-openshift-default-dd6bc8b77-b65mw"]
Oct 31 12:53:57.661095 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:57.660833    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-71bc05-openshift-default-dd6bc8b77-b65mw"
Oct 31 12:53:57.667238 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:57.667196    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-71bc05-openshift-default-dd6bc8b77-b65mw"]
Oct 31 12:53:57.675470 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:57.675444    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-71bc05-openshift-default-dockercfg-2cfjt\""
Oct 31 12:53:57.732367 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:57.732331    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85ab7b73-48a3-430a-b489-30985757d106-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-fb2b1d-r8vf7\" (UID: \"85ab7b73-48a3-430a-b489-30985757d106\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-fb2b1d-r8vf7"
Oct 31 12:53:57.732581 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:57.732510    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btcs6\" (UniqueName: \"kubernetes.io/projected/b3992303-31dd-484d-b3ad-3fececdf57fa-kube-api-access-btcs6\") pod \"e2e-inf-b-71bc05-openshift-default-dd6bc8b77-b65mw\" (UID: \"b3992303-31dd-484d-b3ad-3fececdf57fa\") " pod="openshift-ingress/e2e-inf-b-71bc05-openshift-default-dd6bc8b77-b65mw"
Oct 31 12:53:57.732581 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:57.732563    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85ab7b73-48a3-430a-b489-30985757d106-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-fb2b1d-r8vf7\" (UID: \"85ab7b73-48a3-430a-b489-30985757d106\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-fb2b1d-r8vf7"
Oct 31 12:53:57.732752 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:57.732733    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3992303-31dd-484d-b3ad-3fececdf57fa-istio-data\") pod \"e2e-inf-b-71bc05-openshift-default-dd6bc8b77-b65mw\" (UID: \"b3992303-31dd-484d-b3ad-3fececdf57fa\") " pod="openshift-ingress/e2e-inf-b-71bc05-openshift-default-dd6bc8b77-b65mw"
Oct 31 12:53:57.732868 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:57.732844    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khdc9\" (UniqueName: \"kubernetes.io/projected/85ab7b73-48a3-430a-b489-30985757d106-kube-api-access-khdc9\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-fb2b1d-r8vf7\" (UID: \"85ab7b73-48a3-430a-b489-30985757d106\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-fb2b1d-r8vf7"
Oct 31 12:53:57.732940 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:57.732896    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3992303-31dd-484d-b3ad-3fececdf57fa-istio-token\") pod \"e2e-inf-b-71bc05-openshift-default-dd6bc8b77-b65mw\" (UID: \"b3992303-31dd-484d-b3ad-3fececdf57fa\") " pod="openshift-ingress/e2e-inf-b-71bc05-openshift-default-dd6bc8b77-b65mw"
Oct 31 12:53:57.733000 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:57.732982    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3992303-31dd-484d-b3ad-3fececdf57fa-istiod-ca-cert\") pod \"e2e-inf-b-71bc05-openshift-default-dd6bc8b77-b65mw\" (UID: \"b3992303-31dd-484d-b3ad-3fececdf57fa\") " pod="openshift-ingress/e2e-inf-b-71bc05-openshift-default-dd6bc8b77-b65mw"
Oct 31 12:53:57.733045 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:57.733033    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3992303-31dd-484d-b3ad-3fececdf57fa-workload-certs\") pod \"e2e-inf-b-71bc05-openshift-default-dd6bc8b77-b65mw\" (UID: \"b3992303-31dd-484d-b3ad-3fececdf57fa\") " pod="openshift-ingress/e2e-inf-b-71bc05-openshift-default-dd6bc8b77-b65mw"
Oct 31 12:53:57.733141 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:57.733126    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3992303-31dd-484d-b3ad-3fececdf57fa-istio-envoy\") pod \"e2e-inf-b-71bc05-openshift-default-dd6bc8b77-b65mw\" (UID: \"b3992303-31dd-484d-b3ad-3fececdf57fa\") " pod="openshift-ingress/e2e-inf-b-71bc05-openshift-default-dd6bc8b77-b65mw"
Oct 31 12:53:57.733187 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:57.733166    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3992303-31dd-484d-b3ad-3fececdf57fa-istio-podinfo\") pod \"e2e-inf-b-71bc05-openshift-default-dd6bc8b77-b65mw\" (UID: \"b3992303-31dd-484d-b3ad-3fececdf57fa\") " pod="openshift-ingress/e2e-inf-b-71bc05-openshift-default-dd6bc8b77-b65mw"
Oct 31 12:53:57.733241 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:57.733196    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3992303-31dd-484d-b3ad-3fececdf57fa-workload-socket\") pod \"e2e-inf-b-71bc05-openshift-default-dd6bc8b77-b65mw\" (UID: \"b3992303-31dd-484d-b3ad-3fececdf57fa\") " pod="openshift-ingress/e2e-inf-b-71bc05-openshift-default-dd6bc8b77-b65mw"
Oct 31 12:53:57.733324 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:57.733303    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3992303-31dd-484d-b3ad-3fececdf57fa-credential-socket\") pod \"e2e-inf-b-71bc05-openshift-default-dd6bc8b77-b65mw\" (UID: \"b3992303-31dd-484d-b3ad-3fececdf57fa\") " pod="openshift-ingress/e2e-inf-b-71bc05-openshift-default-dd6bc8b77-b65mw"
Oct 31 12:53:57.746180 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:57.746133    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b218d3-khq4b"
Oct 31 12:53:57.835112 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:57.834550    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3992303-31dd-484d-b3ad-3fececdf57fa-istio-data\") pod \"e2e-inf-b-71bc05-openshift-default-dd6bc8b77-b65mw\" (UID: \"b3992303-31dd-484d-b3ad-3fececdf57fa\") " pod="openshift-ingress/e2e-inf-b-71bc05-openshift-default-dd6bc8b77-b65mw"
Oct 31 12:53:57.835112 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:57.834663    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khdc9\" (UniqueName: \"kubernetes.io/projected/85ab7b73-48a3-430a-b489-30985757d106-kube-api-access-khdc9\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-fb2b1d-r8vf7\" (UID: \"85ab7b73-48a3-430a-b489-30985757d106\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-fb2b1d-r8vf7"
Oct 31 12:53:57.835112 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:57.834701    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3992303-31dd-484d-b3ad-3fececdf57fa-istio-token\") pod \"e2e-inf-b-71bc05-openshift-default-dd6bc8b77-b65mw\" (UID: \"b3992303-31dd-484d-b3ad-3fececdf57fa\") " pod="openshift-ingress/e2e-inf-b-71bc05-openshift-default-dd6bc8b77-b65mw"
Oct 31 12:53:57.835112 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:57.834725    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3992303-31dd-484d-b3ad-3fececdf57fa-istiod-ca-cert\") pod \"e2e-inf-b-71bc05-openshift-default-dd6bc8b77-b65mw\" (UID: \"b3992303-31dd-484d-b3ad-3fececdf57fa\") " pod="openshift-ingress/e2e-inf-b-71bc05-openshift-default-dd6bc8b77-b65mw"
Oct 31 12:53:57.835112 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:57.834756    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3992303-31dd-484d-b3ad-3fececdf57fa-workload-certs\") pod \"e2e-inf-b-71bc05-openshift-default-dd6bc8b77-b65mw\" (UID: \"b3992303-31dd-484d-b3ad-3fececdf57fa\") " pod="openshift-ingress/e2e-inf-b-71bc05-openshift-default-dd6bc8b77-b65mw"
Oct 31 12:53:57.835112 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:57.834811    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3992303-31dd-484d-b3ad-3fececdf57fa-istio-envoy\") pod \"e2e-inf-b-71bc05-openshift-default-dd6bc8b77-b65mw\" (UID: \"b3992303-31dd-484d-b3ad-3fececdf57fa\") " pod="openshift-ingress/e2e-inf-b-71bc05-openshift-default-dd6bc8b77-b65mw"
Oct 31 12:53:57.835112 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:57.834848    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3992303-31dd-484d-b3ad-3fececdf57fa-istio-podinfo\") pod \"e2e-inf-b-71bc05-openshift-default-dd6bc8b77-b65mw\" (UID: \"b3992303-31dd-484d-b3ad-3fececdf57fa\") " pod="openshift-ingress/e2e-inf-b-71bc05-openshift-default-dd6bc8b77-b65mw"
Oct 31 12:53:57.835112 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:57.834879    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3992303-31dd-484d-b3ad-3fececdf57fa-workload-socket\") pod \"e2e-inf-b-71bc05-openshift-default-dd6bc8b77-b65mw\" (UID: \"b3992303-31dd-484d-b3ad-3fececdf57fa\") " pod="openshift-ingress/e2e-inf-b-71bc05-openshift-default-dd6bc8b77-b65mw"
Oct 31 12:53:57.835112 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:57.834910    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3992303-31dd-484d-b3ad-3fececdf57fa-credential-socket\") pod \"e2e-inf-b-71bc05-openshift-default-dd6bc8b77-b65mw\" (UID: \"b3992303-31dd-484d-b3ad-3fececdf57fa\") " pod="openshift-ingress/e2e-inf-b-71bc05-openshift-default-dd6bc8b77-b65mw"
Oct 31 12:53:57.835112 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:57.834975    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85ab7b73-48a3-430a-b489-30985757d106-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-fb2b1d-r8vf7\" (UID: \"85ab7b73-48a3-430a-b489-30985757d106\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-fb2b1d-r8vf7"
Oct 31 12:53:57.841682 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:57.835802    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btcs6\" (UniqueName: \"kubernetes.io/projected/b3992303-31dd-484d-b3ad-3fececdf57fa-kube-api-access-btcs6\") pod \"e2e-inf-b-71bc05-openshift-default-dd6bc8b77-b65mw\" (UID: \"b3992303-31dd-484d-b3ad-3fececdf57fa\") " pod="openshift-ingress/e2e-inf-b-71bc05-openshift-default-dd6bc8b77-b65mw"
Oct 31 12:53:57.841682 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:57.835859    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85ab7b73-48a3-430a-b489-30985757d106-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-fb2b1d-r8vf7\" (UID: \"85ab7b73-48a3-430a-b489-30985757d106\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-fb2b1d-r8vf7"
Oct 31 12:53:57.841682 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:57.836631    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3992303-31dd-484d-b3ad-3fececdf57fa-istio-data\") pod \"e2e-inf-b-71bc05-openshift-default-dd6bc8b77-b65mw\" (UID: \"b3992303-31dd-484d-b3ad-3fececdf57fa\") " pod="openshift-ingress/e2e-inf-b-71bc05-openshift-default-dd6bc8b77-b65mw"
Oct 31 12:53:57.841682 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:57.837456    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3992303-31dd-484d-b3ad-3fececdf57fa-istiod-ca-cert\") pod \"e2e-inf-b-71bc05-openshift-default-dd6bc8b77-b65mw\" (UID: \"b3992303-31dd-484d-b3ad-3fececdf57fa\") " pod="openshift-ingress/e2e-inf-b-71bc05-openshift-default-dd6bc8b77-b65mw"
Oct 31 12:53:57.841682 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:57.837961    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3992303-31dd-484d-b3ad-3fececdf57fa-credential-socket\") pod \"e2e-inf-b-71bc05-openshift-default-dd6bc8b77-b65mw\" (UID: \"b3992303-31dd-484d-b3ad-3fececdf57fa\") " pod="openshift-ingress/e2e-inf-b-71bc05-openshift-default-dd6bc8b77-b65mw"
Oct 31 12:53:57.841682 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:57.838344    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3992303-31dd-484d-b3ad-3fececdf57fa-workload-certs\") pod \"e2e-inf-b-71bc05-openshift-default-dd6bc8b77-b65mw\" (UID: \"b3992303-31dd-484d-b3ad-3fececdf57fa\") " pod="openshift-ingress/e2e-inf-b-71bc05-openshift-default-dd6bc8b77-b65mw"
Oct 31 12:53:57.841682 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:57.839090    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85ab7b73-48a3-430a-b489-30985757d106-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-fb2b1d-r8vf7\" (UID: \"85ab7b73-48a3-430a-b489-30985757d106\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-fb2b1d-r8vf7"
Oct 31 12:53:57.841682 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:57.839348    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3992303-31dd-484d-b3ad-3fececdf57fa-workload-socket\") pod \"e2e-inf-b-71bc05-openshift-default-dd6bc8b77-b65mw\" (UID: \"b3992303-31dd-484d-b3ad-3fececdf57fa\") " pod="openshift-ingress/e2e-inf-b-71bc05-openshift-default-dd6bc8b77-b65mw"
Oct 31 12:53:57.864514 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:57.842881    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3992303-31dd-484d-b3ad-3fececdf57fa-istio-podinfo\") pod \"e2e-inf-b-71bc05-openshift-default-dd6bc8b77-b65mw\" (UID: \"b3992303-31dd-484d-b3ad-3fececdf57fa\") " pod="openshift-ingress/e2e-inf-b-71bc05-openshift-default-dd6bc8b77-b65mw"
Oct 31 12:53:57.896567 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:57.865149    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3992303-31dd-484d-b3ad-3fececdf57fa-istio-envoy\") pod \"e2e-inf-b-71bc05-openshift-default-dd6bc8b77-b65mw\" (UID: \"b3992303-31dd-484d-b3ad-3fececdf57fa\") " pod="openshift-ingress/e2e-inf-b-71bc05-openshift-default-dd6bc8b77-b65mw"
Oct 31 12:53:57.896567 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:57.869681    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3992303-31dd-484d-b3ad-3fececdf57fa-istio-token\") pod \"e2e-inf-b-71bc05-openshift-default-dd6bc8b77-b65mw\" (UID: \"b3992303-31dd-484d-b3ad-3fececdf57fa\") " pod="openshift-ingress/e2e-inf-b-71bc05-openshift-default-dd6bc8b77-b65mw"
Oct 31 12:53:57.896567 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:57.872077    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khdc9\" (UniqueName: \"kubernetes.io/projected/85ab7b73-48a3-430a-b489-30985757d106-kube-api-access-khdc9\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-fb2b1d-r8vf7\" (UID: \"85ab7b73-48a3-430a-b489-30985757d106\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-fb2b1d-r8vf7"
Oct 31 12:53:57.950818 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:57.932459    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-btcs6\" (UniqueName: \"kubernetes.io/projected/b3992303-31dd-484d-b3ad-3fececdf57fa-kube-api-access-btcs6\") pod \"e2e-inf-b-71bc05-openshift-default-dd6bc8b77-b65mw\" (UID: \"b3992303-31dd-484d-b3ad-3fececdf57fa\") " pod="openshift-ingress/e2e-inf-b-71bc05-openshift-default-dd6bc8b77-b65mw"
Oct 31 12:53:58.016425 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:57.998908    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-71bc05-openshift-default-dd6bc8b77-b65mw"
Oct 31 12:53:58.142792 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:58.117000    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85ab7b73-48a3-430a-b489-30985757d106-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-fb2b1d-r8vf7\" (UID: \"85ab7b73-48a3-430a-b489-30985757d106\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-fb2b1d-r8vf7"
Oct 31 12:53:58.274880 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:58.267785    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-fb2b1d-r8vf7"
Oct 31 12:53:58.571763 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:53:58.571223    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7972aee8_dbdd_4c32_b46d_6cdf855aa53d.slice/crio-c147293f3e32133c4543425bbe589eaecea2ef6b307ef49eb3d8a3b60560a083 WatchSource:0}: Error finding container c147293f3e32133c4543425bbe589eaecea2ef6b307ef49eb3d8a3b60560a083: Status 404 returned error can't find the container with id c147293f3e32133c4543425bbe589eaecea2ef6b307ef49eb3d8a3b60560a083
Oct 31 12:53:58.643144 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:58.643054    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b218d3-khq4b"]
Oct 31 12:53:58.672734 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:58.667695    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:53:58.672734 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:58.667786    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:53:58.672734 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:58.667829    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:53:58.699284 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:53:58.689425    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85ab7b73_48a3_430a_b489_30985757d106.slice/crio-15934746aa14a7b39208d1c8c57ededc7e501366f9ddd485c70f0e7c15207cd7 WatchSource:0}: Error finding container 15934746aa14a7b39208d1c8c57ededc7e501366f9ddd485c70f0e7c15207cd7: Status 404 returned error can't find the container with id 15934746aa14a7b39208d1c8c57ededc7e501366f9ddd485c70f0e7c15207cd7
Oct 31 12:53:58.699284 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:58.694538    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-71bc05-openshift-default-dd6bc8b77-b65mw"]
Oct 31 12:53:58.702586 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:58.702551    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-fb2b1d-r8vf7"]
Oct 31 12:53:59.274283 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:59.274246    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d1721ec-dq9xk"]
Oct 31 12:53:59.283125 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:59.280626    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d1721ec-dq9xk"
Oct 31 12:53:59.287053 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:59.284253    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1e071a8-vqh87"]
Oct 31 12:53:59.305507 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:59.300741    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1e071a8-vqh87"
Oct 31 12:53:59.322727 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:59.322655    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d1721ec-dq9xk"]
Oct 31 12:53:59.328811 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:59.328748    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1e071a8-vqh87"]
Oct 31 12:53:59.338381 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:59.338345    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e98df8-openshift-default-779d98479-vkrj6"
Oct 31 12:53:59.386991 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:59.386957    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb1bcc49-26c3-46a9-a134-dffba7b91a58-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d1721ec-dq9xk\" (UID: \"bb1bcc49-26c3-46a9-a134-dffba7b91a58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d1721ec-dq9xk"
Oct 31 12:53:59.387207 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:59.387189    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb1bcc49-26c3-46a9-a134-dffba7b91a58-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d1721ec-dq9xk\" (UID: \"bb1bcc49-26c3-46a9-a134-dffba7b91a58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d1721ec-dq9xk"
Oct 31 12:53:59.387364 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:59.387349    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srps7\" (UniqueName: \"kubernetes.io/projected/bb1bcc49-26c3-46a9-a134-dffba7b91a58-kube-api-access-srps7\") pod \"maas-api-revoke-sub-8d1721ec-dq9xk\" (UID: \"bb1bcc49-26c3-46a9-a134-dffba7b91a58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d1721ec-dq9xk"
Oct 31 12:53:59.432395 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:59.432364    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e98df8-openshift-default-779d98479-r2jbq"
Oct 31 12:53:59.488327 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:59.488288    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d62f8e4-b062-4dd4-a23d-0f66566c83da-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d62f8e4-b062-4dd4-a23d-0f66566c83da-istiod-ca-cert\") pod \"0d62f8e4-b062-4dd4-a23d-0f66566c83da\" (UID: \"0d62f8e4-b062-4dd4-a23d-0f66566c83da\") "
Oct 31 12:53:59.488545 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:59.488420    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d62f8e4-b062-4dd4-a23d-0f66566c83da-istio-token\" (UniqueName: \"kubernetes.io/projected/0d62f8e4-b062-4dd4-a23d-0f66566c83da-istio-token\") pod \"0d62f8e4-b062-4dd4-a23d-0f66566c83da\" (UID: \"0d62f8e4-b062-4dd4-a23d-0f66566c83da\") "
Oct 31 12:53:59.488631 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:59.488478    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d62f8e4-b062-4dd4-a23d-0f66566c83da-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d62f8e4-b062-4dd4-a23d-0f66566c83da-istio-envoy\") pod \"0d62f8e4-b062-4dd4-a23d-0f66566c83da\" (UID: \"0d62f8e4-b062-4dd4-a23d-0f66566c83da\") "
Oct 31 12:53:59.488631 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:59.488612    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d62f8e4-b062-4dd4-a23d-0f66566c83da-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d62f8e4-b062-4dd4-a23d-0f66566c83da-workload-certs\") pod \"0d62f8e4-b062-4dd4-a23d-0f66566c83da\" (UID: \"0d62f8e4-b062-4dd4-a23d-0f66566c83da\") "
Oct 31 12:53:59.488734 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:59.488660    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d62f8e4-b062-4dd4-a23d-0f66566c83da-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d62f8e4-b062-4dd4-a23d-0f66566c83da-credential-socket\") pod \"0d62f8e4-b062-4dd4-a23d-0f66566c83da\" (UID: \"0d62f8e4-b062-4dd4-a23d-0f66566c83da\") "
Oct 31 12:53:59.488734 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:59.488724    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d62f8e4-b062-4dd4-a23d-0f66566c83da-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d62f8e4-b062-4dd4-a23d-0f66566c83da-workload-socket\") pod \"0d62f8e4-b062-4dd4-a23d-0f66566c83da\" (UID: \"0d62f8e4-b062-4dd4-a23d-0f66566c83da\") "
Oct 31 12:53:59.488837 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:59.488778    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0d62f8e4-b062-4dd4-a23d-0f66566c83da-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d62f8e4-b062-4dd4-a23d-0f66566c83da-istio-podinfo\") pod \"0d62f8e4-b062-4dd4-a23d-0f66566c83da\" (UID: \"0d62f8e4-b062-4dd4-a23d-0f66566c83da\") "
Oct 31 12:53:59.488837 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:59.488805    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d62f8e4-b062-4dd4-a23d-0f66566c83da-kube-api-access-bspb4\" (UniqueName: \"kubernetes.io/projected/0d62f8e4-b062-4dd4-a23d-0f66566c83da-kube-api-access-bspb4\") pod \"0d62f8e4-b062-4dd4-a23d-0f66566c83da\" (UID: \"0d62f8e4-b062-4dd4-a23d-0f66566c83da\") "
Oct 31 12:53:59.488943 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:59.488844    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d62f8e4-b062-4dd4-a23d-0f66566c83da-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d62f8e4-b062-4dd4-a23d-0f66566c83da-istio-data\") pod \"0d62f8e4-b062-4dd4-a23d-0f66566c83da\" (UID: \"0d62f8e4-b062-4dd4-a23d-0f66566c83da\") "
Oct 31 12:53:59.489288 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:59.489262    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb1bcc49-26c3-46a9-a134-dffba7b91a58-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d1721ec-dq9xk\" (UID: \"bb1bcc49-26c3-46a9-a134-dffba7b91a58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d1721ec-dq9xk"
Oct 31 12:53:59.489587 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:59.489566    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb1bcc49-26c3-46a9-a134-dffba7b91a58-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d1721ec-dq9xk\" (UID: \"bb1bcc49-26c3-46a9-a134-dffba7b91a58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d1721ec-dq9xk"
Oct 31 12:53:59.489694 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:59.489683    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc885fe1-b05b-48a3-9617-2d9475ec888a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e1e071a8-vqh87\" (UID: \"fc885fe1-b05b-48a3-9617-2d9475ec888a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1e071a8-vqh87"
Oct 31 12:53:59.489821 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:59.489790    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8zp5\" (UniqueName: \"kubernetes.io/projected/fc885fe1-b05b-48a3-9617-2d9475ec888a-kube-api-access-m8zp5\") pod \"maas-api-revoke-sub-e1e071a8-vqh87\" (UID: \"fc885fe1-b05b-48a3-9617-2d9475ec888a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1e071a8-vqh87"
Oct 31 12:53:59.489875 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:59.489834    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc885fe1-b05b-48a3-9617-2d9475ec888a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e1e071a8-vqh87\" (UID: \"fc885fe1-b05b-48a3-9617-2d9475ec888a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1e071a8-vqh87"
Oct 31 12:53:59.489927 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:59.489886    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srps7\" (UniqueName: \"kubernetes.io/projected/bb1bcc49-26c3-46a9-a134-dffba7b91a58-kube-api-access-srps7\") pod \"maas-api-revoke-sub-8d1721ec-dq9xk\" (UID: \"bb1bcc49-26c3-46a9-a134-dffba7b91a58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d1721ec-dq9xk"
Oct 31 12:53:59.490641 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:59.490589    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d62f8e4-b062-4dd4-a23d-0f66566c83da-istiod-ca-cert" pod "0d62f8e4-b062-4dd4-a23d-0f66566c83da" (UID: "0d62f8e4-b062-4dd4-a23d-0f66566c83da"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:53:59.491609 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:59.491364    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb1bcc49-26c3-46a9-a134-dffba7b91a58-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d1721ec-dq9xk\" (UID: \"bb1bcc49-26c3-46a9-a134-dffba7b91a58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d1721ec-dq9xk"
Oct 31 12:53:59.491949 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:59.491758    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d62f8e4-b062-4dd4-a23d-0f66566c83da-istio-data" pod "0d62f8e4-b062-4dd4-a23d-0f66566c83da" (UID: "0d62f8e4-b062-4dd4-a23d-0f66566c83da"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:53:59.493369 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:59.493128    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d62f8e4-b062-4dd4-a23d-0f66566c83da-workload-socket" pod "0d62f8e4-b062-4dd4-a23d-0f66566c83da" (UID: "0d62f8e4-b062-4dd4-a23d-0f66566c83da"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:53:59.493369 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:59.493171    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d62f8e4-b062-4dd4-a23d-0f66566c83da-credential-socket" pod "0d62f8e4-b062-4dd4-a23d-0f66566c83da" (UID: "0d62f8e4-b062-4dd4-a23d-0f66566c83da"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:53:59.493369 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:59.493334    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d62f8e4-b062-4dd4-a23d-0f66566c83da-workload-certs" pod "0d62f8e4-b062-4dd4-a23d-0f66566c83da" (UID: "0d62f8e4-b062-4dd4-a23d-0f66566c83da"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:53:59.493369 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:59.493333    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d62f8e4-b062-4dd4-a23d-0f66566c83da-istio-token" pod "0d62f8e4-b062-4dd4-a23d-0f66566c83da" (UID: "0d62f8e4-b062-4dd4-a23d-0f66566c83da"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:53:59.495995 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:59.495274    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb1bcc49-26c3-46a9-a134-dffba7b91a58-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d1721ec-dq9xk\" (UID: \"bb1bcc49-26c3-46a9-a134-dffba7b91a58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d1721ec-dq9xk"
Oct 31 12:53:59.495995 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:59.495393    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d62f8e4-b062-4dd4-a23d-0f66566c83da-istio-envoy" pod "0d62f8e4-b062-4dd4-a23d-0f66566c83da" (UID: "0d62f8e4-b062-4dd4-a23d-0f66566c83da"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:53:59.495995 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:59.495541    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d62f8e4-b062-4dd4-a23d-0f66566c83da-kube-api-access-bspb4" pod "0d62f8e4-b062-4dd4-a23d-0f66566c83da" (UID: "0d62f8e4-b062-4dd4-a23d-0f66566c83da"). InnerVolumeSpecName "kube-api-access-bspb4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:53:59.497283 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:59.497261    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0d62f8e4-b062-4dd4-a23d-0f66566c83da-istio-podinfo" pod "0d62f8e4-b062-4dd4-a23d-0f66566c83da" (UID: "0d62f8e4-b062-4dd4-a23d-0f66566c83da"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 12:53:59.504855 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:59.504829    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srps7\" (UniqueName: \"kubernetes.io/projected/bb1bcc49-26c3-46a9-a134-dffba7b91a58-kube-api-access-srps7\") pod \"maas-api-revoke-sub-8d1721ec-dq9xk\" (UID: \"bb1bcc49-26c3-46a9-a134-dffba7b91a58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d1721ec-dq9xk"
Oct 31 12:53:59.542669 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:59.542633    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-fb2b1d-r8vf7" event={"ID":"85ab7b73-48a3-430a-b489-30985757d106","Type":"ContainerStarted","Data":"342a263e5f70b61f3edd4c3c8dbeb77e2de25a21c0fd87613713fbfbf5e5dbea"}
Oct 31 12:53:59.542808 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:59.542697    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-fb2b1d-r8vf7" event={"ID":"85ab7b73-48a3-430a-b489-30985757d106","Type":"ContainerStarted","Data":"15934746aa14a7b39208d1c8c57ededc7e501366f9ddd485c70f0e7c15207cd7"}
Oct 31 12:53:59.544660 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:59.544630    5060 generic.go:356] "Generic (PLEG): container finished" podID="0d62f8e4-b062-4dd4-a23d-0f66566c83da" containerID="75f2642303e354ccd77cddad60790638c3b1c480e432f7aad1fb90ca360b1992" exitCode=0
Oct 31 12:53:59.544773 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:59.544670    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e98df8-openshift-default-779d98479-vkrj6"
Oct 31 12:53:59.544773 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:59.544686    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e98df8-openshift-default-779d98479-vkrj6" event={"ID":"0d62f8e4-b062-4dd4-a23d-0f66566c83da","Type":"ContainerDied","Data":"75f2642303e354ccd77cddad60790638c3b1c480e432f7aad1fb90ca360b1992"}
Oct 31 12:53:59.544773 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:59.544740    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e98df8-openshift-default-779d98479-vkrj6" event={"ID":"0d62f8e4-b062-4dd4-a23d-0f66566c83da","Type":"ContainerDied","Data":"a2648314fc5d81306544915fa7cbabfcf11182a2ab6539373f7a5accb94bef42"}
Oct 31 12:53:59.544931 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:59.544786    5060 scope.go:122] "RemoveContainer" containerID="75f2642303e354ccd77cddad60790638c3b1c480e432f7aad1fb90ca360b1992"
Oct 31 12:53:59.549003 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:59.548900    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-71bc05-openshift-default-dd6bc8b77-b65mw" event={"ID":"b3992303-31dd-484d-b3ad-3fececdf57fa","Type":"ContainerStarted","Data":"e33caf28b00f04205fa360aaf1fadccce6cb70b25e057f52a58a026e4d2c2ea6"}
Oct 31 12:53:59.549003 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:59.548972    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-71bc05-openshift-default-dd6bc8b77-b65mw" event={"ID":"b3992303-31dd-484d-b3ad-3fececdf57fa","Type":"ContainerStarted","Data":"83309eb552ff9b9ac2d79b78e2d5606c33e8c50b44c711e80bb3ecfa6e1b5a70"}
Oct 31 12:53:59.551933 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:59.551904    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b218d3-khq4b" event={"ID":"7972aee8-dbdd-4c32-b46d-6cdf855aa53d","Type":"ContainerStarted","Data":"d7bca43b9b15ecff60de5ccd7e6e5372115f4a5f0704486e24f8c2ca77f7b331"}
Oct 31 12:53:59.552061 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:59.551954    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b218d3-khq4b" event={"ID":"7972aee8-dbdd-4c32-b46d-6cdf855aa53d","Type":"ContainerStarted","Data":"c147293f3e32133c4543425bbe589eaecea2ef6b307ef49eb3d8a3b60560a083"}
Oct 31 12:53:59.555120 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:59.555094    5060 generic.go:356] "Generic (PLEG): container finished" podID="e08b08ff-4f40-474a-a103-2c499e999942" containerID="69e50444a6492290d3c0f1f8e259cce1808ee453214910de09e68331867cc777" exitCode=0
Oct 31 12:53:59.555285 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:59.555136    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e98df8-openshift-default-779d98479-r2jbq" event={"ID":"e08b08ff-4f40-474a-a103-2c499e999942","Type":"ContainerDied","Data":"69e50444a6492290d3c0f1f8e259cce1808ee453214910de09e68331867cc777"}
Oct 31 12:53:59.555285 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:59.555167    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e98df8-openshift-default-779d98479-r2jbq"
Oct 31 12:53:59.555285 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:59.555175    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e98df8-openshift-default-779d98479-r2jbq" event={"ID":"e08b08ff-4f40-474a-a103-2c499e999942","Type":"ContainerDied","Data":"afbfc93954bf646eccabaeeb83fb2b76c0ecce13d4945e9b381f2c06c00d9d94"}
Oct 31 12:53:59.563629 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:59.563612    5060 scope.go:122] "RemoveContainer" containerID="75f2642303e354ccd77cddad60790638c3b1c480e432f7aad1fb90ca360b1992"
Oct 31 12:53:59.564011 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:53:59.563982    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75f2642303e354ccd77cddad60790638c3b1c480e432f7aad1fb90ca360b1992\": container with ID starting with 75f2642303e354ccd77cddad60790638c3b1c480e432f7aad1fb90ca360b1992 not found: ID does not exist" containerID="75f2642303e354ccd77cddad60790638c3b1c480e432f7aad1fb90ca360b1992"
Oct 31 12:53:59.564127 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:59.564020    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75f2642303e354ccd77cddad60790638c3b1c480e432f7aad1fb90ca360b1992"} err="failed to get container status \"75f2642303e354ccd77cddad60790638c3b1c480e432f7aad1fb90ca360b1992\": rpc error: code = NotFound desc = could not find container \"75f2642303e354ccd77cddad60790638c3b1c480e432f7aad1fb90ca360b1992\": container with ID starting with 75f2642303e354ccd77cddad60790638c3b1c480e432f7aad1fb90ca360b1992 not found: ID does not exist"
Oct 31 12:53:59.564127 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:59.564046    5060 scope.go:122] "RemoveContainer" containerID="69e50444a6492290d3c0f1f8e259cce1808ee453214910de09e68331867cc777"
Oct 31 12:53:59.570019 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:59.569979    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-fb2b1d-r8vf7" podStartSLOduration=2.569962692 podStartE2EDuration="2.569962692s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:53:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:53:59.566222243 +0000 UTC m=+1602.669289957" watchObservedRunningTime="2026-10-31 12:53:59.569962692 +0000 UTC m=+1602.673030402"
Oct 31 12:53:59.578879 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:59.578846    5060 scope.go:122] "RemoveContainer" containerID="b58adec0fddb21aa3934309a7e11d9cd4382f30d2afafd76ca744f39eef3cd01"
Oct 31 12:53:59.589978 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:59.589953    5060 scope.go:122] "RemoveContainer" containerID="69e50444a6492290d3c0f1f8e259cce1808ee453214910de09e68331867cc777"
Oct 31 12:53:59.590279 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:53:59.590249    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69e50444a6492290d3c0f1f8e259cce1808ee453214910de09e68331867cc777\": container with ID starting with 69e50444a6492290d3c0f1f8e259cce1808ee453214910de09e68331867cc777 not found: ID does not exist" containerID="69e50444a6492290d3c0f1f8e259cce1808ee453214910de09e68331867cc777"
Oct 31 12:53:59.590354 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:59.590290    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69e50444a6492290d3c0f1f8e259cce1808ee453214910de09e68331867cc777"} err="failed to get container status \"69e50444a6492290d3c0f1f8e259cce1808ee453214910de09e68331867cc777\": rpc error: code = NotFound desc = could not find container \"69e50444a6492290d3c0f1f8e259cce1808ee453214910de09e68331867cc777\": container with ID starting with 69e50444a6492290d3c0f1f8e259cce1808ee453214910de09e68331867cc777 not found: ID does not exist"
Oct 31 12:53:59.590354 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:59.590308    5060 scope.go:122] "RemoveContainer" containerID="b58adec0fddb21aa3934309a7e11d9cd4382f30d2afafd76ca744f39eef3cd01"
Oct 31 12:53:59.590472 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:59.590451    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e08b08ff-4f40-474a-a103-2c499e999942-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e08b08ff-4f40-474a-a103-2c499e999942-workload-certs\") pod \"e08b08ff-4f40-474a-a103-2c499e999942\" (UID: \"e08b08ff-4f40-474a-a103-2c499e999942\") "
Oct 31 12:53:59.590547 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:59.590536    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e08b08ff-4f40-474a-a103-2c499e999942-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e08b08ff-4f40-474a-a103-2c499e999942-istio-data\") pod \"e08b08ff-4f40-474a-a103-2c499e999942\" (UID: \"e08b08ff-4f40-474a-a103-2c499e999942\") "
Oct 31 12:53:59.590622 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:59.590606    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e08b08ff-4f40-474a-a103-2c499e999942-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e08b08ff-4f40-474a-a103-2c499e999942-istiod-ca-cert\") pod \"e08b08ff-4f40-474a-a103-2c499e999942\" (UID: \"e08b08ff-4f40-474a-a103-2c499e999942\") "
Oct 31 12:53:59.590708 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:53:59.590679    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b58adec0fddb21aa3934309a7e11d9cd4382f30d2afafd76ca744f39eef3cd01\": container with ID starting with b58adec0fddb21aa3934309a7e11d9cd4382f30d2afafd76ca744f39eef3cd01 not found: ID does not exist" containerID="b58adec0fddb21aa3934309a7e11d9cd4382f30d2afafd76ca744f39eef3cd01"
Oct 31 12:53:59.590768 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:59.590721    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e08b08ff-4f40-474a-a103-2c499e999942-workload-certs" pod "e08b08ff-4f40-474a-a103-2c499e999942" (UID: "e08b08ff-4f40-474a-a103-2c499e999942"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:53:59.590768 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:59.590719    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b58adec0fddb21aa3934309a7e11d9cd4382f30d2afafd76ca744f39eef3cd01"} err="failed to get container status \"b58adec0fddb21aa3934309a7e11d9cd4382f30d2afafd76ca744f39eef3cd01\": rpc error: code = NotFound desc = could not find container \"b58adec0fddb21aa3934309a7e11d9cd4382f30d2afafd76ca744f39eef3cd01\": container with ID starting with b58adec0fddb21aa3934309a7e11d9cd4382f30d2afafd76ca744f39eef3cd01 not found: ID does not exist"
Oct 31 12:53:59.590768 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:59.590746    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e08b08ff-4f40-474a-a103-2c499e999942-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e08b08ff-4f40-474a-a103-2c499e999942-workload-socket\") pod \"e08b08ff-4f40-474a-a103-2c499e999942\" (UID: \"e08b08ff-4f40-474a-a103-2c499e999942\") "
Oct 31 12:53:59.590948 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:59.590816    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e08b08ff-4f40-474a-a103-2c499e999942-istio-data" pod "e08b08ff-4f40-474a-a103-2c499e999942" (UID: "e08b08ff-4f40-474a-a103-2c499e999942"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:53:59.590948 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:59.590844    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e08b08ff-4f40-474a-a103-2c499e999942-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e08b08ff-4f40-474a-a103-2c499e999942-credential-socket\") pod \"e08b08ff-4f40-474a-a103-2c499e999942\" (UID: \"e08b08ff-4f40-474a-a103-2c499e999942\") "
Oct 31 12:53:59.590948 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:59.590900    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e08b08ff-4f40-474a-a103-2c499e999942-istio-token\" (UniqueName: \"kubernetes.io/projected/e08b08ff-4f40-474a-a103-2c499e999942-istio-token\") pod \"e08b08ff-4f40-474a-a103-2c499e999942\" (UID: \"e08b08ff-4f40-474a-a103-2c499e999942\") "
Oct 31 12:53:59.591145 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:59.590997    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e08b08ff-4f40-474a-a103-2c499e999942-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e08b08ff-4f40-474a-a103-2c499e999942-istio-envoy\") pod \"e08b08ff-4f40-474a-a103-2c499e999942\" (UID: \"e08b08ff-4f40-474a-a103-2c499e999942\") "
Oct 31 12:53:59.591145 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:59.591023    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e08b08ff-4f40-474a-a103-2c499e999942-workload-socket" pod "e08b08ff-4f40-474a-a103-2c499e999942" (UID: "e08b08ff-4f40-474a-a103-2c499e999942"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:53:59.591145 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:59.591093    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e08b08ff-4f40-474a-a103-2c499e999942-kube-api-access-5m885\" (UniqueName: \"kubernetes.io/projected/e08b08ff-4f40-474a-a103-2c499e999942-kube-api-access-5m885\") pod \"e08b08ff-4f40-474a-a103-2c499e999942\" (UID: \"e08b08ff-4f40-474a-a103-2c499e999942\") "
Oct 31 12:53:59.591145 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:59.591104    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e08b08ff-4f40-474a-a103-2c499e999942-credential-socket" pod "e08b08ff-4f40-474a-a103-2c499e999942" (UID: "e08b08ff-4f40-474a-a103-2c499e999942"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:53:59.591145 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:59.591035    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e08b08ff-4f40-474a-a103-2c499e999942-istiod-ca-cert" pod "e08b08ff-4f40-474a-a103-2c499e999942" (UID: "e08b08ff-4f40-474a-a103-2c499e999942"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:53:59.591145 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:59.591138    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e08b08ff-4f40-474a-a103-2c499e999942-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e08b08ff-4f40-474a-a103-2c499e999942-istio-podinfo\") pod \"e08b08ff-4f40-474a-a103-2c499e999942\" (UID: \"e08b08ff-4f40-474a-a103-2c499e999942\") "
Oct 31 12:53:59.591629 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:59.591607    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc885fe1-b05b-48a3-9617-2d9475ec888a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e1e071a8-vqh87\" (UID: \"fc885fe1-b05b-48a3-9617-2d9475ec888a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1e071a8-vqh87"
Oct 31 12:53:59.591762 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:59.591707    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8zp5\" (UniqueName: \"kubernetes.io/projected/fc885fe1-b05b-48a3-9617-2d9475ec888a-kube-api-access-m8zp5\") pod \"maas-api-revoke-sub-e1e071a8-vqh87\" (UID: \"fc885fe1-b05b-48a3-9617-2d9475ec888a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1e071a8-vqh87"
Oct 31 12:53:59.591762 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:59.591753    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc885fe1-b05b-48a3-9617-2d9475ec888a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e1e071a8-vqh87\" (UID: \"fc885fe1-b05b-48a3-9617-2d9475ec888a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1e071a8-vqh87"
Oct 31 12:53:59.591957 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:59.591942    5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d62f8e4-b062-4dd4-a23d-0f66566c83da-istio-token\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:59.592088 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:59.591960    5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e08b08ff-4f40-474a-a103-2c499e999942-workload-socket\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:59.592088 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:59.591975    5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d62f8e4-b062-4dd4-a23d-0f66566c83da-istio-envoy\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:59.592088 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:59.591993    5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d62f8e4-b062-4dd4-a23d-0f66566c83da-workload-certs\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:59.592088 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:59.592007    5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e08b08ff-4f40-474a-a103-2c499e999942-credential-socket\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:59.592088 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:59.592021    5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d62f8e4-b062-4dd4-a23d-0f66566c83da-credential-socket\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:59.592088 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:59.592036    5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d62f8e4-b062-4dd4-a23d-0f66566c83da-workload-socket\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:59.592088 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:59.592049    5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d62f8e4-b062-4dd4-a23d-0f66566c83da-istio-podinfo\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:59.592088 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:59.592063    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bspb4\" (UniqueName: \"kubernetes.io/projected/0d62f8e4-b062-4dd4-a23d-0f66566c83da-kube-api-access-bspb4\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:59.592088 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:59.592078    5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d62f8e4-b062-4dd4-a23d-0f66566c83da-istio-data\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:59.592088 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:59.592091    5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e08b08ff-4f40-474a-a103-2c499e999942-workload-certs\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:59.592663 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:59.592106    5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e08b08ff-4f40-474a-a103-2c499e999942-istio-data\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:59.592663 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:59.592119    5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e08b08ff-4f40-474a-a103-2c499e999942-istiod-ca-cert\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:59.592663 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:59.592136    5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d62f8e4-b062-4dd4-a23d-0f66566c83da-istiod-ca-cert\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:59.592827 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:59.592671    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc885fe1-b05b-48a3-9617-2d9475ec888a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e1e071a8-vqh87\" (UID: \"fc885fe1-b05b-48a3-9617-2d9475ec888a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1e071a8-vqh87"
Oct 31 12:53:59.593620 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:59.593596    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e08b08ff-4f40-474a-a103-2c499e999942-istio-token" pod "e08b08ff-4f40-474a-a103-2c499e999942" (UID: "e08b08ff-4f40-474a-a103-2c499e999942"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:53:59.593722 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:59.593693    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e08b08ff-4f40-474a-a103-2c499e999942-istio-envoy" pod "e08b08ff-4f40-474a-a103-2c499e999942" (UID: "e08b08ff-4f40-474a-a103-2c499e999942"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:53:59.594111 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:59.594088    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e08b08ff-4f40-474a-a103-2c499e999942-kube-api-access-5m885" pod "e08b08ff-4f40-474a-a103-2c499e999942" (UID: "e08b08ff-4f40-474a-a103-2c499e999942"). InnerVolumeSpecName "kube-api-access-5m885". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:53:59.594242 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:59.594221    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e08b08ff-4f40-474a-a103-2c499e999942-istio-podinfo" pod "e08b08ff-4f40-474a-a103-2c499e999942" (UID: "e08b08ff-4f40-474a-a103-2c499e999942"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 12:53:59.594409 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:59.594393    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc885fe1-b05b-48a3-9617-2d9475ec888a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e1e071a8-vqh87\" (UID: \"fc885fe1-b05b-48a3-9617-2d9475ec888a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1e071a8-vqh87"
Oct 31 12:53:59.599302 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:59.599256    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-71bc05-openshift-default-dd6bc8b77-b65mw" podStartSLOduration=2.599239533 podStartE2EDuration="2.599239533s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:53:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:53:59.596253134 +0000 UTC m=+1602.699320846" watchObservedRunningTime="2026-10-31 12:53:59.599239533 +0000 UTC m=+1602.702307249"
Oct 31 12:53:59.620887 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:59.620783    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d1721ec-dq9xk"
Oct 31 12:53:59.625126 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:59.625099    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8zp5\" (UniqueName: \"kubernetes.io/projected/fc885fe1-b05b-48a3-9617-2d9475ec888a-kube-api-access-m8zp5\") pod \"maas-api-revoke-sub-e1e071a8-vqh87\" (UID: \"fc885fe1-b05b-48a3-9617-2d9475ec888a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1e071a8-vqh87"
Oct 31 12:53:59.630748 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:59.630725    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-e98df8-openshift-default-779d98479-vkrj6"]
Oct 31 12:53:59.635516 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:59.635469    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-e98df8-openshift-default-779d98479-vkrj6"]
Oct 31 12:53:59.635700 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:59.635679    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1e071a8-vqh87"
Oct 31 12:53:59.648889 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:59.648843    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b218d3-khq4b" podStartSLOduration=2.648828834 podStartE2EDuration="2.648828834s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:53:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:53:59.645105508 +0000 UTC m=+1602.748173236" watchObservedRunningTime="2026-10-31 12:53:59.648828834 +0000 UTC m=+1602.751896546"
Oct 31 12:53:59.696110 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:59.696081    5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e08b08ff-4f40-474a-a103-2c499e999942-istio-podinfo\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:59.696479 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:59.696453    5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e08b08ff-4f40-474a-a103-2c499e999942-istio-token\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:59.696615 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:59.696510    5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e08b08ff-4f40-474a-a103-2c499e999942-istio-envoy\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:59.696615 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:59.696528    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5m885\" (UniqueName: \"kubernetes.io/projected/e08b08ff-4f40-474a-a103-2c499e999942-kube-api-access-5m885\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:59.914341 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:59.914298    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-e98df8-openshift-default-779d98479-r2jbq"]
Oct 31 12:53:59.919890 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:59.919859    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-e98df8-openshift-default-779d98479-r2jbq"]
Oct 31 12:53:59.948349 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:53:59.948273    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7phb" containerName="extractor"
Oct 31 12:53:59.948466 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:59.948414    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d1721ec-dq9xk"]
Oct 31 12:53:59.959279 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:53:59.959226    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb1bcc49_26c3_46a9_a134_dffba7b91a58.slice/crio-0309901f13e0c278608ae1ecbac3c24fecab1b0fb03a0c87db6a1340a224497a WatchSource:0}: Error finding container 0309901f13e0c278608ae1ecbac3c24fecab1b0fb03a0c87db6a1340a224497a: Status 404 returned error can't find the container with id 0309901f13e0c278608ae1ecbac3c24fecab1b0fb03a0c87db6a1340a224497a
Oct 31 12:53:59.985573 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:53:59.985538    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-85ce08-5f656fc7f9-z5g9p" podUID="b03c9dc3-2388-458b-8e7f-09cf2811021f" containerName="payload-processing" containerID="cri-o://b0a736c77932c00d5497c4435970c017fd800e61c497ebcfb4c5d9fc653ab315" gracePeriod=25
Oct 31 12:54:00.000635 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:00.000598    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-71bc05-openshift-default-dd6bc8b77-b65mw"
Oct 31 12:54:00.007356 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:00.007328    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-71bc05-openshift-default-dd6bc8b77-b65mw"
Oct 31 12:54:00.056463 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:00.056432    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1e071a8-vqh87"]
Oct 31 12:54:00.056928 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:54:00.056903    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc885fe1_b05b_48a3_9617_2d9475ec888a.slice/crio-73db0c62cfdf32d7119526a1d6edcbcec0882a3badb8f5dedf60f66266a78f2a WatchSource:0}: Error finding container 73db0c62cfdf32d7119526a1d6edcbcec0882a3badb8f5dedf60f66266a78f2a: Status 404 returned error can't find the container with id 73db0c62cfdf32d7119526a1d6edcbcec0882a3badb8f5dedf60f66266a78f2a
Oct 31 12:54:00.464234 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:54:00.464190    5060 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod330523c8_06c9_42a2_ac3f_9dd4ad62c392.slice/crio-d190e7520eebe52b0c749ad77fb372857ff3f35218eb3dc8fea952d63280f3d9.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod330523c8_06c9_42a2_ac3f_9dd4ad62c392.slice/crio-conmon-d190e7520eebe52b0c749ad77fb372857ff3f35218eb3dc8fea952d63280f3d9.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1e6c7f5_d872_4549_b950_1e751b6374ab.slice/crio-15eb93212f666238582935bcfb7fb66c4776d563bb22273bc4eb80596e90659b.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1e6c7f5_d872_4549_b950_1e751b6374ab.slice/crio-conmon-15eb93212f666238582935bcfb7fb66c4776d563bb22273bc4eb80596e90659b.scope\": RecentStats: unable to find data in memory cache]"
Oct 31 12:54:00.644428 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:00.644389    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1e071a8-vqh87" event={"ID":"fc885fe1-b05b-48a3-9617-2d9475ec888a","Type":"ContainerStarted","Data":"56971a4a3a477e8095fef84a7d644d486e7fda1a8ac07cd046f5df7327080b1c"}
Oct 31 12:54:00.644704 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:00.644454    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1e071a8-vqh87" event={"ID":"fc885fe1-b05b-48a3-9617-2d9475ec888a","Type":"ContainerStarted","Data":"73db0c62cfdf32d7119526a1d6edcbcec0882a3badb8f5dedf60f66266a78f2a"}
Oct 31 12:54:00.647093 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:00.647063    5060 generic.go:356] "Generic (PLEG): container finished" podID="85ab7b73-48a3-430a-b489-30985757d106" containerID="342a263e5f70b61f3edd4c3c8dbeb77e2de25a21c0fd87613713fbfbf5e5dbea" exitCode=0
Oct 31 12:54:00.647371 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:00.647349    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-fb2b1d-r8vf7" event={"ID":"85ab7b73-48a3-430a-b489-30985757d106","Type":"ContainerDied","Data":"342a263e5f70b61f3edd4c3c8dbeb77e2de25a21c0fd87613713fbfbf5e5dbea"}
Oct 31 12:54:00.649857 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:00.649830    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d1721ec-dq9xk" event={"ID":"bb1bcc49-26c3-46a9-a134-dffba7b91a58","Type":"ContainerStarted","Data":"223c129273c47e87f1c3e2d957c390796115b371212736d6cecd42833b9091a7"}
Oct 31 12:54:00.649974 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:00.649877    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d1721ec-dq9xk" event={"ID":"bb1bcc49-26c3-46a9-a134-dffba7b91a58","Type":"ContainerStarted","Data":"0309901f13e0c278608ae1ecbac3c24fecab1b0fb03a0c87db6a1340a224497a"}
Oct 31 12:54:00.651929 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:00.651903    5060 generic.go:356] "Generic (PLEG): container finished" podID="7972aee8-dbdd-4c32-b46d-6cdf855aa53d" containerID="d7bca43b9b15ecff60de5ccd7e6e5372115f4a5f0704486e24f8c2ca77f7b331" exitCode=0
Oct 31 12:54:00.652186 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:00.652164    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b218d3-khq4b" event={"ID":"7972aee8-dbdd-4c32-b46d-6cdf855aa53d","Type":"ContainerDied","Data":"d7bca43b9b15ecff60de5ccd7e6e5372115f4a5f0704486e24f8c2ca77f7b331"}
Oct 31 12:54:00.652642 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:00.652589    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-71bc05-openshift-default-dd6bc8b77-b65mw"
Oct 31 12:54:00.653772 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:00.653753    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-71bc05-openshift-default-dd6bc8b77-b65mw"
Oct 31 12:54:00.690139 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:00.690090    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1e071a8-vqh87" podStartSLOduration=1.690077762 podStartE2EDuration="1.690077762s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:53:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:54:00.683292549 +0000 UTC m=+1603.786360262" watchObservedRunningTime="2026-10-31 12:54:00.690077762 +0000 UTC m=+1603.793145473"
Oct 31 12:54:00.837638 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:00.837576    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d1721ec-dq9xk" podStartSLOduration=1.8375615920000001 podStartE2EDuration="1.837561592s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:53:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:54:00.836103402 +0000 UTC m=+1603.939171115" watchObservedRunningTime="2026-10-31 12:54:00.837561592 +0000 UTC m=+1603.940629303"
Oct 31 12:54:01.051880 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:01.051844    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d62f8e4-b062-4dd4-a23d-0f66566c83da" path="/var/lib/kubelet/pods/0d62f8e4-b062-4dd4-a23d-0f66566c83da/volumes"
Oct 31 12:54:01.052426 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:01.052408    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e08b08ff-4f40-474a-a103-2c499e999942" path="/var/lib/kubelet/pods/e08b08ff-4f40-474a-a103-2c499e999942/volumes"
Oct 31 12:54:01.663908 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:01.663868    5060 generic.go:356] "Generic (PLEG): container finished" podID="bb1bcc49-26c3-46a9-a134-dffba7b91a58" containerID="223c129273c47e87f1c3e2d957c390796115b371212736d6cecd42833b9091a7" exitCode=0
Oct 31 12:54:01.664387 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:01.663952    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d1721ec-dq9xk" event={"ID":"bb1bcc49-26c3-46a9-a134-dffba7b91a58","Type":"ContainerDied","Data":"223c129273c47e87f1c3e2d957c390796115b371212736d6cecd42833b9091a7"}
Oct 31 12:54:01.665845 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:01.665815    5060 generic.go:356] "Generic (PLEG): container finished" podID="fc885fe1-b05b-48a3-9617-2d9475ec888a" containerID="56971a4a3a477e8095fef84a7d644d486e7fda1a8ac07cd046f5df7327080b1c" exitCode=0
Oct 31 12:54:01.666011 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:01.665896    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1e071a8-vqh87" event={"ID":"fc885fe1-b05b-48a3-9617-2d9475ec888a","Type":"ContainerDied","Data":"56971a4a3a477e8095fef84a7d644d486e7fda1a8ac07cd046f5df7327080b1c"}
Oct 31 12:54:01.871697 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:01.871665    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-fb2b1d-r8vf7"
Oct 31 12:54:01.938387 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:01.938324    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b218d3-khq4b"
Oct 31 12:54:01.953197 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:01.953171    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85ab7b73-48a3-430a-b489-30985757d106-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85ab7b73-48a3-430a-b489-30985757d106-maas-api-openshift-ca\") pod \"85ab7b73-48a3-430a-b489-30985757d106\" (UID: \"85ab7b73-48a3-430a-b489-30985757d106\") "
Oct 31 12:54:01.953364 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:01.953207    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/85ab7b73-48a3-430a-b489-30985757d106-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85ab7b73-48a3-430a-b489-30985757d106-maas-api-xks-ca\") pod \"85ab7b73-48a3-430a-b489-30985757d106\" (UID: \"85ab7b73-48a3-430a-b489-30985757d106\") "
Oct 31 12:54:01.953364 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:01.953259    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7972aee8-dbdd-4c32-b46d-6cdf855aa53d-kube-api-access-wz7lt\" (UniqueName: \"kubernetes.io/projected/7972aee8-dbdd-4c32-b46d-6cdf855aa53d-kube-api-access-wz7lt\") pod \"7972aee8-dbdd-4c32-b46d-6cdf855aa53d\" (UID: \"7972aee8-dbdd-4c32-b46d-6cdf855aa53d\") "
Oct 31 12:54:01.953521 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:01.953400    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7972aee8-dbdd-4c32-b46d-6cdf855aa53d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7972aee8-dbdd-4c32-b46d-6cdf855aa53d-maas-api-xks-ca\") pod \"7972aee8-dbdd-4c32-b46d-6cdf855aa53d\" (UID: \"7972aee8-dbdd-4c32-b46d-6cdf855aa53d\") "
Oct 31 12:54:01.953521 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:01.953419    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85ab7b73-48a3-430a-b489-30985757d106-kube-api-access-khdc9\" (UniqueName: \"kubernetes.io/projected/85ab7b73-48a3-430a-b489-30985757d106-kube-api-access-khdc9\") pod \"85ab7b73-48a3-430a-b489-30985757d106\" (UID: \"85ab7b73-48a3-430a-b489-30985757d106\") "
Oct 31 12:54:01.953521 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:01.953468    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7972aee8-dbdd-4c32-b46d-6cdf855aa53d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7972aee8-dbdd-4c32-b46d-6cdf855aa53d-maas-api-openshift-ca\") pod \"7972aee8-dbdd-4c32-b46d-6cdf855aa53d\" (UID: \"7972aee8-dbdd-4c32-b46d-6cdf855aa53d\") "
Oct 31 12:54:01.953685 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:01.953604    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85ab7b73-48a3-430a-b489-30985757d106-maas-api-openshift-ca" pod "85ab7b73-48a3-430a-b489-30985757d106" (UID: "85ab7b73-48a3-430a-b489-30985757d106"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:54:01.953930 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:01.953900    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7972aee8-dbdd-4c32-b46d-6cdf855aa53d-maas-api-openshift-ca" pod "7972aee8-dbdd-4c32-b46d-6cdf855aa53d" (UID: "7972aee8-dbdd-4c32-b46d-6cdf855aa53d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:54:01.954230 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:01.954210    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7972aee8-dbdd-4c32-b46d-6cdf855aa53d-maas-api-openshift-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:54:01.954300 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:01.954237    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85ab7b73-48a3-430a-b489-30985757d106-maas-api-openshift-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:54:01.955794 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:01.955767    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7972aee8-dbdd-4c32-b46d-6cdf855aa53d-maas-api-xks-ca" pod "7972aee8-dbdd-4c32-b46d-6cdf855aa53d" (UID: "7972aee8-dbdd-4c32-b46d-6cdf855aa53d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:54:01.956037 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:01.956015    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85ab7b73-48a3-430a-b489-30985757d106-maas-api-xks-ca" pod "85ab7b73-48a3-430a-b489-30985757d106" (UID: "85ab7b73-48a3-430a-b489-30985757d106"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:54:01.956177 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:01.956158    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85ab7b73-48a3-430a-b489-30985757d106-kube-api-access-khdc9" pod "85ab7b73-48a3-430a-b489-30985757d106" (UID: "85ab7b73-48a3-430a-b489-30985757d106"). InnerVolumeSpecName "kube-api-access-khdc9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:54:01.957934 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:01.957907    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7972aee8-dbdd-4c32-b46d-6cdf855aa53d-kube-api-access-wz7lt" pod "7972aee8-dbdd-4c32-b46d-6cdf855aa53d" (UID: "7972aee8-dbdd-4c32-b46d-6cdf855aa53d"). InnerVolumeSpecName "kube-api-access-wz7lt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:54:02.055848 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:02.055812    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85ab7b73-48a3-430a-b489-30985757d106-maas-api-xks-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:54:02.055848 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:02.055845    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wz7lt\" (UniqueName: \"kubernetes.io/projected/7972aee8-dbdd-4c32-b46d-6cdf855aa53d-kube-api-access-wz7lt\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:54:02.055848 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:02.055856    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7972aee8-dbdd-4c32-b46d-6cdf855aa53d-maas-api-xks-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:54:02.056122 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:02.055867    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khdc9\" (UniqueName: \"kubernetes.io/projected/85ab7b73-48a3-430a-b489-30985757d106-kube-api-access-khdc9\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:54:02.380514 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:54:02.375783    5060 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod330523c8_06c9_42a2_ac3f_9dd4ad62c392.slice/crio-conmon-d190e7520eebe52b0c749ad77fb372857ff3f35218eb3dc8fea952d63280f3d9.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod330523c8_06c9_42a2_ac3f_9dd4ad62c392.slice/crio-d190e7520eebe52b0c749ad77fb372857ff3f35218eb3dc8fea952d63280f3d9.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1e6c7f5_d872_4549_b950_1e751b6374ab.slice/crio-conmon-15eb93212f666238582935bcfb7fb66c4776d563bb22273bc4eb80596e90659b.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1e6c7f5_d872_4549_b950_1e751b6374ab.slice/crio-15eb93212f666238582935bcfb7fb66c4776d563bb22273bc4eb80596e90659b.scope\": RecentStats: unable to find data in memory cache]"
Oct 31 12:54:02.679923 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:02.679836    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b218d3-khq4b"
Oct 31 12:54:02.680564 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:02.680526    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b218d3-khq4b" event={"ID":"7972aee8-dbdd-4c32-b46d-6cdf855aa53d","Type":"ContainerDied","Data":"c147293f3e32133c4543425bbe589eaecea2ef6b307ef49eb3d8a3b60560a083"}
Oct 31 12:54:02.680702 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:02.680592    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c147293f3e32133c4543425bbe589eaecea2ef6b307ef49eb3d8a3b60560a083"
Oct 31 12:54:02.686076 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:02.682409    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-fb2b1d-r8vf7"
Oct 31 12:54:02.686076 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:02.682412    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-fb2b1d-r8vf7" event={"ID":"85ab7b73-48a3-430a-b489-30985757d106","Type":"ContainerDied","Data":"15934746aa14a7b39208d1c8c57ededc7e501366f9ddd485c70f0e7c15207cd7"}
Oct 31 12:54:02.686076 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:02.682551    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="15934746aa14a7b39208d1c8c57ededc7e501366f9ddd485c70f0e7c15207cd7"
Oct 31 12:54:02.908988 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:02.908960    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d1721ec-dq9xk"
Oct 31 12:54:02.942476 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:02.942422    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1e071a8-vqh87"
Oct 31 12:54:02.989203 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:02.989168    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc885fe1-b05b-48a3-9617-2d9475ec888a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc885fe1-b05b-48a3-9617-2d9475ec888a-maas-api-xks-ca\") pod \"fc885fe1-b05b-48a3-9617-2d9475ec888a\" (UID: \"fc885fe1-b05b-48a3-9617-2d9475ec888a\") "
Oct 31 12:54:02.995853 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:02.989213    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc885fe1-b05b-48a3-9617-2d9475ec888a-kube-api-access-m8zp5\" (UniqueName: \"kubernetes.io/projected/fc885fe1-b05b-48a3-9617-2d9475ec888a-kube-api-access-m8zp5\") pod \"fc885fe1-b05b-48a3-9617-2d9475ec888a\" (UID: \"fc885fe1-b05b-48a3-9617-2d9475ec888a\") "
Oct 31 12:54:02.995853 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:02.989269    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bb1bcc49-26c3-46a9-a134-dffba7b91a58-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb1bcc49-26c3-46a9-a134-dffba7b91a58-maas-api-xks-ca\") pod \"bb1bcc49-26c3-46a9-a134-dffba7b91a58\" (UID: \"bb1bcc49-26c3-46a9-a134-dffba7b91a58\") "
Oct 31 12:54:02.995853 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:02.989358    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc885fe1-b05b-48a3-9617-2d9475ec888a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc885fe1-b05b-48a3-9617-2d9475ec888a-maas-api-openshift-ca\") pod \"fc885fe1-b05b-48a3-9617-2d9475ec888a\" (UID: \"fc885fe1-b05b-48a3-9617-2d9475ec888a\") "
Oct 31 12:54:02.995853 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:02.989416    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb1bcc49-26c3-46a9-a134-dffba7b91a58-kube-api-access-srps7\" (UniqueName: \"kubernetes.io/projected/bb1bcc49-26c3-46a9-a134-dffba7b91a58-kube-api-access-srps7\") pod \"bb1bcc49-26c3-46a9-a134-dffba7b91a58\" (UID: \"bb1bcc49-26c3-46a9-a134-dffba7b91a58\") "
Oct 31 12:54:02.995853 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:02.989627    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb1bcc49-26c3-46a9-a134-dffba7b91a58-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb1bcc49-26c3-46a9-a134-dffba7b91a58-maas-api-openshift-ca\") pod \"bb1bcc49-26c3-46a9-a134-dffba7b91a58\" (UID: \"bb1bcc49-26c3-46a9-a134-dffba7b91a58\") "
Oct 31 12:54:02.995853 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:02.991922    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc885fe1-b05b-48a3-9617-2d9475ec888a-maas-api-openshift-ca" pod "fc885fe1-b05b-48a3-9617-2d9475ec888a" (UID: "fc885fe1-b05b-48a3-9617-2d9475ec888a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:54:02.995853 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:02.994728    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb1bcc49-26c3-46a9-a134-dffba7b91a58-maas-api-openshift-ca" pod "bb1bcc49-26c3-46a9-a134-dffba7b91a58" (UID: "bb1bcc49-26c3-46a9-a134-dffba7b91a58"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:54:02.997297 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:02.997022    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb1bcc49-26c3-46a9-a134-dffba7b91a58-maas-api-xks-ca" pod "bb1bcc49-26c3-46a9-a134-dffba7b91a58" (UID: "bb1bcc49-26c3-46a9-a134-dffba7b91a58"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:54:02.998904 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:02.998875    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc885fe1-b05b-48a3-9617-2d9475ec888a-kube-api-access-m8zp5" pod "fc885fe1-b05b-48a3-9617-2d9475ec888a" (UID: "fc885fe1-b05b-48a3-9617-2d9475ec888a"). InnerVolumeSpecName "kube-api-access-m8zp5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:54:02.999361 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:02.999335    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb1bcc49-26c3-46a9-a134-dffba7b91a58-kube-api-access-srps7" pod "bb1bcc49-26c3-46a9-a134-dffba7b91a58" (UID: "bb1bcc49-26c3-46a9-a134-dffba7b91a58"). InnerVolumeSpecName "kube-api-access-srps7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:54:03.003371 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:03.003321    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc885fe1-b05b-48a3-9617-2d9475ec888a-maas-api-xks-ca" pod "fc885fe1-b05b-48a3-9617-2d9475ec888a" (UID: "fc885fe1-b05b-48a3-9617-2d9475ec888a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:54:03.092080 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:03.092035    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc885fe1-b05b-48a3-9617-2d9475ec888a-maas-api-xks-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:54:03.092080 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:03.092073    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8zp5\" (UniqueName: \"kubernetes.io/projected/fc885fe1-b05b-48a3-9617-2d9475ec888a-kube-api-access-m8zp5\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:54:03.092295 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:03.092088    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb1bcc49-26c3-46a9-a134-dffba7b91a58-maas-api-xks-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:54:03.092295 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:03.092102    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc885fe1-b05b-48a3-9617-2d9475ec888a-maas-api-openshift-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:54:03.092295 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:03.092119    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-srps7\" (UniqueName: \"kubernetes.io/projected/bb1bcc49-26c3-46a9-a134-dffba7b91a58-kube-api-access-srps7\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:54:03.092295 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:03.092132    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb1bcc49-26c3-46a9-a134-dffba7b91a58-maas-api-openshift-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:54:03.704759 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:03.704728    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d1721ec-dq9xk"
Oct 31 12:54:03.705425 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:03.704753    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d1721ec-dq9xk" event={"ID":"bb1bcc49-26c3-46a9-a134-dffba7b91a58","Type":"ContainerDied","Data":"0309901f13e0c278608ae1ecbac3c24fecab1b0fb03a0c87db6a1340a224497a"}
Oct 31 12:54:03.705567 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:03.705465    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0309901f13e0c278608ae1ecbac3c24fecab1b0fb03a0c87db6a1340a224497a"
Oct 31 12:54:03.709567 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:03.707738    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1e071a8-vqh87" event={"ID":"fc885fe1-b05b-48a3-9617-2d9475ec888a","Type":"ContainerDied","Data":"73db0c62cfdf32d7119526a1d6edcbcec0882a3badb8f5dedf60f66266a78f2a"}
Oct 31 12:54:03.709567 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:03.707790    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="73db0c62cfdf32d7119526a1d6edcbcec0882a3badb8f5dedf60f66266a78f2a"
Oct 31 12:54:03.709567 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:03.707852    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1e071a8-vqh87"
Oct 31 12:54:04.935649 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:04.935601    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-85ce08-5f656fc7f9-z5g9p" podUID="b03c9dc3-2388-458b-8e7f-09cf2811021f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 12:54:07.250052 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:07.250011    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-b218d3-766c68ffbf-76pwg"]
Oct 31 12:54:09.915826 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:09.915786    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-zswkp"]
Oct 31 12:54:09.916285 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:09.915997    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-zswkp" podUID="90387d5f-2bb8-4922-a1ad-9466e0745aaf" containerName="istio-proxy" containerID="cri-o://0b8ba48fe21bde7067b3ae3fffa99971a3a916655997de34860726a10b9a24de" gracePeriod=30
Oct 31 12:54:10.593596 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:54:10.593552    5060 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1e6c7f5_d872_4549_b950_1e751b6374ab.slice/crio-15eb93212f666238582935bcfb7fb66c4776d563bb22273bc4eb80596e90659b.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod330523c8_06c9_42a2_ac3f_9dd4ad62c392.slice/crio-d190e7520eebe52b0c749ad77fb372857ff3f35218eb3dc8fea952d63280f3d9.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod330523c8_06c9_42a2_ac3f_9dd4ad62c392.slice/crio-conmon-d190e7520eebe52b0c749ad77fb372857ff3f35218eb3dc8fea952d63280f3d9.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1e6c7f5_d872_4549_b950_1e751b6374ab.slice/crio-conmon-15eb93212f666238582935bcfb7fb66c4776d563bb22273bc4eb80596e90659b.scope\": RecentStats: unable to find data in memory cache]"
Oct 31 12:54:12.250767 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:12.250727    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-b218d3-766c68ffbf-76pwg" podUID="2ec2c204-58e3-4d0f-b06b-5112806add83" containerName="payload-processing" containerID="cri-o://3990f663560e6baf90c55d56bae73742e4746e910ea63c074d0ad6670e4db361" gracePeriod=25
Oct 31 12:54:13.614509 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:13.614010    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2c393f-77f8c6697-l7t2b"]
Oct 31 12:54:13.615005 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:13.614978    5060 cpu_manager.go:414] "containerMap: removing container" podUID="0d62f8e4-b062-4dd4-a23d-0f66566c83da" containerName="istio-proxy"
Oct 31 12:54:13.615005 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:13.614994    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d62f8e4-b062-4dd4-a23d-0f66566c83da" containerName="istio-proxy"
Oct 31 12:54:13.615123 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:13.615012    5060 cpu_manager.go:414] "containerMap: removing container" podUID="fc885fe1-b05b-48a3-9617-2d9475ec888a" containerName="revoke-keys"
Oct 31 12:54:13.615123 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:13.615017    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc885fe1-b05b-48a3-9617-2d9475ec888a" containerName="revoke-keys"
Oct 31 12:54:13.615123 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:13.615032    5060 cpu_manager.go:414] "containerMap: removing container" podUID="bb1bcc49-26c3-46a9-a134-dffba7b91a58" containerName="revoke-keys"
Oct 31 12:54:13.615123 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:13.615037    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb1bcc49-26c3-46a9-a134-dffba7b91a58" containerName="revoke-keys"
Oct 31 12:54:13.615123 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:13.615046    5060 cpu_manager.go:414] "containerMap: removing container" podUID="7972aee8-dbdd-4c32-b46d-6cdf855aa53d" containerName="revoke-keys"
Oct 31 12:54:13.615123 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:13.615051    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7972aee8-dbdd-4c32-b46d-6cdf855aa53d" containerName="revoke-keys"
Oct 31 12:54:13.615123 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:13.615061    5060 cpu_manager.go:414] "containerMap: removing container" podUID="e08b08ff-4f40-474a-a103-2c499e999942" containerName="istio-proxy"
Oct 31 12:54:13.615123 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:13.615065    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e08b08ff-4f40-474a-a103-2c499e999942" containerName="istio-proxy"
Oct 31 12:54:13.615123 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:13.615085    5060 cpu_manager.go:414] "containerMap: removing container" podUID="e08b08ff-4f40-474a-a103-2c499e999942" containerName="istio-proxy"
Oct 31 12:54:13.615123 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:13.615089    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e08b08ff-4f40-474a-a103-2c499e999942" containerName="istio-proxy"
Oct 31 12:54:13.615123 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:13.615097    5060 cpu_manager.go:414] "containerMap: removing container" podUID="85ab7b73-48a3-430a-b489-30985757d106" containerName="revoke-keys"
Oct 31 12:54:13.615123 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:13.615102    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85ab7b73-48a3-430a-b489-30985757d106" containerName="revoke-keys"
Oct 31 12:54:13.615820 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:13.615214    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc885fe1-b05b-48a3-9617-2d9475ec888a" containerName="revoke-keys"
Oct 31 12:54:13.615820 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:13.615225    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb1bcc49-26c3-46a9-a134-dffba7b91a58" containerName="revoke-keys"
Oct 31 12:54:13.615820 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:13.615231    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="7972aee8-dbdd-4c32-b46d-6cdf855aa53d" containerName="revoke-keys"
Oct 31 12:54:13.615820 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:13.615238    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="e08b08ff-4f40-474a-a103-2c499e999942" containerName="istio-proxy"
Oct 31 12:54:13.615820 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:13.615245    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="e08b08ff-4f40-474a-a103-2c499e999942" containerName="istio-proxy"
Oct 31 12:54:13.615820 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:13.615252    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="85ab7b73-48a3-430a-b489-30985757d106" containerName="revoke-keys"
Oct 31 12:54:13.615820 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:13.615262    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d62f8e4-b062-4dd4-a23d-0f66566c83da" containerName="istio-proxy"
Oct 31 12:54:13.619960 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:13.619931    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2c393f-77f8c6697-l7t2b"
Oct 31 12:54:13.632284 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:13.631833    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xl9jd\" (UniqueName: \"kubernetes.io/projected/6a1124a2-55e6-424f-a91e-9f89f37db017-kube-api-access-xl9jd\") pod \"maas-api-e2e-inf-b-2c393f-77f8c6697-l7t2b\" (UID: \"6a1124a2-55e6-424f-a91e-9f89f37db017\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2c393f-77f8c6697-l7t2b"
Oct 31 12:54:13.632284 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:13.631908    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a1124a2-55e6-424f-a91e-9f89f37db017-openshift-service-ca\") pod \"maas-api-e2e-inf-b-2c393f-77f8c6697-l7t2b\" (UID: \"6a1124a2-55e6-424f-a91e-9f89f37db017\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2c393f-77f8c6697-l7t2b"
Oct 31 12:54:13.632284 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:13.632055    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a1124a2-55e6-424f-a91e-9f89f37db017-metrics-certs\") pod \"maas-api-e2e-inf-b-2c393f-77f8c6697-l7t2b\" (UID: \"6a1124a2-55e6-424f-a91e-9f89f37db017\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2c393f-77f8c6697-l7t2b"
Oct 31 12:54:13.632284 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:13.632083    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a1124a2-55e6-424f-a91e-9f89f37db017-maas-api-tls\") pod \"maas-api-e2e-inf-b-2c393f-77f8c6697-l7t2b\" (UID: \"6a1124a2-55e6-424f-a91e-9f89f37db017\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2c393f-77f8c6697-l7t2b"
Oct 31 12:54:13.636690 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:13.636666    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-2c393f-serving-cert\""
Oct 31 12:54:13.651125 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:13.651090    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2c393f-77f8c6697-l7t2b"]
Oct 31 12:54:13.733077 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:13.733041    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xl9jd\" (UniqueName: \"kubernetes.io/projected/6a1124a2-55e6-424f-a91e-9f89f37db017-kube-api-access-xl9jd\") pod \"maas-api-e2e-inf-b-2c393f-77f8c6697-l7t2b\" (UID: \"6a1124a2-55e6-424f-a91e-9f89f37db017\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2c393f-77f8c6697-l7t2b"
Oct 31 12:54:13.733250 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:13.733090    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a1124a2-55e6-424f-a91e-9f89f37db017-openshift-service-ca\") pod \"maas-api-e2e-inf-b-2c393f-77f8c6697-l7t2b\" (UID: \"6a1124a2-55e6-424f-a91e-9f89f37db017\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2c393f-77f8c6697-l7t2b"
Oct 31 12:54:13.733250 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:13.733168    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a1124a2-55e6-424f-a91e-9f89f37db017-metrics-certs\") pod \"maas-api-e2e-inf-b-2c393f-77f8c6697-l7t2b\" (UID: \"6a1124a2-55e6-424f-a91e-9f89f37db017\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2c393f-77f8c6697-l7t2b"
Oct 31 12:54:13.733250 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:13.733195    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a1124a2-55e6-424f-a91e-9f89f37db017-maas-api-tls\") pod \"maas-api-e2e-inf-b-2c393f-77f8c6697-l7t2b\" (UID: \"6a1124a2-55e6-424f-a91e-9f89f37db017\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2c393f-77f8c6697-l7t2b"
Oct 31 12:54:13.733412 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:54:13.733384    5060 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-2c393f-serving-cert: secret "maas-api-e2e-inf-b-2c393f-serving-cert" not found
Oct 31 12:54:13.733502 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:54:13.733478    5060 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a1124a2-55e6-424f-a91e-9f89f37db017-maas-api-tls podName:6a1124a2-55e6-424f-a91e-9f89f37db017 nodeName:}" failed. No retries permitted until 2026-10-31 12:54:14.233457301 +0000 UTC m=+1617.336524996 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6a1124a2-55e6-424f-a91e-9f89f37db017-maas-api-tls") pod "maas-api-e2e-inf-b-2c393f-77f8c6697-l7t2b" (UID: "6a1124a2-55e6-424f-a91e-9f89f37db017") : secret "maas-api-e2e-inf-b-2c393f-serving-cert" not found
Oct 31 12:54:13.733809 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:13.733785    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a1124a2-55e6-424f-a91e-9f89f37db017-openshift-service-ca\") pod \"maas-api-e2e-inf-b-2c393f-77f8c6697-l7t2b\" (UID: \"6a1124a2-55e6-424f-a91e-9f89f37db017\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2c393f-77f8c6697-l7t2b"
Oct 31 12:54:13.735685 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:13.735664    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a1124a2-55e6-424f-a91e-9f89f37db017-metrics-certs\") pod \"maas-api-e2e-inf-b-2c393f-77f8c6697-l7t2b\" (UID: \"6a1124a2-55e6-424f-a91e-9f89f37db017\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2c393f-77f8c6697-l7t2b"
Oct 31 12:54:13.777463 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:13.777425    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xl9jd\" (UniqueName: \"kubernetes.io/projected/6a1124a2-55e6-424f-a91e-9f89f37db017-kube-api-access-xl9jd\") pod \"maas-api-e2e-inf-b-2c393f-77f8c6697-l7t2b\" (UID: \"6a1124a2-55e6-424f-a91e-9f89f37db017\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2c393f-77f8c6697-l7t2b"
Oct 31 12:54:14.240651 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:14.240603    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a1124a2-55e6-424f-a91e-9f89f37db017-maas-api-tls\") pod \"maas-api-e2e-inf-b-2c393f-77f8c6697-l7t2b\" (UID: \"6a1124a2-55e6-424f-a91e-9f89f37db017\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2c393f-77f8c6697-l7t2b"
Oct 31 12:54:14.243116 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:14.243095    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a1124a2-55e6-424f-a91e-9f89f37db017-maas-api-tls\") pod \"maas-api-e2e-inf-b-2c393f-77f8c6697-l7t2b\" (UID: \"6a1124a2-55e6-424f-a91e-9f89f37db017\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2c393f-77f8c6697-l7t2b"
Oct 31 12:54:14.256548 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:14.256040    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2c393f-77f8c6697-l7t2b"
Oct 31 12:54:14.745512 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:14.742605    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-fd988c-798cc7df69-jdx5x"]
Oct 31 12:54:14.770510 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:14.767049    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fd988c-798cc7df69-jdx5x"
Oct 31 12:54:14.780509 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:14.775741    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-fd988c-798cc7df69-jdx5x"]
Oct 31 12:54:14.787527 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:14.784859    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-fd988c-serving-cert\""
Oct 31 12:54:14.825850 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:14.825777    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2c393f-77f8c6697-l7t2b"]
Oct 31 12:54:14.849629 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:14.849607    5060 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 12:54:14.870875 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:14.867554    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d442c3e-cd82-423f-92ba-d10ca01bc97b-metrics-certs\") pod \"maas-api-e2e-inf-b-fd988c-798cc7df69-jdx5x\" (UID: \"9d442c3e-cd82-423f-92ba-d10ca01bc97b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fd988c-798cc7df69-jdx5x"
Oct 31 12:54:14.870875 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:14.867607    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxj2x\" (UniqueName: \"kubernetes.io/projected/9d442c3e-cd82-423f-92ba-d10ca01bc97b-kube-api-access-pxj2x\") pod \"maas-api-e2e-inf-b-fd988c-798cc7df69-jdx5x\" (UID: \"9d442c3e-cd82-423f-92ba-d10ca01bc97b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fd988c-798cc7df69-jdx5x"
Oct 31 12:54:14.870875 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:14.867698    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d442c3e-cd82-423f-92ba-d10ca01bc97b-maas-api-tls\") pod \"maas-api-e2e-inf-b-fd988c-798cc7df69-jdx5x\" (UID: \"9d442c3e-cd82-423f-92ba-d10ca01bc97b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fd988c-798cc7df69-jdx5x"
Oct 31 12:54:14.870875 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:14.867727    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d442c3e-cd82-423f-92ba-d10ca01bc97b-openshift-service-ca\") pod \"maas-api-e2e-inf-b-fd988c-798cc7df69-jdx5x\" (UID: \"9d442c3e-cd82-423f-92ba-d10ca01bc97b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fd988c-798cc7df69-jdx5x"
Oct 31 12:54:14.935616 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:14.930692    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-85ce08-5f656fc7f9-z5g9p" podUID="b03c9dc3-2388-458b-8e7f-09cf2811021f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 12:54:14.974539 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:14.970803    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d442c3e-cd82-423f-92ba-d10ca01bc97b-metrics-certs\") pod \"maas-api-e2e-inf-b-fd988c-798cc7df69-jdx5x\" (UID: \"9d442c3e-cd82-423f-92ba-d10ca01bc97b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fd988c-798cc7df69-jdx5x"
Oct 31 12:54:14.974539 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:14.970861    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxj2x\" (UniqueName: \"kubernetes.io/projected/9d442c3e-cd82-423f-92ba-d10ca01bc97b-kube-api-access-pxj2x\") pod \"maas-api-e2e-inf-b-fd988c-798cc7df69-jdx5x\" (UID: \"9d442c3e-cd82-423f-92ba-d10ca01bc97b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fd988c-798cc7df69-jdx5x"
Oct 31 12:54:14.974539 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:14.970936    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d442c3e-cd82-423f-92ba-d10ca01bc97b-maas-api-tls\") pod \"maas-api-e2e-inf-b-fd988c-798cc7df69-jdx5x\" (UID: \"9d442c3e-cd82-423f-92ba-d10ca01bc97b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fd988c-798cc7df69-jdx5x"
Oct 31 12:54:14.974539 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:14.970964    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d442c3e-cd82-423f-92ba-d10ca01bc97b-openshift-service-ca\") pod \"maas-api-e2e-inf-b-fd988c-798cc7df69-jdx5x\" (UID: \"9d442c3e-cd82-423f-92ba-d10ca01bc97b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fd988c-798cc7df69-jdx5x"
Oct 31 12:54:14.974539 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:14.972052    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d442c3e-cd82-423f-92ba-d10ca01bc97b-openshift-service-ca\") pod \"maas-api-e2e-inf-b-fd988c-798cc7df69-jdx5x\" (UID: \"9d442c3e-cd82-423f-92ba-d10ca01bc97b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fd988c-798cc7df69-jdx5x"
Oct 31 12:54:15.016057 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:54:14.990664    5060 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-fd988c-serving-cert: secret "maas-api-e2e-inf-b-fd988c-serving-cert" not found
Oct 31 12:54:15.016057 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:54:14.990743    5060 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d442c3e-cd82-423f-92ba-d10ca01bc97b-maas-api-tls podName:9d442c3e-cd82-423f-92ba-d10ca01bc97b nodeName:}" failed. No retries permitted until 2026-10-31 12:54:15.490723174 +0000 UTC m=+1618.593790865 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9d442c3e-cd82-423f-92ba-d10ca01bc97b-maas-api-tls") pod "maas-api-e2e-inf-b-fd988c-798cc7df69-jdx5x" (UID: "9d442c3e-cd82-423f-92ba-d10ca01bc97b") : secret "maas-api-e2e-inf-b-fd988c-serving-cert" not found
Oct 31 12:54:15.016057 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:15.005447    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d442c3e-cd82-423f-92ba-d10ca01bc97b-metrics-certs\") pod \"maas-api-e2e-inf-b-fd988c-798cc7df69-jdx5x\" (UID: \"9d442c3e-cd82-423f-92ba-d10ca01bc97b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fd988c-798cc7df69-jdx5x"
Oct 31 12:54:15.036167 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:15.036120    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxj2x\" (UniqueName: \"kubernetes.io/projected/9d442c3e-cd82-423f-92ba-d10ca01bc97b-kube-api-access-pxj2x\") pod \"maas-api-e2e-inf-b-fd988c-798cc7df69-jdx5x\" (UID: \"9d442c3e-cd82-423f-92ba-d10ca01bc97b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fd988c-798cc7df69-jdx5x"
Oct 31 12:54:15.444828 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:15.426327    5060 generic.go:356] "Generic (PLEG): container finished" podID="90387d5f-2bb8-4922-a1ad-9466e0745aaf" containerID="0b8ba48fe21bde7067b3ae3fffa99971a3a916655997de34860726a10b9a24de" exitCode=0
Oct 31 12:54:15.444828 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:15.426852    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-zswkp" event={"ID":"90387d5f-2bb8-4922-a1ad-9466e0745aaf","Type":"ContainerDied","Data":"0b8ba48fe21bde7067b3ae3fffa99971a3a916655997de34860726a10b9a24de"}
Oct 31 12:54:15.444828 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:15.431772    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2c393f-77f8c6697-l7t2b" event={"ID":"6a1124a2-55e6-424f-a91e-9f89f37db017","Type":"ContainerStarted","Data":"ad6494eeea14f1efe5544eb7116cb6812f6feb98923df7aaebfc647a561e707c"}
Oct 31 12:54:15.598641 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:15.587397    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d442c3e-cd82-423f-92ba-d10ca01bc97b-maas-api-tls\") pod \"maas-api-e2e-inf-b-fd988c-798cc7df69-jdx5x\" (UID: \"9d442c3e-cd82-423f-92ba-d10ca01bc97b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fd988c-798cc7df69-jdx5x"
Oct 31 12:54:15.665456 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:15.665422    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d442c3e-cd82-423f-92ba-d10ca01bc97b-maas-api-tls\") pod \"maas-api-e2e-inf-b-fd988c-798cc7df69-jdx5x\" (UID: \"9d442c3e-cd82-423f-92ba-d10ca01bc97b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fd988c-798cc7df69-jdx5x"
Oct 31 12:54:15.733839 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:15.733393    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fd988c-798cc7df69-jdx5x"
Oct 31 12:54:16.073655 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:16.073096    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-zswkp"
Oct 31 12:54:16.128749 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:16.128713    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90387d5f-2bb8-4922-a1ad-9466e0745aaf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90387d5f-2bb8-4922-a1ad-9466e0745aaf-workload-certs\") pod \"90387d5f-2bb8-4922-a1ad-9466e0745aaf\" (UID: \"90387d5f-2bb8-4922-a1ad-9466e0745aaf\") "
Oct 31 12:54:16.128928 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:16.128807    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90387d5f-2bb8-4922-a1ad-9466e0745aaf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90387d5f-2bb8-4922-a1ad-9466e0745aaf-workload-socket\") pod \"90387d5f-2bb8-4922-a1ad-9466e0745aaf\" (UID: \"90387d5f-2bb8-4922-a1ad-9466e0745aaf\") "
Oct 31 12:54:16.128928 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:16.128906    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/90387d5f-2bb8-4922-a1ad-9466e0745aaf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90387d5f-2bb8-4922-a1ad-9466e0745aaf-istio-podinfo\") pod \"90387d5f-2bb8-4922-a1ad-9466e0745aaf\" (UID: \"90387d5f-2bb8-4922-a1ad-9466e0745aaf\") "
Oct 31 12:54:16.129045 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:16.129004    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90387d5f-2bb8-4922-a1ad-9466e0745aaf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90387d5f-2bb8-4922-a1ad-9466e0745aaf-credential-socket\") pod \"90387d5f-2bb8-4922-a1ad-9466e0745aaf\" (UID: \"90387d5f-2bb8-4922-a1ad-9466e0745aaf\") "
Oct 31 12:54:16.129131 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:16.129113    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90387d5f-2bb8-4922-a1ad-9466e0745aaf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90387d5f-2bb8-4922-a1ad-9466e0745aaf-istio-envoy\") pod \"90387d5f-2bb8-4922-a1ad-9466e0745aaf\" (UID: \"90387d5f-2bb8-4922-a1ad-9466e0745aaf\") "
Oct 31 12:54:16.129287 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:16.129266    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90387d5f-2bb8-4922-a1ad-9466e0745aaf-kube-api-access-v6m7m\" (UniqueName: \"kubernetes.io/projected/90387d5f-2bb8-4922-a1ad-9466e0745aaf-kube-api-access-v6m7m\") pod \"90387d5f-2bb8-4922-a1ad-9466e0745aaf\" (UID: \"90387d5f-2bb8-4922-a1ad-9466e0745aaf\") "
Oct 31 12:54:16.129378 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:16.129346    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90387d5f-2bb8-4922-a1ad-9466e0745aaf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90387d5f-2bb8-4922-a1ad-9466e0745aaf-istiod-ca-cert\") pod \"90387d5f-2bb8-4922-a1ad-9466e0745aaf\" (UID: \"90387d5f-2bb8-4922-a1ad-9466e0745aaf\") "
Oct 31 12:54:16.129680 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:16.129615    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90387d5f-2bb8-4922-a1ad-9466e0745aaf-istio-token\" (UniqueName: \"kubernetes.io/projected/90387d5f-2bb8-4922-a1ad-9466e0745aaf-istio-token\") pod \"90387d5f-2bb8-4922-a1ad-9466e0745aaf\" (UID: \"90387d5f-2bb8-4922-a1ad-9466e0745aaf\") "
Oct 31 12:54:16.129931 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:16.129912    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90387d5f-2bb8-4922-a1ad-9466e0745aaf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90387d5f-2bb8-4922-a1ad-9466e0745aaf-istio-data\") pod \"90387d5f-2bb8-4922-a1ad-9466e0745aaf\" (UID: \"90387d5f-2bb8-4922-a1ad-9466e0745aaf\") "
Oct 31 12:54:16.131105 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:16.131084    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90387d5f-2bb8-4922-a1ad-9466e0745aaf-istio-data" pod "90387d5f-2bb8-4922-a1ad-9466e0745aaf" (UID: "90387d5f-2bb8-4922-a1ad-9466e0745aaf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:54:16.131390 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:16.131365    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90387d5f-2bb8-4922-a1ad-9466e0745aaf-credential-socket" pod "90387d5f-2bb8-4922-a1ad-9466e0745aaf" (UID: "90387d5f-2bb8-4922-a1ad-9466e0745aaf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:54:16.131828 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:16.131805    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90387d5f-2bb8-4922-a1ad-9466e0745aaf-workload-certs" pod "90387d5f-2bb8-4922-a1ad-9466e0745aaf" (UID: "90387d5f-2bb8-4922-a1ad-9466e0745aaf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:54:16.132348 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:16.132307    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90387d5f-2bb8-4922-a1ad-9466e0745aaf-workload-socket" pod "90387d5f-2bb8-4922-a1ad-9466e0745aaf" (UID: "90387d5f-2bb8-4922-a1ad-9466e0745aaf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:54:16.133113 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:16.133087    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90387d5f-2bb8-4922-a1ad-9466e0745aaf-istiod-ca-cert" pod "90387d5f-2bb8-4922-a1ad-9466e0745aaf" (UID: "90387d5f-2bb8-4922-a1ad-9466e0745aaf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:54:16.135725 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:16.135686    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/90387d5f-2bb8-4922-a1ad-9466e0745aaf-istio-podinfo" pod "90387d5f-2bb8-4922-a1ad-9466e0745aaf" (UID: "90387d5f-2bb8-4922-a1ad-9466e0745aaf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 12:54:16.136797 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:16.136761    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90387d5f-2bb8-4922-a1ad-9466e0745aaf-kube-api-access-v6m7m" pod "90387d5f-2bb8-4922-a1ad-9466e0745aaf" (UID: "90387d5f-2bb8-4922-a1ad-9466e0745aaf"). InnerVolumeSpecName "kube-api-access-v6m7m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:54:16.138869 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:16.138816    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90387d5f-2bb8-4922-a1ad-9466e0745aaf-istio-envoy" pod "90387d5f-2bb8-4922-a1ad-9466e0745aaf" (UID: "90387d5f-2bb8-4922-a1ad-9466e0745aaf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:54:16.140806 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:16.140783    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90387d5f-2bb8-4922-a1ad-9466e0745aaf-istio-token" pod "90387d5f-2bb8-4922-a1ad-9466e0745aaf" (UID: "90387d5f-2bb8-4922-a1ad-9466e0745aaf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:54:16.162320 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:16.162286    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-fd988c-798cc7df69-jdx5x"]
Oct 31 12:54:16.167975 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:54:16.167938    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d442c3e_cd82_423f_92ba_d10ca01bc97b.slice/crio-1c56b7a3074ab562ee0bc3aa7051f11a5c54eda8e3afb03eb3e14caf7180f123 WatchSource:0}: Error finding container 1c56b7a3074ab562ee0bc3aa7051f11a5c54eda8e3afb03eb3e14caf7180f123: Status 404 returned error can't find the container with id 1c56b7a3074ab562ee0bc3aa7051f11a5c54eda8e3afb03eb3e14caf7180f123
Oct 31 12:54:16.234246 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:16.233007    5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90387d5f-2bb8-4922-a1ad-9466e0745aaf-istio-podinfo\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:54:16.234246 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:16.233072    5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90387d5f-2bb8-4922-a1ad-9466e0745aaf-credential-socket\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:54:16.234246 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:16.233091    5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90387d5f-2bb8-4922-a1ad-9466e0745aaf-istio-envoy\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:54:16.234246 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:16.233109    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6m7m\" (UniqueName: \"kubernetes.io/projected/90387d5f-2bb8-4922-a1ad-9466e0745aaf-kube-api-access-v6m7m\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:54:16.234246 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:16.233123    5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90387d5f-2bb8-4922-a1ad-9466e0745aaf-istiod-ca-cert\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:54:16.234246 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:16.233136    5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90387d5f-2bb8-4922-a1ad-9466e0745aaf-istio-token\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:54:16.234246 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:16.233150    5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90387d5f-2bb8-4922-a1ad-9466e0745aaf-istio-data\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:54:16.234246 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:16.233163    5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90387d5f-2bb8-4922-a1ad-9466e0745aaf-workload-certs\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:54:16.234246 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:16.233176    5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90387d5f-2bb8-4922-a1ad-9466e0745aaf-workload-socket\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:54:16.576250 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:16.576210    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2c393f-77f8c6697-l7t2b" event={"ID":"6a1124a2-55e6-424f-a91e-9f89f37db017","Type":"ContainerStarted","Data":"1888c2e313cec61c6abe3682b20c02f48554be8b77acf81a7b0a7138a3f2da43"}
Oct 31 12:54:16.577660 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:16.577624    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2c393f-77f8c6697-l7t2b"
Oct 31 12:54:16.579790 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:16.579763    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fd988c-798cc7df69-jdx5x" event={"ID":"9d442c3e-cd82-423f-92ba-d10ca01bc97b","Type":"ContainerStarted","Data":"1c56b7a3074ab562ee0bc3aa7051f11a5c54eda8e3afb03eb3e14caf7180f123"}
Oct 31 12:54:16.582875 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:16.582851    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-zswkp" event={"ID":"90387d5f-2bb8-4922-a1ad-9466e0745aaf","Type":"ContainerDied","Data":"d9df11aeb8265c983c2221e0969335a6b73f2f895ee4add240070d9c77dca3cf"}
Oct 31 12:54:16.583063 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:16.583046    5060 scope.go:122] "RemoveContainer" containerID="0b8ba48fe21bde7067b3ae3fffa99971a3a916655997de34860726a10b9a24de"
Oct 31 12:54:16.583503 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:16.583472    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-zswkp"
Oct 31 12:54:16.640946 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:16.635622    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2c393f-77f8c6697-l7t2b" podStartSLOduration=2.579003944 podStartE2EDuration="3.635598697s" totalImagesPullingTime="1.056594753s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:54:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:54:16.624900575 +0000 UTC m=+1619.727968311" watchObservedRunningTime="2026-10-31 12:54:16.635598697 +0000 UTC m=+1619.738666407"
Oct 31 12:54:16.738907 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:16.738872    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-zswkp"]
Oct 31 12:54:16.777425 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:16.777382    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-zswkp"]
Oct 31 12:54:17.234560 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:17.228123    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90387d5f-2bb8-4922-a1ad-9466e0745aaf" path="/var/lib/kubelet/pods/90387d5f-2bb8-4922-a1ad-9466e0745aaf/volumes"
Oct 31 12:54:17.674309 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:17.674244    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fd988c-798cc7df69-jdx5x" event={"ID":"9d442c3e-cd82-423f-92ba-d10ca01bc97b","Type":"ContainerStarted","Data":"f403d67a50f834db69ffd1a2b6d2fd9edd1dac184afc268aad5b9ec2676c10a2"}
Oct 31 12:54:17.675631 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:17.675604    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fd988c-798cc7df69-jdx5x"
Oct 31 12:54:17.707062 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:17.707001    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fd988c-798cc7df69-jdx5x" podStartSLOduration=3.087764403 podStartE2EDuration="3.706982273s" totalImagesPullingTime="619.21787ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:54:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:54:17.702157196 +0000 UTC m=+1620.805224908" watchObservedRunningTime="2026-10-31 12:54:17.706982273 +0000 UTC m=+1620.810049986"
Oct 31 12:54:17.725467 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:54:17.725426    5060 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1e6c7f5_d872_4549_b950_1e751b6374ab.slice/crio-15eb93212f666238582935bcfb7fb66c4776d563bb22273bc4eb80596e90659b.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1e6c7f5_d872_4549_b950_1e751b6374ab.slice/crio-conmon-15eb93212f666238582935bcfb7fb66c4776d563bb22273bc4eb80596e90659b.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod330523c8_06c9_42a2_ac3f_9dd4ad62c392.slice/crio-conmon-d190e7520eebe52b0c749ad77fb372857ff3f35218eb3dc8fea952d63280f3d9.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod330523c8_06c9_42a2_ac3f_9dd4ad62c392.slice/crio-d190e7520eebe52b0c749ad77fb372857ff3f35218eb3dc8fea952d63280f3d9.scope\": RecentStats: unable to find data in memory cache]"
Oct 31 12:54:19.382462 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:19.382426    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-85ce08-vl7tk"]
Oct 31 12:54:19.387111 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:19.387083    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-85ce08-vl7tk"]
Oct 31 12:54:20.283720 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:20.283688    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-85ce08-5f656fc7f9-z5g9p"
Oct 31 12:54:20.365262 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:20.365226    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b03c9dc3-2388-458b-8e7f-09cf2811021f-kube-api-access-xmkqf\" (UniqueName: \"kubernetes.io/projected/b03c9dc3-2388-458b-8e7f-09cf2811021f-kube-api-access-xmkqf\") pod \"b03c9dc3-2388-458b-8e7f-09cf2811021f\" (UID: \"b03c9dc3-2388-458b-8e7f-09cf2811021f\") "
Oct 31 12:54:20.365455 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:20.365279    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b03c9dc3-2388-458b-8e7f-09cf2811021f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b03c9dc3-2388-458b-8e7f-09cf2811021f-plugins-config-volume\") pod \"b03c9dc3-2388-458b-8e7f-09cf2811021f\" (UID: \"b03c9dc3-2388-458b-8e7f-09cf2811021f\") "
Oct 31 12:54:20.366285 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:20.366254    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b03c9dc3-2388-458b-8e7f-09cf2811021f-plugins-config-volume" pod "b03c9dc3-2388-458b-8e7f-09cf2811021f" (UID: "b03c9dc3-2388-458b-8e7f-09cf2811021f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:54:20.369113 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:20.369086    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b03c9dc3-2388-458b-8e7f-09cf2811021f-kube-api-access-xmkqf" pod "b03c9dc3-2388-458b-8e7f-09cf2811021f" (UID: "b03c9dc3-2388-458b-8e7f-09cf2811021f"). InnerVolumeSpecName "kube-api-access-xmkqf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:54:20.467248 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:20.467208    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xmkqf\" (UniqueName: \"kubernetes.io/projected/b03c9dc3-2388-458b-8e7f-09cf2811021f-kube-api-access-xmkqf\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:54:20.467248 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:20.467243    5060 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b03c9dc3-2388-458b-8e7f-09cf2811021f-plugins-config-volume\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:54:20.758379 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:20.758342    5060 generic.go:356] "Generic (PLEG): container finished" podID="b03c9dc3-2388-458b-8e7f-09cf2811021f" containerID="b0a736c77932c00d5497c4435970c017fd800e61c497ebcfb4c5d9fc653ab315" exitCode=0
Oct 31 12:54:20.758626 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:20.758413    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-85ce08-5f656fc7f9-z5g9p"
Oct 31 12:54:20.758626 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:20.758438    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-85ce08-5f656fc7f9-z5g9p" event={"ID":"b03c9dc3-2388-458b-8e7f-09cf2811021f","Type":"ContainerDied","Data":"b0a736c77932c00d5497c4435970c017fd800e61c497ebcfb4c5d9fc653ab315"}
Oct 31 12:54:20.758626 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:20.758539    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-85ce08-5f656fc7f9-z5g9p" event={"ID":"b03c9dc3-2388-458b-8e7f-09cf2811021f","Type":"ContainerDied","Data":"4c5d05924f4d06fac07adcf6f71af0b2c6d4ece9becd17018b77cfe8205859c9"}
Oct 31 12:54:20.758626 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:20.758568    5060 scope.go:122] "RemoveContainer" containerID="b0a736c77932c00d5497c4435970c017fd800e61c497ebcfb4c5d9fc653ab315"
Oct 31 12:54:20.813745 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:20.793095    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-85ce08-5f656fc7f9-z5g9p"]
Oct 31 12:54:20.839556 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:20.837577    5060 scope.go:122] "RemoveContainer" containerID="b0a736c77932c00d5497c4435970c017fd800e61c497ebcfb4c5d9fc653ab315"
Oct 31 12:54:20.840331 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:54:20.840232    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0a736c77932c00d5497c4435970c017fd800e61c497ebcfb4c5d9fc653ab315\": container with ID starting with b0a736c77932c00d5497c4435970c017fd800e61c497ebcfb4c5d9fc653ab315 not found: ID does not exist" containerID="b0a736c77932c00d5497c4435970c017fd800e61c497ebcfb4c5d9fc653ab315"
Oct 31 12:54:20.840331 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:20.840280    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0a736c77932c00d5497c4435970c017fd800e61c497ebcfb4c5d9fc653ab315"} err="failed to get container status \"b0a736c77932c00d5497c4435970c017fd800e61c497ebcfb4c5d9fc653ab315\": rpc error: code = NotFound desc = could not find container \"b0a736c77932c00d5497c4435970c017fd800e61c497ebcfb4c5d9fc653ab315\": container with ID starting with b0a736c77932c00d5497c4435970c017fd800e61c497ebcfb4c5d9fc653ab315 not found: ID does not exist"
Oct 31 12:54:20.919514 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:20.915984    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-85ce08-5f656fc7f9-z5g9p"]
Oct 31 12:54:20.983762 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:20.956611    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-b218d3-766c68ffbf-76pwg" podUID="2ec2c204-58e3-4d0f-b06b-5112806add83" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 12:54:21.108768 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:21.108732    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b03c9dc3-2388-458b-8e7f-09cf2811021f" path="/var/lib/kubelet/pods/b03c9dc3-2388-458b-8e7f-09cf2811021f/volumes"
Oct 31 12:54:21.109968 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:21.109925    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5017d07-e40d-45e5-8a08-1d1866bc0d54" path="/var/lib/kubelet/pods/b5017d07-e40d-45e5-8a08-1d1866bc0d54/volumes"
Oct 31 12:54:21.375462 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:21.375370    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-fb2b1d-r8vf7"]
Oct 31 12:54:21.381024 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:21.380993    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-fb2b1d-r8vf7"]
Oct 31 12:54:23.050853 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:23.050818    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85ab7b73-48a3-430a-b489-30985757d106" path="/var/lib/kubelet/pods/85ab7b73-48a3-430a-b489-30985757d106/volumes"
Oct 31 12:54:23.701770 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:23.701738    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2c393f-77f8c6697-l7t2b"
Oct 31 12:54:24.756279 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:24.756244    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fd988c-798cc7df69-jdx5x"
Oct 31 12:54:24.765706 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:24.765681    5060 scope.go:122] "RemoveContainer" containerID="33255c13e0d2189ee73cca104bdebc8a142182d63396105123cd5b7e49177626"
Oct 31 12:54:24.778179 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:24.778157    5060 scope.go:122] "RemoveContainer" containerID="e42fb51c5312380b1f24ee6c93eb46ca1a7a43bb3fcf2bff8253c47a2fd87f7e"
Oct 31 12:54:24.816990 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:24.816966    5060 scope.go:122] "RemoveContainer" containerID="4c8798e6a22a01eb0ee9bdfd352175b6ac22c4444cfd5d89c4e08e492a2f20d6"
Oct 31 12:54:24.828648 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:24.828622    5060 scope.go:122] "RemoveContainer" containerID="a5457bbc2fa14cea13e8f2a52bd54a89bd65735191b1d3ee74a53726e7ca69fe"
Oct 31 12:54:25.380923 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:25.380884    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27a1e456-tpdkw"]
Oct 31 12:54:25.382161 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:25.382133    5060 cpu_manager.go:414] "containerMap: removing container" podUID="90387d5f-2bb8-4922-a1ad-9466e0745aaf" containerName="istio-proxy"
Oct 31 12:54:25.382161 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:25.382161    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90387d5f-2bb8-4922-a1ad-9466e0745aaf" containerName="istio-proxy"
Oct 31 12:54:25.382376 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:25.382284    5060 cpu_manager.go:414] "containerMap: removing container" podUID="b03c9dc3-2388-458b-8e7f-09cf2811021f" containerName="payload-processing"
Oct 31 12:54:25.382376 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:25.382293    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b03c9dc3-2388-458b-8e7f-09cf2811021f" containerName="payload-processing"
Oct 31 12:54:25.382531 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:25.382482    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="90387d5f-2bb8-4922-a1ad-9466e0745aaf" containerName="istio-proxy"
Oct 31 12:54:25.382531 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:25.382529    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="b03c9dc3-2388-458b-8e7f-09cf2811021f" containerName="payload-processing"
Oct 31 12:54:25.389654 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:25.389631    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27a1e456-tpdkw"
Oct 31 12:54:25.397321 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:25.397296    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7wsmb\""
Oct 31 12:54:25.398459 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:25.398437    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 12:54:25.408020 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:25.407989    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27a1e456-tpdkw"]
Oct 31 12:54:25.415556 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:25.415518    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b8de4c2b-4cvrx"]
Oct 31 12:54:25.421703 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:25.421683    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8de4c2b-4cvrx"
Oct 31 12:54:25.435877 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:25.435849    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b8de4c2b-4cvrx"]
Oct 31 12:54:25.459533 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:25.458855    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8bm4\" (UniqueName: \"kubernetes.io/projected/f31bbc5b-c0a2-4380-b1c7-e2571b544260-kube-api-access-m8bm4\") pod \"maas-api-revoke-sub-27a1e456-tpdkw\" (UID: \"f31bbc5b-c0a2-4380-b1c7-e2571b544260\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27a1e456-tpdkw"
Oct 31 12:54:25.459533 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:25.458911    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1721b308-47a8-44a0-a941-7c09b71f9d69-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b8de4c2b-4cvrx\" (UID: \"1721b308-47a8-44a0-a941-7c09b71f9d69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8de4c2b-4cvrx"
Oct 31 12:54:25.459533 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:25.458995    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1721b308-47a8-44a0-a941-7c09b71f9d69-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b8de4c2b-4cvrx\" (UID: \"1721b308-47a8-44a0-a941-7c09b71f9d69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8de4c2b-4cvrx"
Oct 31 12:54:25.459533 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:25.459012    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhlm9\" (UniqueName: \"kubernetes.io/projected/1721b308-47a8-44a0-a941-7c09b71f9d69-kube-api-access-lhlm9\") pod \"maas-api-revoke-sub-b8de4c2b-4cvrx\" (UID: \"1721b308-47a8-44a0-a941-7c09b71f9d69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8de4c2b-4cvrx"
Oct 31 12:54:25.459533 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:25.459057    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f31bbc5b-c0a2-4380-b1c7-e2571b544260-maas-api-xks-ca\") pod \"maas-api-revoke-sub-27a1e456-tpdkw\" (UID: \"f31bbc5b-c0a2-4380-b1c7-e2571b544260\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27a1e456-tpdkw"
Oct 31 12:54:25.459533 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:25.459095    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f31bbc5b-c0a2-4380-b1c7-e2571b544260-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-27a1e456-tpdkw\" (UID: \"f31bbc5b-c0a2-4380-b1c7-e2571b544260\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27a1e456-tpdkw"
Oct 31 12:54:25.560244 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:25.560209    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1721b308-47a8-44a0-a941-7c09b71f9d69-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b8de4c2b-4cvrx\" (UID: \"1721b308-47a8-44a0-a941-7c09b71f9d69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8de4c2b-4cvrx"
Oct 31 12:54:25.560400 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:25.560313    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1721b308-47a8-44a0-a941-7c09b71f9d69-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b8de4c2b-4cvrx\" (UID: \"1721b308-47a8-44a0-a941-7c09b71f9d69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8de4c2b-4cvrx"
Oct 31 12:54:25.560400 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:25.560334    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhlm9\" (UniqueName: \"kubernetes.io/projected/1721b308-47a8-44a0-a941-7c09b71f9d69-kube-api-access-lhlm9\") pod \"maas-api-revoke-sub-b8de4c2b-4cvrx\" (UID: \"1721b308-47a8-44a0-a941-7c09b71f9d69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8de4c2b-4cvrx"
Oct 31 12:54:25.560400 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:25.560378    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f31bbc5b-c0a2-4380-b1c7-e2571b544260-maas-api-xks-ca\") pod \"maas-api-revoke-sub-27a1e456-tpdkw\" (UID: \"f31bbc5b-c0a2-4380-b1c7-e2571b544260\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27a1e456-tpdkw"
Oct 31 12:54:25.560630 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:25.560430    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f31bbc5b-c0a2-4380-b1c7-e2571b544260-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-27a1e456-tpdkw\" (UID: \"f31bbc5b-c0a2-4380-b1c7-e2571b544260\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27a1e456-tpdkw"
Oct 31 12:54:25.560731 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:25.560622    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8bm4\" (UniqueName: \"kubernetes.io/projected/f31bbc5b-c0a2-4380-b1c7-e2571b544260-kube-api-access-m8bm4\") pod \"maas-api-revoke-sub-27a1e456-tpdkw\" (UID: \"f31bbc5b-c0a2-4380-b1c7-e2571b544260\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27a1e456-tpdkw"
Oct 31 12:54:25.561164 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:25.561141    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1721b308-47a8-44a0-a941-7c09b71f9d69-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b8de4c2b-4cvrx\" (UID: \"1721b308-47a8-44a0-a941-7c09b71f9d69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8de4c2b-4cvrx"
Oct 31 12:54:25.561400 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:25.561375    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f31bbc5b-c0a2-4380-b1c7-e2571b544260-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-27a1e456-tpdkw\" (UID: \"f31bbc5b-c0a2-4380-b1c7-e2571b544260\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27a1e456-tpdkw"
Oct 31 12:54:25.563213 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:25.563168    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f31bbc5b-c0a2-4380-b1c7-e2571b544260-maas-api-xks-ca\") pod \"maas-api-revoke-sub-27a1e456-tpdkw\" (UID: \"f31bbc5b-c0a2-4380-b1c7-e2571b544260\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27a1e456-tpdkw"
Oct 31 12:54:25.563618 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:25.563578    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1721b308-47a8-44a0-a941-7c09b71f9d69-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b8de4c2b-4cvrx\" (UID: \"1721b308-47a8-44a0-a941-7c09b71f9d69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8de4c2b-4cvrx"
Oct 31 12:54:25.574899 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:25.574871    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhlm9\" (UniqueName: \"kubernetes.io/projected/1721b308-47a8-44a0-a941-7c09b71f9d69-kube-api-access-lhlm9\") pod \"maas-api-revoke-sub-b8de4c2b-4cvrx\" (UID: \"1721b308-47a8-44a0-a941-7c09b71f9d69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8de4c2b-4cvrx"
Oct 31 12:54:25.576111 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:25.576084    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8bm4\" (UniqueName: \"kubernetes.io/projected/f31bbc5b-c0a2-4380-b1c7-e2571b544260-kube-api-access-m8bm4\") pod \"maas-api-revoke-sub-27a1e456-tpdkw\" (UID: \"f31bbc5b-c0a2-4380-b1c7-e2571b544260\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27a1e456-tpdkw"
Oct 31 12:54:25.725080 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:25.718067    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27a1e456-tpdkw"
Oct 31 12:54:25.764519 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:25.763646    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8de4c2b-4cvrx"
Oct 31 12:54:26.092403 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:26.092106    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27a1e456-tpdkw"]
Oct 31 12:54:26.092403 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:54:26.092176    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf31bbc5b_c0a2_4380_b1c7_e2571b544260.slice/crio-ad5d05e84f804a6452e75b561beef23d268978327842cd96c08704886b6eb87b WatchSource:0}: Error finding container ad5d05e84f804a6452e75b561beef23d268978327842cd96c08704886b6eb87b: Status 404 returned error can't find the container with id ad5d05e84f804a6452e75b561beef23d268978327842cd96c08704886b6eb87b
Oct 31 12:54:26.148264 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:54:26.148231    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1721b308_47a8_44a0_a941_7c09b71f9d69.slice/crio-12e1a604cc9b31c5f8d918ea64d494e188f62c2ac9741b3aaac01b463d77587c WatchSource:0}: Error finding container 12e1a604cc9b31c5f8d918ea64d494e188f62c2ac9741b3aaac01b463d77587c: Status 404 returned error can't find the container with id 12e1a604cc9b31c5f8d918ea64d494e188f62c2ac9741b3aaac01b463d77587c
Oct 31 12:54:26.150902 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:26.150873    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b8de4c2b-4cvrx"]
Oct 31 12:54:27.019595 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:27.019484    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27a1e456-tpdkw" event={"ID":"f31bbc5b-c0a2-4380-b1c7-e2571b544260","Type":"ContainerStarted","Data":"f1b54df8213f1c477052f21e28ec5fc98f2842f44810408a13c4b1bc3c275e07"}
Oct 31 12:54:27.019595 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:27.019564    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27a1e456-tpdkw" event={"ID":"f31bbc5b-c0a2-4380-b1c7-e2571b544260","Type":"ContainerStarted","Data":"ad5d05e84f804a6452e75b561beef23d268978327842cd96c08704886b6eb87b"}
Oct 31 12:54:27.022088 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:27.022053    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8de4c2b-4cvrx" event={"ID":"1721b308-47a8-44a0-a941-7c09b71f9d69","Type":"ContainerStarted","Data":"955e669d2572d6c92e972dc8dfc208d142fa4331bbbacdf311993f1a752e35c9"}
Oct 31 12:54:27.022205 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:27.022115    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8de4c2b-4cvrx" event={"ID":"1721b308-47a8-44a0-a941-7c09b71f9d69","Type":"ContainerStarted","Data":"12e1a604cc9b31c5f8d918ea64d494e188f62c2ac9741b3aaac01b463d77587c"}
Oct 31 12:54:27.040025 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:27.039968    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27a1e456-tpdkw" podStartSLOduration=2.039951823 podStartE2EDuration="2.039951823s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:54:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:54:27.037013248 +0000 UTC m=+1630.140080962" watchObservedRunningTime="2026-10-31 12:54:27.039951823 +0000 UTC m=+1630.143019535"
Oct 31 12:54:27.063530 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:27.063432    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8de4c2b-4cvrx" podStartSLOduration=2.063413153 podStartE2EDuration="2.063413153s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:54:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:54:27.058599281 +0000 UTC m=+1630.161666987" watchObservedRunningTime="2026-10-31 12:54:27.063413153 +0000 UTC m=+1630.166480867"
Oct 31 12:54:28.051927 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:28.051885    5060 generic.go:356] "Generic (PLEG): container finished" podID="f31bbc5b-c0a2-4380-b1c7-e2571b544260" containerID="f1b54df8213f1c477052f21e28ec5fc98f2842f44810408a13c4b1bc3c275e07" exitCode=0
Oct 31 12:54:28.052392 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:28.052028    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27a1e456-tpdkw" event={"ID":"f31bbc5b-c0a2-4380-b1c7-e2571b544260","Type":"ContainerDied","Data":"f1b54df8213f1c477052f21e28ec5fc98f2842f44810408a13c4b1bc3c275e07"}
Oct 31 12:54:28.054405 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:28.054357    5060 generic.go:356] "Generic (PLEG): container finished" podID="1721b308-47a8-44a0-a941-7c09b71f9d69" containerID="955e669d2572d6c92e972dc8dfc208d142fa4331bbbacdf311993f1a752e35c9" exitCode=0
Oct 31 12:54:28.054624 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:28.054530    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8de4c2b-4cvrx" event={"ID":"1721b308-47a8-44a0-a941-7c09b71f9d69","Type":"ContainerDied","Data":"955e669d2572d6c92e972dc8dfc208d142fa4331bbbacdf311993f1a752e35c9"}
Oct 31 12:54:29.060576 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:29.060537    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-58cq4"]
Oct 31 12:54:29.061720 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:29.061666    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-58cq4" podUID="c86d1b8b-6bc6-45ff-9412-179f069cf080" containerName="istio-proxy" containerID="cri-o://436dd479231de8934ac874f816c15f4b0d01f071f8d5b37931d737444f7b4523" gracePeriod=30
Oct 31 12:54:29.072089 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:29.072057    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-f8gp2"]
Oct 31 12:54:29.072850 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:29.072818    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-f8gp2" podUID="df05433b-b6c1-4bd0-88a1-11e769461d69" containerName="istio-proxy" containerID="cri-o://be887fbca1475bb75a00814d220dfaf71c3ea6f0884c58c1ac59ae8c2b8730cd" gracePeriod=30
Oct 31 12:54:29.635768 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:29.635742    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27a1e456-tpdkw"
Oct 31 12:54:29.741824 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:29.741792    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f31bbc5b-c0a2-4380-b1c7-e2571b544260-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f31bbc5b-c0a2-4380-b1c7-e2571b544260-maas-api-xks-ca\") pod \"f31bbc5b-c0a2-4380-b1c7-e2571b544260\" (UID: \"f31bbc5b-c0a2-4380-b1c7-e2571b544260\") "
Oct 31 12:54:29.742002 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:29.741868    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f31bbc5b-c0a2-4380-b1c7-e2571b544260-kube-api-access-m8bm4\" (UniqueName: \"kubernetes.io/projected/f31bbc5b-c0a2-4380-b1c7-e2571b544260-kube-api-access-m8bm4\") pod \"f31bbc5b-c0a2-4380-b1c7-e2571b544260\" (UID: \"f31bbc5b-c0a2-4380-b1c7-e2571b544260\") "
Oct 31 12:54:29.742082 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:29.742056    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f31bbc5b-c0a2-4380-b1c7-e2571b544260-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f31bbc5b-c0a2-4380-b1c7-e2571b544260-maas-api-openshift-ca\") pod \"f31bbc5b-c0a2-4380-b1c7-e2571b544260\" (UID: \"f31bbc5b-c0a2-4380-b1c7-e2571b544260\") "
Oct 31 12:54:29.742632 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:29.742598    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f31bbc5b-c0a2-4380-b1c7-e2571b544260-maas-api-openshift-ca" pod "f31bbc5b-c0a2-4380-b1c7-e2571b544260" (UID: "f31bbc5b-c0a2-4380-b1c7-e2571b544260"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:54:29.742792 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:29.742757    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8de4c2b-4cvrx"
Oct 31 12:54:29.742983 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:29.742960    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f31bbc5b-c0a2-4380-b1c7-e2571b544260-maas-api-openshift-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:54:29.744630 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:29.744600    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f31bbc5b-c0a2-4380-b1c7-e2571b544260-maas-api-xks-ca" pod "f31bbc5b-c0a2-4380-b1c7-e2571b544260" (UID: "f31bbc5b-c0a2-4380-b1c7-e2571b544260"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:54:29.744732 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:29.744641    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f31bbc5b-c0a2-4380-b1c7-e2571b544260-kube-api-access-m8bm4" pod "f31bbc5b-c0a2-4380-b1c7-e2571b544260" (UID: "f31bbc5b-c0a2-4380-b1c7-e2571b544260"). InnerVolumeSpecName "kube-api-access-m8bm4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:54:29.844654 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:29.844564    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1721b308-47a8-44a0-a941-7c09b71f9d69-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1721b308-47a8-44a0-a941-7c09b71f9d69-maas-api-xks-ca\") pod \"1721b308-47a8-44a0-a941-7c09b71f9d69\" (UID: \"1721b308-47a8-44a0-a941-7c09b71f9d69\") "
Oct 31 12:54:29.844856 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:29.844835    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1721b308-47a8-44a0-a941-7c09b71f9d69-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1721b308-47a8-44a0-a941-7c09b71f9d69-maas-api-openshift-ca\") pod \"1721b308-47a8-44a0-a941-7c09b71f9d69\" (UID: \"1721b308-47a8-44a0-a941-7c09b71f9d69\") "
Oct 31 12:54:29.844931 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:29.844885    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1721b308-47a8-44a0-a941-7c09b71f9d69-kube-api-access-lhlm9\" (UniqueName: \"kubernetes.io/projected/1721b308-47a8-44a0-a941-7c09b71f9d69-kube-api-access-lhlm9\") pod \"1721b308-47a8-44a0-a941-7c09b71f9d69\" (UID: \"1721b308-47a8-44a0-a941-7c09b71f9d69\") "
Oct 31 12:54:29.845311 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:29.845278    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1721b308-47a8-44a0-a941-7c09b71f9d69-maas-api-openshift-ca" pod "1721b308-47a8-44a0-a941-7c09b71f9d69" (UID: "1721b308-47a8-44a0-a941-7c09b71f9d69"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:54:29.845563 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:29.845539    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8bm4\" (UniqueName: \"kubernetes.io/projected/f31bbc5b-c0a2-4380-b1c7-e2571b544260-kube-api-access-m8bm4\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:54:29.845654 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:29.845636    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f31bbc5b-c0a2-4380-b1c7-e2571b544260-maas-api-xks-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:54:29.845713 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:29.845662    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1721b308-47a8-44a0-a941-7c09b71f9d69-maas-api-openshift-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:54:29.846958 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:29.846925    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1721b308-47a8-44a0-a941-7c09b71f9d69-maas-api-xks-ca" pod "1721b308-47a8-44a0-a941-7c09b71f9d69" (UID: "1721b308-47a8-44a0-a941-7c09b71f9d69"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:54:29.847414 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:29.847389    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1721b308-47a8-44a0-a941-7c09b71f9d69-kube-api-access-lhlm9" pod "1721b308-47a8-44a0-a941-7c09b71f9d69" (UID: "1721b308-47a8-44a0-a941-7c09b71f9d69"). InnerVolumeSpecName "kube-api-access-lhlm9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:54:29.923620 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:54:29.923581    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7phb" containerName="extractor"
Oct 31 12:54:29.948091 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:29.948055    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhlm9\" (UniqueName: \"kubernetes.io/projected/1721b308-47a8-44a0-a941-7c09b71f9d69-kube-api-access-lhlm9\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:54:29.948091 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:29.948094    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1721b308-47a8-44a0-a941-7c09b71f9d69-maas-api-xks-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:54:30.093070 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:30.093032    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27a1e456-tpdkw"
Oct 31 12:54:30.093558 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:30.093066    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27a1e456-tpdkw" event={"ID":"f31bbc5b-c0a2-4380-b1c7-e2571b544260","Type":"ContainerDied","Data":"ad5d05e84f804a6452e75b561beef23d268978327842cd96c08704886b6eb87b"}
Oct 31 12:54:30.093558 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:30.093128    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ad5d05e84f804a6452e75b561beef23d268978327842cd96c08704886b6eb87b"
Oct 31 12:54:30.095235 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:30.095178    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8de4c2b-4cvrx"
Oct 31 12:54:30.095427 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:30.095267    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8de4c2b-4cvrx" event={"ID":"1721b308-47a8-44a0-a941-7c09b71f9d69","Type":"ContainerDied","Data":"12e1a604cc9b31c5f8d918ea64d494e188f62c2ac9741b3aaac01b463d77587c"}
Oct 31 12:54:30.095637 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:30.095454    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="12e1a604cc9b31c5f8d918ea64d494e188f62c2ac9741b3aaac01b463d77587c"
Oct 31 12:54:30.895575 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:30.892863    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-b218d3-766c68ffbf-76pwg" podUID="2ec2c204-58e3-4d0f-b06b-5112806add83" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 12:54:32.572997 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:32.572961    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-b218d3-766c68ffbf-76pwg"
Oct 31 12:54:32.628042 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:32.627999    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ec2c204-58e3-4d0f-b06b-5112806add83-kube-api-access-dvvj5\" (UniqueName: \"kubernetes.io/projected/2ec2c204-58e3-4d0f-b06b-5112806add83-kube-api-access-dvvj5\") pod \"2ec2c204-58e3-4d0f-b06b-5112806add83\" (UID: \"2ec2c204-58e3-4d0f-b06b-5112806add83\") "
Oct 31 12:54:32.628423 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:32.628397    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ec2c204-58e3-4d0f-b06b-5112806add83-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ec2c204-58e3-4d0f-b06b-5112806add83-plugins-config-volume\") pod \"2ec2c204-58e3-4d0f-b06b-5112806add83\" (UID: \"2ec2c204-58e3-4d0f-b06b-5112806add83\") "
Oct 31 12:54:32.630167 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:32.629584    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ec2c204-58e3-4d0f-b06b-5112806add83-plugins-config-volume" pod "2ec2c204-58e3-4d0f-b06b-5112806add83" (UID: "2ec2c204-58e3-4d0f-b06b-5112806add83"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:54:32.640568 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:32.640528    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ec2c204-58e3-4d0f-b06b-5112806add83-kube-api-access-dvvj5" pod "2ec2c204-58e3-4d0f-b06b-5112806add83" (UID: "2ec2c204-58e3-4d0f-b06b-5112806add83"). InnerVolumeSpecName "kube-api-access-dvvj5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:54:32.731187 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:32.731147    5060 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ec2c204-58e3-4d0f-b06b-5112806add83-plugins-config-volume\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:54:32.731187 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:32.731189    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvvj5\" (UniqueName: \"kubernetes.io/projected/2ec2c204-58e3-4d0f-b06b-5112806add83-kube-api-access-dvvj5\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:54:32.884354 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:32.884267    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-hkvl5"]
Oct 31 12:54:32.884681 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:32.884603    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-hkvl5" podUID="4e300ab1-6fe3-4548-90b3-0cbb8099e114" containerName="istio-proxy" containerID="cri-o://ff3576ccdf821939a0cd5ae44ea67bdc37f3e3711c25c2282a554d05810389d6" gracePeriod=30
Oct 31 12:54:32.888520 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:32.888475    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-72mg2"]
Oct 31 12:54:32.888733 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:32.888708    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-72mg2" podUID="346f4c45-c06a-4ae5-9fa0-c426467700f9" containerName="istio-proxy" containerID="cri-o://ea8748ac2ab76bad1ceca1fd5047a546529d7eb073f2611aa57bbbb73e6ea836" gracePeriod=30
Oct 31 12:54:33.241332 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:33.240170    5060 generic.go:356] "Generic (PLEG): container finished" podID="2ec2c204-58e3-4d0f-b06b-5112806add83" containerID="3990f663560e6baf90c55d56bae73742e4746e910ea63c074d0ad6670e4db361" exitCode=0
Oct 31 12:54:33.241332 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:33.240309    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-b218d3-766c68ffbf-76pwg" event={"ID":"2ec2c204-58e3-4d0f-b06b-5112806add83","Type":"ContainerDied","Data":"3990f663560e6baf90c55d56bae73742e4746e910ea63c074d0ad6670e4db361"}
Oct 31 12:54:33.241332 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:33.240364    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-b218d3-766c68ffbf-76pwg" event={"ID":"2ec2c204-58e3-4d0f-b06b-5112806add83","Type":"ContainerDied","Data":"1f2dae6f86b7390a2d71e8061033f3cb7898368a436f615cc175f317c437463d"}
Oct 31 12:54:33.241332 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:33.240398    5060 scope.go:122] "RemoveContainer" containerID="3990f663560e6baf90c55d56bae73742e4746e910ea63c074d0ad6670e4db361"
Oct 31 12:54:33.241332 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:33.240600    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-b218d3-766c68ffbf-76pwg"
Oct 31 12:54:33.260975 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:33.260946    5060 scope.go:122] "RemoveContainer" containerID="3990f663560e6baf90c55d56bae73742e4746e910ea63c074d0ad6670e4db361"
Oct 31 12:54:33.261379 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:54:33.261346    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3990f663560e6baf90c55d56bae73742e4746e910ea63c074d0ad6670e4db361\": container with ID starting with 3990f663560e6baf90c55d56bae73742e4746e910ea63c074d0ad6670e4db361 not found: ID does not exist" containerID="3990f663560e6baf90c55d56bae73742e4746e910ea63c074d0ad6670e4db361"
Oct 31 12:54:33.261507 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:33.261389    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3990f663560e6baf90c55d56bae73742e4746e910ea63c074d0ad6670e4db361"} err="failed to get container status \"3990f663560e6baf90c55d56bae73742e4746e910ea63c074d0ad6670e4db361\": rpc error: code = NotFound desc = could not find container \"3990f663560e6baf90c55d56bae73742e4746e910ea63c074d0ad6670e4db361\": container with ID starting with 3990f663560e6baf90c55d56bae73742e4746e910ea63c074d0ad6670e4db361 not found: ID does not exist"
Oct 31 12:54:33.305111 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:33.305074    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-b218d3-766c68ffbf-76pwg"]
Oct 31 12:54:33.317450 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:33.317421    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-b218d3-766c68ffbf-76pwg"]
Oct 31 12:54:33.546922 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:33.546886    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d4dd68-xfh6r"]
Oct 31 12:54:33.547851 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:33.547826    5060 cpu_manager.go:414] "containerMap: removing container" podUID="f31bbc5b-c0a2-4380-b1c7-e2571b544260" containerName="revoke-keys"
Oct 31 12:54:33.547851 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:33.547850    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f31bbc5b-c0a2-4380-b1c7-e2571b544260" containerName="revoke-keys"
Oct 31 12:54:33.548001 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:33.547871    5060 cpu_manager.go:414] "containerMap: removing container" podUID="1721b308-47a8-44a0-a941-7c09b71f9d69" containerName="revoke-keys"
Oct 31 12:54:33.548001 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:33.547879    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1721b308-47a8-44a0-a941-7c09b71f9d69" containerName="revoke-keys"
Oct 31 12:54:33.548001 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:33.547894    5060 cpu_manager.go:414] "containerMap: removing container" podUID="2ec2c204-58e3-4d0f-b06b-5112806add83" containerName="payload-processing"
Oct 31 12:54:33.548001 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:33.547899    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ec2c204-58e3-4d0f-b06b-5112806add83" containerName="payload-processing"
Oct 31 12:54:33.548124 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:33.548021    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="f31bbc5b-c0a2-4380-b1c7-e2571b544260" containerName="revoke-keys"
Oct 31 12:54:33.548124 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:33.548036    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="1721b308-47a8-44a0-a941-7c09b71f9d69" containerName="revoke-keys"
Oct 31 12:54:33.548124 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:33.548045    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ec2c204-58e3-4d0f-b06b-5112806add83" containerName="payload-processing"
Oct 31 12:54:33.553330 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:33.553309    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d4dd68-xfh6r"
Oct 31 12:54:33.570424 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:33.570401    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d4dd68-xfh6r"]
Oct 31 12:54:33.581282 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:33.581254    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 12:54:33.581708 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:33.581408    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7wsmb\""
Oct 31 12:54:33.725265 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:33.725223    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17e10309-cb84-4a27-b994-55863b40f305-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-d4dd68-xfh6r\" (UID: \"17e10309-cb84-4a27-b994-55863b40f305\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d4dd68-xfh6r"
Oct 31 12:54:33.725466 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:33.725405    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4rjv\" (UniqueName: \"kubernetes.io/projected/17e10309-cb84-4a27-b994-55863b40f305-kube-api-access-j4rjv\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-d4dd68-xfh6r\" (UID: \"17e10309-cb84-4a27-b994-55863b40f305\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d4dd68-xfh6r"
Oct 31 12:54:33.725466 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:33.725440    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17e10309-cb84-4a27-b994-55863b40f305-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-d4dd68-xfh6r\" (UID: \"17e10309-cb84-4a27-b994-55863b40f305\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d4dd68-xfh6r"
Oct 31 12:54:33.827215 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:33.827129    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17e10309-cb84-4a27-b994-55863b40f305-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-d4dd68-xfh6r\" (UID: \"17e10309-cb84-4a27-b994-55863b40f305\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d4dd68-xfh6r"
Oct 31 12:54:33.827414 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:33.827253    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4rjv\" (UniqueName: \"kubernetes.io/projected/17e10309-cb84-4a27-b994-55863b40f305-kube-api-access-j4rjv\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-d4dd68-xfh6r\" (UID: \"17e10309-cb84-4a27-b994-55863b40f305\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d4dd68-xfh6r"
Oct 31 12:54:33.827414 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:33.827274    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17e10309-cb84-4a27-b994-55863b40f305-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-d4dd68-xfh6r\" (UID: \"17e10309-cb84-4a27-b994-55863b40f305\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d4dd68-xfh6r"
Oct 31 12:54:33.828153 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:33.828123    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17e10309-cb84-4a27-b994-55863b40f305-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-d4dd68-xfh6r\" (UID: \"17e10309-cb84-4a27-b994-55863b40f305\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d4dd68-xfh6r"
Oct 31 12:54:33.830124 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:33.830093    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17e10309-cb84-4a27-b994-55863b40f305-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-d4dd68-xfh6r\" (UID: \"17e10309-cb84-4a27-b994-55863b40f305\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d4dd68-xfh6r"
Oct 31 12:54:33.845662 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:33.845630    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4rjv\" (UniqueName: \"kubernetes.io/projected/17e10309-cb84-4a27-b994-55863b40f305-kube-api-access-j4rjv\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-d4dd68-xfh6r\" (UID: \"17e10309-cb84-4a27-b994-55863b40f305\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d4dd68-xfh6r"
Oct 31 12:54:33.869726 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:33.869696    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d4dd68-xfh6r"
Oct 31 12:54:34.180054 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:34.180021    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d4dd68-xfh6r"]
Oct 31 12:54:34.198457 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:54:34.198426    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17e10309_cb84_4a27_b994_55863b40f305.slice/crio-c9a6250278c202b86f58cc9f297118b8b5b64cb627efe5d7550b8789bf7a54b8 WatchSource:0}: Error finding container c9a6250278c202b86f58cc9f297118b8b5b64cb627efe5d7550b8789bf7a54b8: Status 404 returned error can't find the container with id c9a6250278c202b86f58cc9f297118b8b5b64cb627efe5d7550b8789bf7a54b8
Oct 31 12:54:34.252723 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:34.252691    5060 generic.go:356] "Generic (PLEG): container finished" podID="c86d1b8b-6bc6-45ff-9412-179f069cf080" containerID="436dd479231de8934ac874f816c15f4b0d01f071f8d5b37931d737444f7b4523" exitCode=0
Oct 31 12:54:34.252859 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:34.252827    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-58cq4" event={"ID":"c86d1b8b-6bc6-45ff-9412-179f069cf080","Type":"ContainerDied","Data":"436dd479231de8934ac874f816c15f4b0d01f071f8d5b37931d737444f7b4523"}
Oct 31 12:54:34.252913 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:34.252873    5060 scope.go:122] "RemoveContainer" containerID="c5d4d07861968414635e12412ade308cb520be5914e7ae0cad43698ac25e5dda"
Oct 31 12:54:34.257426 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:34.257403    5060 generic.go:356] "Generic (PLEG): container finished" podID="df05433b-b6c1-4bd0-88a1-11e769461d69" containerID="be887fbca1475bb75a00814d220dfaf71c3ea6f0884c58c1ac59ae8c2b8730cd" exitCode=0
Oct 31 12:54:34.257726 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:34.257526    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-f8gp2" event={"ID":"df05433b-b6c1-4bd0-88a1-11e769461d69","Type":"ContainerDied","Data":"be887fbca1475bb75a00814d220dfaf71c3ea6f0884c58c1ac59ae8c2b8730cd"}
Oct 31 12:54:34.259777 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:34.259750    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d4dd68-xfh6r" event={"ID":"17e10309-cb84-4a27-b994-55863b40f305","Type":"ContainerStarted","Data":"c9a6250278c202b86f58cc9f297118b8b5b64cb627efe5d7550b8789bf7a54b8"}
Oct 31 12:54:34.432542 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:34.430330    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-58cq4"
Oct 31 12:54:34.574532 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:34.559999    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c86d1b8b-6bc6-45ff-9412-179f069cf080-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c86d1b8b-6bc6-45ff-9412-179f069cf080-istio-data\") pod \"c86d1b8b-6bc6-45ff-9412-179f069cf080\" (UID: \"c86d1b8b-6bc6-45ff-9412-179f069cf080\") "
Oct 31 12:54:34.574532 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:34.560184    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c86d1b8b-6bc6-45ff-9412-179f069cf080-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c86d1b8b-6bc6-45ff-9412-179f069cf080-workload-certs\") pod \"c86d1b8b-6bc6-45ff-9412-179f069cf080\" (UID: \"c86d1b8b-6bc6-45ff-9412-179f069cf080\") "
Oct 31 12:54:34.574532 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:34.560221    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c86d1b8b-6bc6-45ff-9412-179f069cf080-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c86d1b8b-6bc6-45ff-9412-179f069cf080-workload-socket\") pod \"c86d1b8b-6bc6-45ff-9412-179f069cf080\" (UID: \"c86d1b8b-6bc6-45ff-9412-179f069cf080\") "
Oct 31 12:54:34.574532 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:34.560281    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c86d1b8b-6bc6-45ff-9412-179f069cf080-kube-api-access-4dnth\" (UniqueName: \"kubernetes.io/projected/c86d1b8b-6bc6-45ff-9412-179f069cf080-kube-api-access-4dnth\") pod \"c86d1b8b-6bc6-45ff-9412-179f069cf080\" (UID: \"c86d1b8b-6bc6-45ff-9412-179f069cf080\") "
Oct 31 12:54:34.574532 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:34.560361    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c86d1b8b-6bc6-45ff-9412-179f069cf080-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c86d1b8b-6bc6-45ff-9412-179f069cf080-credential-socket\") pod \"c86d1b8b-6bc6-45ff-9412-179f069cf080\" (UID: \"c86d1b8b-6bc6-45ff-9412-179f069cf080\") "
Oct 31 12:54:34.574532 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:34.560420    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c86d1b8b-6bc6-45ff-9412-179f069cf080-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c86d1b8b-6bc6-45ff-9412-179f069cf080-istiod-ca-cert\") pod \"c86d1b8b-6bc6-45ff-9412-179f069cf080\" (UID: \"c86d1b8b-6bc6-45ff-9412-179f069cf080\") "
Oct 31 12:54:34.574532 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:34.560457    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c86d1b8b-6bc6-45ff-9412-179f069cf080-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c86d1b8b-6bc6-45ff-9412-179f069cf080-istio-envoy\") pod \"c86d1b8b-6bc6-45ff-9412-179f069cf080\" (UID: \"c86d1b8b-6bc6-45ff-9412-179f069cf080\") "
Oct 31 12:54:34.574532 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:34.560535    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c86d1b8b-6bc6-45ff-9412-179f069cf080-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c86d1b8b-6bc6-45ff-9412-179f069cf080-istio-podinfo\") pod \"c86d1b8b-6bc6-45ff-9412-179f069cf080\" (UID: \"c86d1b8b-6bc6-45ff-9412-179f069cf080\") "
Oct 31 12:54:34.574532 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:34.560572    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c86d1b8b-6bc6-45ff-9412-179f069cf080-istio-token\" (UniqueName: \"kubernetes.io/projected/c86d1b8b-6bc6-45ff-9412-179f069cf080-istio-token\") pod \"c86d1b8b-6bc6-45ff-9412-179f069cf080\" (UID: \"c86d1b8b-6bc6-45ff-9412-179f069cf080\") "
Oct 31 12:54:34.574532 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:34.573220    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c86d1b8b-6bc6-45ff-9412-179f069cf080-istiod-ca-cert" pod "c86d1b8b-6bc6-45ff-9412-179f069cf080" (UID: "c86d1b8b-6bc6-45ff-9412-179f069cf080"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:54:34.574532 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:34.573503    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c86d1b8b-6bc6-45ff-9412-179f069cf080-credential-socket" pod "c86d1b8b-6bc6-45ff-9412-179f069cf080" (UID: "c86d1b8b-6bc6-45ff-9412-179f069cf080"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:54:34.574532 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:34.573544    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c86d1b8b-6bc6-45ff-9412-179f069cf080-workload-certs" pod "c86d1b8b-6bc6-45ff-9412-179f069cf080" (UID: "c86d1b8b-6bc6-45ff-9412-179f069cf080"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:54:34.589584 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:34.576683    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c86d1b8b-6bc6-45ff-9412-179f069cf080-istio-data" pod "c86d1b8b-6bc6-45ff-9412-179f069cf080" (UID: "c86d1b8b-6bc6-45ff-9412-179f069cf080"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:54:34.589584 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:34.578592    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c86d1b8b-6bc6-45ff-9412-179f069cf080-workload-socket" pod "c86d1b8b-6bc6-45ff-9412-179f069cf080" (UID: "c86d1b8b-6bc6-45ff-9412-179f069cf080"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:54:34.589584 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:34.586592    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-f8gp2"
Oct 31 12:54:34.589584 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:34.587919    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c86d1b8b-6bc6-45ff-9412-179f069cf080-istio-envoy" pod "c86d1b8b-6bc6-45ff-9412-179f069cf080" (UID: "c86d1b8b-6bc6-45ff-9412-179f069cf080"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:54:34.589584 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:34.588116    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c86d1b8b-6bc6-45ff-9412-179f069cf080-istio-token" pod "c86d1b8b-6bc6-45ff-9412-179f069cf080" (UID: "c86d1b8b-6bc6-45ff-9412-179f069cf080"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:54:34.589584 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:34.588261    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c86d1b8b-6bc6-45ff-9412-179f069cf080-kube-api-access-4dnth" pod "c86d1b8b-6bc6-45ff-9412-179f069cf080" (UID: "c86d1b8b-6bc6-45ff-9412-179f069cf080"). InnerVolumeSpecName "kube-api-access-4dnth". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:54:34.656138 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:34.656104    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c86d1b8b-6bc6-45ff-9412-179f069cf080-istio-podinfo" pod "c86d1b8b-6bc6-45ff-9412-179f069cf080" (UID: "c86d1b8b-6bc6-45ff-9412-179f069cf080"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 12:54:34.674915 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:34.674887    5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c86d1b8b-6bc6-45ff-9412-179f069cf080-istio-podinfo\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:54:34.675050 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:34.674917    5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c86d1b8b-6bc6-45ff-9412-179f069cf080-istio-token\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:54:34.675050 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:34.674932    5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c86d1b8b-6bc6-45ff-9412-179f069cf080-istio-data\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:54:34.675050 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:34.674946    5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c86d1b8b-6bc6-45ff-9412-179f069cf080-workload-certs\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:54:34.675050 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:34.674959    5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c86d1b8b-6bc6-45ff-9412-179f069cf080-workload-socket\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:54:34.675050 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:34.674976    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4dnth\" (UniqueName: \"kubernetes.io/projected/c86d1b8b-6bc6-45ff-9412-179f069cf080-kube-api-access-4dnth\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:54:34.675050 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:34.674990    5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c86d1b8b-6bc6-45ff-9412-179f069cf080-credential-socket\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:54:34.675050 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:34.675005    5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c86d1b8b-6bc6-45ff-9412-179f069cf080-istiod-ca-cert\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:54:34.675050 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:34.675021    5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c86d1b8b-6bc6-45ff-9412-179f069cf080-istio-envoy\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:54:34.776803 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:34.776765    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df05433b-b6c1-4bd0-88a1-11e769461d69-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df05433b-b6c1-4bd0-88a1-11e769461d69-istio-envoy\") pod \"df05433b-b6c1-4bd0-88a1-11e769461d69\" (UID: \"df05433b-b6c1-4bd0-88a1-11e769461d69\") "
Oct 31 12:54:34.776975 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:34.776925    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df05433b-b6c1-4bd0-88a1-11e769461d69-kube-api-access-pthhn\" (UniqueName: \"kubernetes.io/projected/df05433b-b6c1-4bd0-88a1-11e769461d69-kube-api-access-pthhn\") pod \"df05433b-b6c1-4bd0-88a1-11e769461d69\" (UID: \"df05433b-b6c1-4bd0-88a1-11e769461d69\") "
Oct 31 12:54:34.777116 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:34.777095    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df05433b-b6c1-4bd0-88a1-11e769461d69-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df05433b-b6c1-4bd0-88a1-11e769461d69-workload-socket\") pod \"df05433b-b6c1-4bd0-88a1-11e769461d69\" (UID: \"df05433b-b6c1-4bd0-88a1-11e769461d69\") "
Oct 31 12:54:34.777230 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:34.777163    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df05433b-b6c1-4bd0-88a1-11e769461d69-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df05433b-b6c1-4bd0-88a1-11e769461d69-istio-data\") pod \"df05433b-b6c1-4bd0-88a1-11e769461d69\" (UID: \"df05433b-b6c1-4bd0-88a1-11e769461d69\") "
Oct 31 12:54:34.777300 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:34.777249    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df05433b-b6c1-4bd0-88a1-11e769461d69-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df05433b-b6c1-4bd0-88a1-11e769461d69-credential-socket\") pod \"df05433b-b6c1-4bd0-88a1-11e769461d69\" (UID: \"df05433b-b6c1-4bd0-88a1-11e769461d69\") "
Oct 31 12:54:34.777300 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:34.777292    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df05433b-b6c1-4bd0-88a1-11e769461d69-istio-token\" (UniqueName: \"kubernetes.io/projected/df05433b-b6c1-4bd0-88a1-11e769461d69-istio-token\") pod \"df05433b-b6c1-4bd0-88a1-11e769461d69\" (UID: \"df05433b-b6c1-4bd0-88a1-11e769461d69\") "
Oct 31 12:54:34.777422 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:34.777350    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df05433b-b6c1-4bd0-88a1-11e769461d69-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df05433b-b6c1-4bd0-88a1-11e769461d69-workload-certs\") pod \"df05433b-b6c1-4bd0-88a1-11e769461d69\" (UID: \"df05433b-b6c1-4bd0-88a1-11e769461d69\") "
Oct 31 12:54:34.777422 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:34.777388    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/df05433b-b6c1-4bd0-88a1-11e769461d69-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df05433b-b6c1-4bd0-88a1-11e769461d69-istio-podinfo\") pod \"df05433b-b6c1-4bd0-88a1-11e769461d69\" (UID: \"df05433b-b6c1-4bd0-88a1-11e769461d69\") "
Oct 31 12:54:34.777591 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:34.777467    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df05433b-b6c1-4bd0-88a1-11e769461d69-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df05433b-b6c1-4bd0-88a1-11e769461d69-istiod-ca-cert\") pod \"df05433b-b6c1-4bd0-88a1-11e769461d69\" (UID: \"df05433b-b6c1-4bd0-88a1-11e769461d69\") "
Oct 31 12:54:34.780685 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:34.777878    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df05433b-b6c1-4bd0-88a1-11e769461d69-credential-socket" pod "df05433b-b6c1-4bd0-88a1-11e769461d69" (UID: "df05433b-b6c1-4bd0-88a1-11e769461d69"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:54:34.780685 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:34.778226    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df05433b-b6c1-4bd0-88a1-11e769461d69-istiod-ca-cert" pod "df05433b-b6c1-4bd0-88a1-11e769461d69" (UID: "df05433b-b6c1-4bd0-88a1-11e769461d69"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:54:34.780685 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:34.778430    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df05433b-b6c1-4bd0-88a1-11e769461d69-workload-socket" pod "df05433b-b6c1-4bd0-88a1-11e769461d69" (UID: "df05433b-b6c1-4bd0-88a1-11e769461d69"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:54:34.780685 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:34.778553    5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df05433b-b6c1-4bd0-88a1-11e769461d69-credential-socket\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:54:34.780685 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:34.778576    5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df05433b-b6c1-4bd0-88a1-11e769461d69-istiod-ca-cert\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:54:34.780685 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:34.778623    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df05433b-b6c1-4bd0-88a1-11e769461d69-istio-data" pod "df05433b-b6c1-4bd0-88a1-11e769461d69" (UID: "df05433b-b6c1-4bd0-88a1-11e769461d69"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:54:34.780685 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:34.778836    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df05433b-b6c1-4bd0-88a1-11e769461d69-workload-certs" pod "df05433b-b6c1-4bd0-88a1-11e769461d69" (UID: "df05433b-b6c1-4bd0-88a1-11e769461d69"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:54:34.780685 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:34.779744    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df05433b-b6c1-4bd0-88a1-11e769461d69-istio-envoy" pod "df05433b-b6c1-4bd0-88a1-11e769461d69" (UID: "df05433b-b6c1-4bd0-88a1-11e769461d69"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:54:34.780685 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:34.779858    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df05433b-b6c1-4bd0-88a1-11e769461d69-kube-api-access-pthhn" pod "df05433b-b6c1-4bd0-88a1-11e769461d69" (UID: "df05433b-b6c1-4bd0-88a1-11e769461d69"). InnerVolumeSpecName "kube-api-access-pthhn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:54:34.784952 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:34.784922    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/df05433b-b6c1-4bd0-88a1-11e769461d69-istio-podinfo" pod "df05433b-b6c1-4bd0-88a1-11e769461d69" (UID: "df05433b-b6c1-4bd0-88a1-11e769461d69"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 12:54:34.785259 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:34.785233    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df05433b-b6c1-4bd0-88a1-11e769461d69-istio-token" pod "df05433b-b6c1-4bd0-88a1-11e769461d69" (UID: "df05433b-b6c1-4bd0-88a1-11e769461d69"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:54:34.880278 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:34.880246    5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df05433b-b6c1-4bd0-88a1-11e769461d69-workload-socket\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:54:34.880278 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:34.880278    5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df05433b-b6c1-4bd0-88a1-11e769461d69-istio-data\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:54:34.880473 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:34.880290    5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df05433b-b6c1-4bd0-88a1-11e769461d69-istio-token\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:54:34.880473 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:34.880298    5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df05433b-b6c1-4bd0-88a1-11e769461d69-workload-certs\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:54:34.880473 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:34.880309    5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df05433b-b6c1-4bd0-88a1-11e769461d69-istio-podinfo\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:54:34.880473 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:34.880318    5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df05433b-b6c1-4bd0-88a1-11e769461d69-istio-envoy\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:54:34.880473 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:34.880332    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pthhn\" (UniqueName: \"kubernetes.io/projected/df05433b-b6c1-4bd0-88a1-11e769461d69-kube-api-access-pthhn\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:54:35.054092 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:35.054049    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ec2c204-58e3-4d0f-b06b-5112806add83" path="/var/lib/kubelet/pods/2ec2c204-58e3-4d0f-b06b-5112806add83/volumes"
Oct 31 12:54:35.270603 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:35.270481    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-58cq4"
Oct 31 12:54:35.270795 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:35.270524    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-58cq4" event={"ID":"c86d1b8b-6bc6-45ff-9412-179f069cf080","Type":"ContainerDied","Data":"c1b4131e812b90c5f160ed0d22334ecdf4a1e19946f47559cbc11c099f90472c"}
Oct 31 12:54:35.270795 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:35.270654    5060 scope.go:122] "RemoveContainer" containerID="436dd479231de8934ac874f816c15f4b0d01f071f8d5b37931d737444f7b4523"
Oct 31 12:54:35.273128 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:35.273097    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-f8gp2"
Oct 31 12:54:35.273269 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:35.273244    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-f8gp2" event={"ID":"df05433b-b6c1-4bd0-88a1-11e769461d69","Type":"ContainerDied","Data":"86f7523e6a052e987cc7c8a070144118d6f5d57526670bdf9f8fcb7df7308bc9"}
Oct 31 12:54:35.276147 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:35.276121    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d4dd68-xfh6r" event={"ID":"17e10309-cb84-4a27-b994-55863b40f305","Type":"ContainerStarted","Data":"bef6629e9804a28bc44e435352e7dd68d4480f04713f5aed4f01142413cd271e"}
Oct 31 12:54:35.283795 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:35.283777    5060 scope.go:122] "RemoveContainer" containerID="be887fbca1475bb75a00814d220dfaf71c3ea6f0884c58c1ac59ae8c2b8730cd"
Oct 31 12:54:35.308945 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:35.308916    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-58cq4"]
Oct 31 12:54:35.321411 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:35.321375    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-58cq4"]
Oct 31 12:54:35.348091 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:35.348031    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d4dd68-xfh6r" podStartSLOduration=2.348011803 podStartE2EDuration="2.348011803s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:54:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:54:35.326047509 +0000 UTC m=+1638.429115222" watchObservedRunningTime="2026-10-31 12:54:35.348011803 +0000 UTC m=+1638.451079517"
Oct 31 12:54:35.351713 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:35.351586    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-f8gp2"]
Oct 31 12:54:35.357270 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:35.357246    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-f8gp2"]
Oct 31 12:54:36.290574 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:36.290536    5060 generic.go:356] "Generic (PLEG): container finished" podID="17e10309-cb84-4a27-b994-55863b40f305" containerID="bef6629e9804a28bc44e435352e7dd68d4480f04713f5aed4f01142413cd271e" exitCode=0
Oct 31 12:54:36.291113 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:36.290615    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d4dd68-xfh6r" event={"ID":"17e10309-cb84-4a27-b994-55863b40f305","Type":"ContainerDied","Data":"bef6629e9804a28bc44e435352e7dd68d4480f04713f5aed4f01142413cd271e"}
Oct 31 12:54:36.609170 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:36.609079    5060 patch_prober.go:29] interesting pod/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-72mg2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 12:54:36.609170 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:36.609154    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-72mg2" podUID="346f4c45-c06a-4ae5-9fa0-c426467700f9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 12:54:37.053365 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:37.053331    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c86d1b8b-6bc6-45ff-9412-179f069cf080" path="/var/lib/kubelet/pods/c86d1b8b-6bc6-45ff-9412-179f069cf080/volumes"
Oct 31 12:54:37.054055 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:37.054031    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df05433b-b6c1-4bd0-88a1-11e769461d69" path="/var/lib/kubelet/pods/df05433b-b6c1-4bd0-88a1-11e769461d69/volumes"
Oct 31 12:54:37.482916 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:37.482890    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d4dd68-xfh6r"
Oct 31 12:54:37.539107 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:37.539065    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17e10309-cb84-4a27-b994-55863b40f305-kube-api-access-j4rjv\" (UniqueName: \"kubernetes.io/projected/17e10309-cb84-4a27-b994-55863b40f305-kube-api-access-j4rjv\") pod \"17e10309-cb84-4a27-b994-55863b40f305\" (UID: \"17e10309-cb84-4a27-b994-55863b40f305\") "
Oct 31 12:54:37.539107 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:37.539113    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/17e10309-cb84-4a27-b994-55863b40f305-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17e10309-cb84-4a27-b994-55863b40f305-maas-api-xks-ca\") pod \"17e10309-cb84-4a27-b994-55863b40f305\" (UID: \"17e10309-cb84-4a27-b994-55863b40f305\") "
Oct 31 12:54:37.539373 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:37.539243    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17e10309-cb84-4a27-b994-55863b40f305-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17e10309-cb84-4a27-b994-55863b40f305-maas-api-openshift-ca\") pod \"17e10309-cb84-4a27-b994-55863b40f305\" (UID: \"17e10309-cb84-4a27-b994-55863b40f305\") "
Oct 31 12:54:37.539711 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:37.539674    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17e10309-cb84-4a27-b994-55863b40f305-maas-api-openshift-ca" pod "17e10309-cb84-4a27-b994-55863b40f305" (UID: "17e10309-cb84-4a27-b994-55863b40f305"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:54:37.539935 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:37.539914    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17e10309-cb84-4a27-b994-55863b40f305-maas-api-openshift-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:54:37.541943 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:37.541912    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17e10309-cb84-4a27-b994-55863b40f305-maas-api-xks-ca" pod "17e10309-cb84-4a27-b994-55863b40f305" (UID: "17e10309-cb84-4a27-b994-55863b40f305"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:54:37.542064 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:37.542037    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17e10309-cb84-4a27-b994-55863b40f305-kube-api-access-j4rjv" pod "17e10309-cb84-4a27-b994-55863b40f305" (UID: "17e10309-cb84-4a27-b994-55863b40f305"). InnerVolumeSpecName "kube-api-access-j4rjv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:54:37.647652 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:37.645781    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4rjv\" (UniqueName: \"kubernetes.io/projected/17e10309-cb84-4a27-b994-55863b40f305-kube-api-access-j4rjv\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:54:37.647652 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:37.645828    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17e10309-cb84-4a27-b994-55863b40f305-maas-api-xks-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:54:37.816529 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:37.811684    5060 patch_prober.go:29] interesting pod/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-hkvl5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 12:54:37.816529 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:37.812094    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-hkvl5" podUID="4e300ab1-6fe3-4548-90b3-0cbb8099e114" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 12:54:38.272962 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:38.272927    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-72mg2"
Oct 31 12:54:38.330852 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:38.330817    5060 generic.go:356] "Generic (PLEG): container finished" podID="346f4c45-c06a-4ae5-9fa0-c426467700f9" containerID="ea8748ac2ab76bad1ceca1fd5047a546529d7eb073f2611aa57bbbb73e6ea836" exitCode=0
Oct 31 12:54:38.331257 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:38.331221    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-72mg2" event={"ID":"346f4c45-c06a-4ae5-9fa0-c426467700f9","Type":"ContainerDied","Data":"ea8748ac2ab76bad1ceca1fd5047a546529d7eb073f2611aa57bbbb73e6ea836"}
Oct 31 12:54:38.331413 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:38.331291    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-72mg2" event={"ID":"346f4c45-c06a-4ae5-9fa0-c426467700f9","Type":"ContainerDied","Data":"73f213b09a1900b7cd2db0acfaaf2b0e581dbcbc881d532dcdd7d3ad880d610a"}
Oct 31 12:54:38.331413 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:38.331325    5060 scope.go:122] "RemoveContainer" containerID="ea8748ac2ab76bad1ceca1fd5047a546529d7eb073f2611aa57bbbb73e6ea836"
Oct 31 12:54:38.331600 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:38.331464    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-72mg2"
Oct 31 12:54:38.331686 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:38.331664    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-hkvl5"
Oct 31 12:54:38.334147 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:38.334121    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d4dd68-xfh6r" event={"ID":"17e10309-cb84-4a27-b994-55863b40f305","Type":"ContainerDied","Data":"c9a6250278c202b86f58cc9f297118b8b5b64cb627efe5d7550b8789bf7a54b8"}
Oct 31 12:54:38.334251 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:38.334172    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c9a6250278c202b86f58cc9f297118b8b5b64cb627efe5d7550b8789bf7a54b8"
Oct 31 12:54:38.334251 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:38.334243    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d4dd68-xfh6r"
Oct 31 12:54:38.338712 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:38.338537    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-hkvl5" event={"ID":"4e300ab1-6fe3-4548-90b3-0cbb8099e114","Type":"ContainerDied","Data":"ff3576ccdf821939a0cd5ae44ea67bdc37f3e3711c25c2282a554d05810389d6"}
Oct 31 12:54:38.338712 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:38.338553    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-hkvl5"
Oct 31 12:54:38.338712 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:38.338371    5060 generic.go:356] "Generic (PLEG): container finished" podID="4e300ab1-6fe3-4548-90b3-0cbb8099e114" containerID="ff3576ccdf821939a0cd5ae44ea67bdc37f3e3711c25c2282a554d05810389d6" exitCode=0
Oct 31 12:54:38.338712 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:38.338629    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-hkvl5" event={"ID":"4e300ab1-6fe3-4548-90b3-0cbb8099e114","Type":"ContainerDied","Data":"3ec910f98a451578444bd506b9a47d60c26096043ad1f9d4b9b1da5a6186d9c1"}
Oct 31 12:54:38.358817 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:38.358794    5060 scope.go:122] "RemoveContainer" containerID="ea8748ac2ab76bad1ceca1fd5047a546529d7eb073f2611aa57bbbb73e6ea836"
Oct 31 12:54:38.359344 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:54:38.359322    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea8748ac2ab76bad1ceca1fd5047a546529d7eb073f2611aa57bbbb73e6ea836\": container with ID starting with ea8748ac2ab76bad1ceca1fd5047a546529d7eb073f2611aa57bbbb73e6ea836 not found: ID does not exist" containerID="ea8748ac2ab76bad1ceca1fd5047a546529d7eb073f2611aa57bbbb73e6ea836"
Oct 31 12:54:38.359481 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:38.359352    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea8748ac2ab76bad1ceca1fd5047a546529d7eb073f2611aa57bbbb73e6ea836"} err="failed to get container status \"ea8748ac2ab76bad1ceca1fd5047a546529d7eb073f2611aa57bbbb73e6ea836\": rpc error: code = NotFound desc = could not find container \"ea8748ac2ab76bad1ceca1fd5047a546529d7eb073f2611aa57bbbb73e6ea836\": container with ID starting with ea8748ac2ab76bad1ceca1fd5047a546529d7eb073f2611aa57bbbb73e6ea836 not found: ID does not exist"
Oct 31 12:54:38.359481 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:38.359370    5060 scope.go:122] "RemoveContainer" containerID="ff3576ccdf821939a0cd5ae44ea67bdc37f3e3711c25c2282a554d05810389d6"
Oct 31 12:54:38.375898 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:38.375871    5060 scope.go:122] "RemoveContainer" containerID="1c42065af8b553bdc7cf3f32752a5147a10231f9339cf04598ee8ae8db27912e"
Oct 31 12:54:38.382162 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:38.382132    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/346f4c45-c06a-4ae5-9fa0-c426467700f9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/346f4c45-c06a-4ae5-9fa0-c426467700f9-istiod-ca-cert\") pod \"346f4c45-c06a-4ae5-9fa0-c426467700f9\" (UID: \"346f4c45-c06a-4ae5-9fa0-c426467700f9\") "
Oct 31 12:54:38.382304 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:38.382182    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/346f4c45-c06a-4ae5-9fa0-c426467700f9-istio-token\" (UniqueName: \"kubernetes.io/projected/346f4c45-c06a-4ae5-9fa0-c426467700f9-istio-token\") pod \"346f4c45-c06a-4ae5-9fa0-c426467700f9\" (UID: \"346f4c45-c06a-4ae5-9fa0-c426467700f9\") "
Oct 31 12:54:38.382304 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:38.382243    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e300ab1-6fe3-4548-90b3-0cbb8099e114-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e300ab1-6fe3-4548-90b3-0cbb8099e114-credential-socket\") pod \"4e300ab1-6fe3-4548-90b3-0cbb8099e114\" (UID: \"4e300ab1-6fe3-4548-90b3-0cbb8099e114\") "
Oct 31 12:54:38.382304 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:38.382278    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/346f4c45-c06a-4ae5-9fa0-c426467700f9-kube-api-access-wqkjq\" (UniqueName: \"kubernetes.io/projected/346f4c45-c06a-4ae5-9fa0-c426467700f9-kube-api-access-wqkjq\") pod \"346f4c45-c06a-4ae5-9fa0-c426467700f9\" (UID: \"346f4c45-c06a-4ae5-9fa0-c426467700f9\") "
Oct 31 12:54:38.382557 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:38.382312    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e300ab1-6fe3-4548-90b3-0cbb8099e114-kube-api-access-9sx4j\" (UniqueName: \"kubernetes.io/projected/4e300ab1-6fe3-4548-90b3-0cbb8099e114-kube-api-access-9sx4j\") pod \"4e300ab1-6fe3-4548-90b3-0cbb8099e114\" (UID: \"4e300ab1-6fe3-4548-90b3-0cbb8099e114\") "
Oct 31 12:54:38.382557 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:38.382343    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/346f4c45-c06a-4ae5-9fa0-c426467700f9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/346f4c45-c06a-4ae5-9fa0-c426467700f9-istio-podinfo\") pod \"346f4c45-c06a-4ae5-9fa0-c426467700f9\" (UID: \"346f4c45-c06a-4ae5-9fa0-c426467700f9\") "
Oct 31 12:54:38.382557 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:38.382448    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e300ab1-6fe3-4548-90b3-0cbb8099e114-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e300ab1-6fe3-4548-90b3-0cbb8099e114-workload-socket\") pod \"4e300ab1-6fe3-4548-90b3-0cbb8099e114\" (UID: \"4e300ab1-6fe3-4548-90b3-0cbb8099e114\") "
Oct 31 12:54:38.382557 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:38.382554    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e300ab1-6fe3-4548-90b3-0cbb8099e114-istio-token\" (UniqueName: \"kubernetes.io/projected/4e300ab1-6fe3-4548-90b3-0cbb8099e114-istio-token\") pod \"4e300ab1-6fe3-4548-90b3-0cbb8099e114\" (UID: \"4e300ab1-6fe3-4548-90b3-0cbb8099e114\") "
Oct 31 12:54:38.382716 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:38.382596    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/346f4c45-c06a-4ae5-9fa0-c426467700f9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/346f4c45-c06a-4ae5-9fa0-c426467700f9-istio-envoy\") pod \"346f4c45-c06a-4ae5-9fa0-c426467700f9\" (UID: \"346f4c45-c06a-4ae5-9fa0-c426467700f9\") "
Oct 31 12:54:38.383076 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:38.383053    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/346f4c45-c06a-4ae5-9fa0-c426467700f9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/346f4c45-c06a-4ae5-9fa0-c426467700f9-istio-data\") pod \"346f4c45-c06a-4ae5-9fa0-c426467700f9\" (UID: \"346f4c45-c06a-4ae5-9fa0-c426467700f9\") "
Oct 31 12:54:38.383172 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:38.383157    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/346f4c45-c06a-4ae5-9fa0-c426467700f9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/346f4c45-c06a-4ae5-9fa0-c426467700f9-credential-socket\") pod \"346f4c45-c06a-4ae5-9fa0-c426467700f9\" (UID: \"346f4c45-c06a-4ae5-9fa0-c426467700f9\") "
Oct 31 12:54:38.383261 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:38.383244    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e300ab1-6fe3-4548-90b3-0cbb8099e114-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e300ab1-6fe3-4548-90b3-0cbb8099e114-workload-certs\") pod \"4e300ab1-6fe3-4548-90b3-0cbb8099e114\" (UID: \"4e300ab1-6fe3-4548-90b3-0cbb8099e114\") "
Oct 31 12:54:38.383330 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:38.383292    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/346f4c45-c06a-4ae5-9fa0-c426467700f9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/346f4c45-c06a-4ae5-9fa0-c426467700f9-workload-certs\") pod \"346f4c45-c06a-4ae5-9fa0-c426467700f9\" (UID: \"346f4c45-c06a-4ae5-9fa0-c426467700f9\") "
Oct 31 12:54:38.383390 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:38.383349    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4e300ab1-6fe3-4548-90b3-0cbb8099e114-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e300ab1-6fe3-4548-90b3-0cbb8099e114-istio-podinfo\") pod \"4e300ab1-6fe3-4548-90b3-0cbb8099e114\" (UID: \"4e300ab1-6fe3-4548-90b3-0cbb8099e114\") "
Oct 31 12:54:38.383390 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:38.383374    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e300ab1-6fe3-4548-90b3-0cbb8099e114-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e300ab1-6fe3-4548-90b3-0cbb8099e114-istio-envoy\") pod \"4e300ab1-6fe3-4548-90b3-0cbb8099e114\" (UID: \"4e300ab1-6fe3-4548-90b3-0cbb8099e114\") "
Oct 31 12:54:38.383525 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:38.383402    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/346f4c45-c06a-4ae5-9fa0-c426467700f9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/346f4c45-c06a-4ae5-9fa0-c426467700f9-workload-socket\") pod \"346f4c45-c06a-4ae5-9fa0-c426467700f9\" (UID: \"346f4c45-c06a-4ae5-9fa0-c426467700f9\") "
Oct 31 12:54:38.383525 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:38.383448    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e300ab1-6fe3-4548-90b3-0cbb8099e114-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e300ab1-6fe3-4548-90b3-0cbb8099e114-istiod-ca-cert\") pod \"4e300ab1-6fe3-4548-90b3-0cbb8099e114\" (UID: \"4e300ab1-6fe3-4548-90b3-0cbb8099e114\") "
Oct 31 12:54:38.383525 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:38.383477    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e300ab1-6fe3-4548-90b3-0cbb8099e114-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e300ab1-6fe3-4548-90b3-0cbb8099e114-istio-data\") pod \"4e300ab1-6fe3-4548-90b3-0cbb8099e114\" (UID: \"4e300ab1-6fe3-4548-90b3-0cbb8099e114\") "
Oct 31 12:54:38.385034 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:38.385005    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/346f4c45-c06a-4ae5-9fa0-c426467700f9-credential-socket" pod "346f4c45-c06a-4ae5-9fa0-c426467700f9" (UID: "346f4c45-c06a-4ae5-9fa0-c426467700f9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:54:38.385200 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:38.385051    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/346f4c45-c06a-4ae5-9fa0-c426467700f9-istiod-ca-cert" pod "346f4c45-c06a-4ae5-9fa0-c426467700f9" (UID: "346f4c45-c06a-4ae5-9fa0-c426467700f9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:54:38.385335 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:38.385308    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/346f4c45-c06a-4ae5-9fa0-c426467700f9-kube-api-access-wqkjq" pod "346f4c45-c06a-4ae5-9fa0-c426467700f9" (UID: "346f4c45-c06a-4ae5-9fa0-c426467700f9"). InnerVolumeSpecName "kube-api-access-wqkjq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:54:38.385864 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:38.385605    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e300ab1-6fe3-4548-90b3-0cbb8099e114-credential-socket" pod "4e300ab1-6fe3-4548-90b3-0cbb8099e114" (UID: "4e300ab1-6fe3-4548-90b3-0cbb8099e114"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:54:38.386558 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:38.386330    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/346f4c45-c06a-4ae5-9fa0-c426467700f9-istio-data" pod "346f4c45-c06a-4ae5-9fa0-c426467700f9" (UID: "346f4c45-c06a-4ae5-9fa0-c426467700f9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:54:38.386687 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:38.386634    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e300ab1-6fe3-4548-90b3-0cbb8099e114-workload-socket" pod "4e300ab1-6fe3-4548-90b3-0cbb8099e114" (UID: "4e300ab1-6fe3-4548-90b3-0cbb8099e114"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:54:38.386922 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:38.386899    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/346f4c45-c06a-4ae5-9fa0-c426467700f9-workload-socket" pod "346f4c45-c06a-4ae5-9fa0-c426467700f9" (UID: "346f4c45-c06a-4ae5-9fa0-c426467700f9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:54:38.387136 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:38.387111    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e300ab1-6fe3-4548-90b3-0cbb8099e114-istio-data" pod "4e300ab1-6fe3-4548-90b3-0cbb8099e114" (UID: "4e300ab1-6fe3-4548-90b3-0cbb8099e114"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:54:38.388615 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:38.387677    5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/346f4c45-c06a-4ae5-9fa0-c426467700f9-istiod-ca-cert\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:54:38.388615 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:38.387706    5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e300ab1-6fe3-4548-90b3-0cbb8099e114-credential-socket\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:54:38.388615 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:38.387720    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqkjq\" (UniqueName: \"kubernetes.io/projected/346f4c45-c06a-4ae5-9fa0-c426467700f9-kube-api-access-wqkjq\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:54:38.388615 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:38.387735    5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e300ab1-6fe3-4548-90b3-0cbb8099e114-workload-socket\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:54:38.388615 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:38.387748    5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/346f4c45-c06a-4ae5-9fa0-c426467700f9-istio-data\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:54:38.388615 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:38.387762    5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/346f4c45-c06a-4ae5-9fa0-c426467700f9-credential-socket\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:54:38.388615 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:38.387776    5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/346f4c45-c06a-4ae5-9fa0-c426467700f9-workload-socket\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:54:38.388615 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:38.387791    5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e300ab1-6fe3-4548-90b3-0cbb8099e114-istio-data\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:54:38.392288 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:38.392263    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e300ab1-6fe3-4548-90b3-0cbb8099e114-workload-certs" pod "4e300ab1-6fe3-4548-90b3-0cbb8099e114" (UID: "4e300ab1-6fe3-4548-90b3-0cbb8099e114"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:54:38.392636 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:38.392615    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/346f4c45-c06a-4ae5-9fa0-c426467700f9-workload-certs" pod "346f4c45-c06a-4ae5-9fa0-c426467700f9" (UID: "346f4c45-c06a-4ae5-9fa0-c426467700f9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:54:38.393523 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:38.393444    5060 scope.go:122] "RemoveContainer" containerID="ff3576ccdf821939a0cd5ae44ea67bdc37f3e3711c25c2282a554d05810389d6"
Oct 31 12:54:38.394035 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:38.393887    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e300ab1-6fe3-4548-90b3-0cbb8099e114-istiod-ca-cert" pod "4e300ab1-6fe3-4548-90b3-0cbb8099e114" (UID: "4e300ab1-6fe3-4548-90b3-0cbb8099e114"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:54:38.394597 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:54:38.394228    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff3576ccdf821939a0cd5ae44ea67bdc37f3e3711c25c2282a554d05810389d6\": container with ID starting with ff3576ccdf821939a0cd5ae44ea67bdc37f3e3711c25c2282a554d05810389d6 not found: ID does not exist" containerID="ff3576ccdf821939a0cd5ae44ea67bdc37f3e3711c25c2282a554d05810389d6"
Oct 31 12:54:38.394597 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:38.394270    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff3576ccdf821939a0cd5ae44ea67bdc37f3e3711c25c2282a554d05810389d6"} err="failed to get container status \"ff3576ccdf821939a0cd5ae44ea67bdc37f3e3711c25c2282a554d05810389d6\": rpc error: code = NotFound desc = could not find container \"ff3576ccdf821939a0cd5ae44ea67bdc37f3e3711c25c2282a554d05810389d6\": container with ID starting with ff3576ccdf821939a0cd5ae44ea67bdc37f3e3711c25c2282a554d05810389d6 not found: ID does not exist"
Oct 31 12:54:38.394597 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:38.394293    5060 scope.go:122] "RemoveContainer" containerID="1c42065af8b553bdc7cf3f32752a5147a10231f9339cf04598ee8ae8db27912e"
Oct 31 12:54:38.395025 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:54:38.395004    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c42065af8b553bdc7cf3f32752a5147a10231f9339cf04598ee8ae8db27912e\": container with ID starting with 1c42065af8b553bdc7cf3f32752a5147a10231f9339cf04598ee8ae8db27912e not found: ID does not exist" containerID="1c42065af8b553bdc7cf3f32752a5147a10231f9339cf04598ee8ae8db27912e"
Oct 31 12:54:38.395154 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:38.395133    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c42065af8b553bdc7cf3f32752a5147a10231f9339cf04598ee8ae8db27912e"} err="failed to get container status \"1c42065af8b553bdc7cf3f32752a5147a10231f9339cf04598ee8ae8db27912e\": rpc error: code = NotFound desc = could not find container \"1c42065af8b553bdc7cf3f32752a5147a10231f9339cf04598ee8ae8db27912e\": container with ID starting with 1c42065af8b553bdc7cf3f32752a5147a10231f9339cf04598ee8ae8db27912e not found: ID does not exist"
Oct 31 12:54:38.395348 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:38.395321    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4e300ab1-6fe3-4548-90b3-0cbb8099e114-istio-podinfo" pod "4e300ab1-6fe3-4548-90b3-0cbb8099e114" (UID: "4e300ab1-6fe3-4548-90b3-0cbb8099e114"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 12:54:38.398090 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:38.398066    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/346f4c45-c06a-4ae5-9fa0-c426467700f9-istio-token" pod "346f4c45-c06a-4ae5-9fa0-c426467700f9" (UID: "346f4c45-c06a-4ae5-9fa0-c426467700f9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:54:38.423537 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:38.420131    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e300ab1-6fe3-4548-90b3-0cbb8099e114-istio-token" pod "4e300ab1-6fe3-4548-90b3-0cbb8099e114" (UID: "4e300ab1-6fe3-4548-90b3-0cbb8099e114"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:54:38.423537 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:38.420925    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/346f4c45-c06a-4ae5-9fa0-c426467700f9-istio-envoy" pod "346f4c45-c06a-4ae5-9fa0-c426467700f9" (UID: "346f4c45-c06a-4ae5-9fa0-c426467700f9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:54:38.542513 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:38.540510    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e300ab1-6fe3-4548-90b3-0cbb8099e114-istio-envoy" pod "4e300ab1-6fe3-4548-90b3-0cbb8099e114" (UID: "4e300ab1-6fe3-4548-90b3-0cbb8099e114"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:54:38.543014 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:38.542610    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/346f4c45-c06a-4ae5-9fa0-c426467700f9-istio-podinfo" pod "346f4c45-c06a-4ae5-9fa0-c426467700f9" (UID: "346f4c45-c06a-4ae5-9fa0-c426467700f9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 12:54:38.655357 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:38.655312    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e300ab1-6fe3-4548-90b3-0cbb8099e114-kube-api-access-9sx4j" pod "4e300ab1-6fe3-4548-90b3-0cbb8099e114" (UID: "4e300ab1-6fe3-4548-90b3-0cbb8099e114"). InnerVolumeSpecName "kube-api-access-9sx4j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:54:38.694515 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:38.692918    5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e300ab1-6fe3-4548-90b3-0cbb8099e114-workload-certs\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:54:38.694515 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:38.692957    5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/346f4c45-c06a-4ae5-9fa0-c426467700f9-workload-certs\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:54:38.694515 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:38.692970    5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e300ab1-6fe3-4548-90b3-0cbb8099e114-istio-podinfo\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:54:38.694515 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:38.692984    5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e300ab1-6fe3-4548-90b3-0cbb8099e114-istio-envoy\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:54:38.694515 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:38.693002    5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e300ab1-6fe3-4548-90b3-0cbb8099e114-istiod-ca-cert\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:54:38.694515 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:38.693018    5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/346f4c45-c06a-4ae5-9fa0-c426467700f9-istio-token\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:54:38.694515 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:38.693032    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9sx4j\" (UniqueName: \"kubernetes.io/projected/4e300ab1-6fe3-4548-90b3-0cbb8099e114-kube-api-access-9sx4j\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:54:38.694515 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:38.693045    5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/346f4c45-c06a-4ae5-9fa0-c426467700f9-istio-podinfo\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:54:38.694515 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:38.693057    5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e300ab1-6fe3-4548-90b3-0cbb8099e114-istio-token\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:54:38.694515 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:38.693072    5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/346f4c45-c06a-4ae5-9fa0-c426467700f9-istio-envoy\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:54:38.799964 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:38.799882    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-hkvl5"]
Oct 31 12:54:38.807237 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:38.807204    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-hkvl5"]
Oct 31 12:54:38.817199 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:38.817144    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-72mg2"]
Oct 31 12:54:38.821646 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:38.821620    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-72mg2"]
Oct 31 12:54:39.270723 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:39.270638    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="346f4c45-c06a-4ae5-9fa0-c426467700f9" path="/var/lib/kubelet/pods/346f4c45-c06a-4ae5-9fa0-c426467700f9/volumes"
Oct 31 12:54:39.272076 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:39.272048    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e300ab1-6fe3-4548-90b3-0cbb8099e114" path="/var/lib/kubelet/pods/4e300ab1-6fe3-4548-90b3-0cbb8099e114/volumes"
Oct 31 12:54:49.893518 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:49.890542    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b218d3-khq4b"]
Oct 31 12:54:49.979221 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:49.974767    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b218d3-khq4b"]
Oct 31 12:54:51.051759 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:51.051723    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7972aee8-dbdd-4c32-b46d-6cdf855aa53d" path="/var/lib/kubelet/pods/7972aee8-dbdd-4c32-b46d-6cdf855aa53d/volumes"
Oct 31 12:54:51.073255 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:51.073212    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d4dd68-xfh6r"]
Oct 31 12:54:51.077620 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:51.077581    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d4dd68-xfh6r"]
Oct 31 12:54:53.064332 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:53.059259    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17e10309-cb84-4a27-b994-55863b40f305" path="/var/lib/kubelet/pods/17e10309-cb84-4a27-b994-55863b40f305/volumes"
Oct 31 12:54:54.746522 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:54.746362    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-jnpqm"]
Oct 31 12:54:54.747023 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:54.746765    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-jnpqm" podUID="fe33e47d-4eb8-411b-8320-21135401155b" containerName="istio-proxy" containerID="cri-o://225fbe10c1ebd91d1bcec7e34a4d214f560578e77150fd8df8a40f988588486c" gracePeriod=30
Oct 31 12:54:56.529850 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:56.529793    5060 patch_prober.go:29] interesting pod/e2e-worker-w6-67d410-openshift-default-676b978549-jnpqm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 12:54:56.530226 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:56.529868    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-jnpqm" podUID="fe33e47d-4eb8-411b-8320-21135401155b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 12:54:59.894586 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:59.893736    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-2fn6t"]
Oct 31 12:54:59.894586 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:59.893973    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-2fn6t" podUID="60d395e9-e4e9-4c68-ab10-9ef5371df1b1" containerName="istio-proxy" containerID="cri-o://6576cc6e6a88037c75e3d89b906c3e21c090af7d8a2f8e801ef2a27753a51791" gracePeriod=30
Oct 31 12:54:59.901820 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:59.901790    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-28t2l"]
Oct 31 12:54:59.902015 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:54:59.901991    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-28t2l" podUID="66295870-6cfc-4f48-b353-566fe73a6bf0" containerName="istio-proxy" containerID="cri-o://4b22833c781d46e9c40b427b1113815ec8395e35c2c4ba287f4237abcc240d0b" gracePeriod=30
Oct 31 12:54:59.978998 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:54:59.978950    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7phb" containerName="extractor"
Oct 31 12:55:00.644307 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:00.644281    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-jnpqm"
Oct 31 12:55:00.671002 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:00.669132    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fe33e47d-4eb8-411b-8320-21135401155b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe33e47d-4eb8-411b-8320-21135401155b-istio-podinfo\") pod \"fe33e47d-4eb8-411b-8320-21135401155b\" (UID: \"fe33e47d-4eb8-411b-8320-21135401155b\") "
Oct 31 12:55:00.671002 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:00.669255    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe33e47d-4eb8-411b-8320-21135401155b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe33e47d-4eb8-411b-8320-21135401155b-istio-envoy\") pod \"fe33e47d-4eb8-411b-8320-21135401155b\" (UID: \"fe33e47d-4eb8-411b-8320-21135401155b\") "
Oct 31 12:55:00.671002 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:00.669341    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe33e47d-4eb8-411b-8320-21135401155b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe33e47d-4eb8-411b-8320-21135401155b-workload-certs\") pod \"fe33e47d-4eb8-411b-8320-21135401155b\" (UID: \"fe33e47d-4eb8-411b-8320-21135401155b\") "
Oct 31 12:55:00.678528 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:00.677321    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe33e47d-4eb8-411b-8320-21135401155b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe33e47d-4eb8-411b-8320-21135401155b-credential-socket\") pod \"fe33e47d-4eb8-411b-8320-21135401155b\" (UID: \"fe33e47d-4eb8-411b-8320-21135401155b\") "
Oct 31 12:55:00.678528 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:00.677371    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe33e47d-4eb8-411b-8320-21135401155b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe33e47d-4eb8-411b-8320-21135401155b-istiod-ca-cert\") pod \"fe33e47d-4eb8-411b-8320-21135401155b\" (UID: \"fe33e47d-4eb8-411b-8320-21135401155b\") "
Oct 31 12:55:00.678528 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:00.677408    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe33e47d-4eb8-411b-8320-21135401155b-kube-api-access-pqg8z\" (UniqueName: \"kubernetes.io/projected/fe33e47d-4eb8-411b-8320-21135401155b-kube-api-access-pqg8z\") pod \"fe33e47d-4eb8-411b-8320-21135401155b\" (UID: \"fe33e47d-4eb8-411b-8320-21135401155b\") "
Oct 31 12:55:00.678528 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:00.677443    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe33e47d-4eb8-411b-8320-21135401155b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe33e47d-4eb8-411b-8320-21135401155b-workload-socket\") pod \"fe33e47d-4eb8-411b-8320-21135401155b\" (UID: \"fe33e47d-4eb8-411b-8320-21135401155b\") "
Oct 31 12:55:00.678528 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:00.677543    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe33e47d-4eb8-411b-8320-21135401155b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe33e47d-4eb8-411b-8320-21135401155b-istio-data\") pod \"fe33e47d-4eb8-411b-8320-21135401155b\" (UID: \"fe33e47d-4eb8-411b-8320-21135401155b\") "
Oct 31 12:55:00.678528 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:00.677682    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe33e47d-4eb8-411b-8320-21135401155b-istio-token\" (UniqueName: \"kubernetes.io/projected/fe33e47d-4eb8-411b-8320-21135401155b-istio-token\") pod \"fe33e47d-4eb8-411b-8320-21135401155b\" (UID: \"fe33e47d-4eb8-411b-8320-21135401155b\") "
Oct 31 12:55:00.679482 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:00.679274    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe33e47d-4eb8-411b-8320-21135401155b-istiod-ca-cert" pod "fe33e47d-4eb8-411b-8320-21135401155b" (UID: "fe33e47d-4eb8-411b-8320-21135401155b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:55:00.679821 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:00.679668    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe33e47d-4eb8-411b-8320-21135401155b-workload-certs" pod "fe33e47d-4eb8-411b-8320-21135401155b" (UID: "fe33e47d-4eb8-411b-8320-21135401155b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:55:00.680587 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:00.679982    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe33e47d-4eb8-411b-8320-21135401155b-credential-socket" pod "fe33e47d-4eb8-411b-8320-21135401155b" (UID: "fe33e47d-4eb8-411b-8320-21135401155b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:55:00.680925 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:00.680733    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe33e47d-4eb8-411b-8320-21135401155b-workload-socket" pod "fe33e47d-4eb8-411b-8320-21135401155b" (UID: "fe33e47d-4eb8-411b-8320-21135401155b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:55:00.681514 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:00.681047    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe33e47d-4eb8-411b-8320-21135401155b-istio-data" pod "fe33e47d-4eb8-411b-8320-21135401155b" (UID: "fe33e47d-4eb8-411b-8320-21135401155b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:55:00.803528 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:00.750733    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe33e47d-4eb8-411b-8320-21135401155b-istio-envoy" pod "fe33e47d-4eb8-411b-8320-21135401155b" (UID: "fe33e47d-4eb8-411b-8320-21135401155b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:55:00.803528 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:00.771805    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe33e47d-4eb8-411b-8320-21135401155b-istio-token" pod "fe33e47d-4eb8-411b-8320-21135401155b" (UID: "fe33e47d-4eb8-411b-8320-21135401155b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:55:00.832513 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:00.803992    5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe33e47d-4eb8-411b-8320-21135401155b-workload-certs\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:55:00.832513 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:00.804031    5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe33e47d-4eb8-411b-8320-21135401155b-credential-socket\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:55:00.832513 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:00.804046    5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe33e47d-4eb8-411b-8320-21135401155b-istiod-ca-cert\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:55:00.832513 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:00.804059    5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe33e47d-4eb8-411b-8320-21135401155b-workload-socket\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:55:00.832513 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:00.804072    5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe33e47d-4eb8-411b-8320-21135401155b-istio-data\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:55:00.832513 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:00.804006    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fe33e47d-4eb8-411b-8320-21135401155b-istio-podinfo" pod "fe33e47d-4eb8-411b-8320-21135401155b" (UID: "fe33e47d-4eb8-411b-8320-21135401155b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 12:55:00.832513 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:00.804087    5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe33e47d-4eb8-411b-8320-21135401155b-istio-token\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:55:00.832513 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:00.804139    5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe33e47d-4eb8-411b-8320-21135401155b-istio-envoy\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:55:00.832513 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:00.826874    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe33e47d-4eb8-411b-8320-21135401155b-kube-api-access-pqg8z" pod "fe33e47d-4eb8-411b-8320-21135401155b" (UID: "fe33e47d-4eb8-411b-8320-21135401155b"). InnerVolumeSpecName "kube-api-access-pqg8z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:55:00.873582 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:00.862432    5060 generic.go:356] "Generic (PLEG): container finished" podID="fe33e47d-4eb8-411b-8320-21135401155b" containerID="225fbe10c1ebd91d1bcec7e34a4d214f560578e77150fd8df8a40f988588486c" exitCode=0
Oct 31 12:55:00.873582 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:00.862761    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-jnpqm" event={"ID":"fe33e47d-4eb8-411b-8320-21135401155b","Type":"ContainerDied","Data":"225fbe10c1ebd91d1bcec7e34a4d214f560578e77150fd8df8a40f988588486c"}
Oct 31 12:55:00.873582 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:00.862850    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-jnpqm" event={"ID":"fe33e47d-4eb8-411b-8320-21135401155b","Type":"ContainerDied","Data":"e38284c36244ff49f621a9313984ef52ae5792dd3c02262a1f061a2cdb46a5c0"}
Oct 31 12:55:00.873582 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:00.862881    5060 scope.go:122] "RemoveContainer" containerID="225fbe10c1ebd91d1bcec7e34a4d214f560578e77150fd8df8a40f988588486c"
Oct 31 12:55:00.873582 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:00.863048    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-jnpqm"
Oct 31 12:55:00.908768 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:00.908684    5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe33e47d-4eb8-411b-8320-21135401155b-istio-podinfo\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:55:00.908768 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:00.908719    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pqg8z\" (UniqueName: \"kubernetes.io/projected/fe33e47d-4eb8-411b-8320-21135401155b-kube-api-access-pqg8z\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:55:00.914098 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:00.913965    5060 scope.go:122] "RemoveContainer" containerID="225fbe10c1ebd91d1bcec7e34a4d214f560578e77150fd8df8a40f988588486c"
Oct 31 12:55:00.914435 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:55:00.914363    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"225fbe10c1ebd91d1bcec7e34a4d214f560578e77150fd8df8a40f988588486c\": container with ID starting with 225fbe10c1ebd91d1bcec7e34a4d214f560578e77150fd8df8a40f988588486c not found: ID does not exist" containerID="225fbe10c1ebd91d1bcec7e34a4d214f560578e77150fd8df8a40f988588486c"
Oct 31 12:55:00.914435 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:00.914409    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"225fbe10c1ebd91d1bcec7e34a4d214f560578e77150fd8df8a40f988588486c"} err="failed to get container status \"225fbe10c1ebd91d1bcec7e34a4d214f560578e77150fd8df8a40f988588486c\": rpc error: code = NotFound desc = could not find container \"225fbe10c1ebd91d1bcec7e34a4d214f560578e77150fd8df8a40f988588486c\": container with ID starting with 225fbe10c1ebd91d1bcec7e34a4d214f560578e77150fd8df8a40f988588486c not found: ID does not exist"
Oct 31 12:55:00.936539 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:00.935839    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-jnpqm"]
Oct 31 12:55:01.008642 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:00.997331    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-jnpqm"]
Oct 31 12:55:01.084244 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:01.084205    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe33e47d-4eb8-411b-8320-21135401155b" path="/var/lib/kubelet/pods/fe33e47d-4eb8-411b-8320-21135401155b/volumes"
Oct 31 12:55:01.275529 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:01.275475    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e6c79d-kh295"]
Oct 31 12:55:01.276276 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:01.276256    5060 cpu_manager.go:414] "containerMap: removing container" podUID="4e300ab1-6fe3-4548-90b3-0cbb8099e114" containerName="istio-proxy"
Oct 31 12:55:01.276332 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:01.276280    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e300ab1-6fe3-4548-90b3-0cbb8099e114" containerName="istio-proxy"
Oct 31 12:55:01.276332 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:01.276300    5060 cpu_manager.go:414] "containerMap: removing container" podUID="4e300ab1-6fe3-4548-90b3-0cbb8099e114" containerName="istio-proxy"
Oct 31 12:55:01.276332 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:01.276308    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e300ab1-6fe3-4548-90b3-0cbb8099e114" containerName="istio-proxy"
Oct 31 12:55:01.276332 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:01.276330    5060 cpu_manager.go:414] "containerMap: removing container" podUID="fe33e47d-4eb8-411b-8320-21135401155b" containerName="istio-proxy"
Oct 31 12:55:01.276501 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:01.276337    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe33e47d-4eb8-411b-8320-21135401155b" containerName="istio-proxy"
Oct 31 12:55:01.276501 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:01.276358    5060 cpu_manager.go:414] "containerMap: removing container" podUID="df05433b-b6c1-4bd0-88a1-11e769461d69" containerName="istio-proxy"
Oct 31 12:55:01.276501 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:01.276365    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df05433b-b6c1-4bd0-88a1-11e769461d69" containerName="istio-proxy"
Oct 31 12:55:01.276501 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:01.276416    5060 cpu_manager.go:414] "containerMap: removing container" podUID="c86d1b8b-6bc6-45ff-9412-179f069cf080" containerName="istio-proxy"
Oct 31 12:55:01.276501 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:01.276424    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c86d1b8b-6bc6-45ff-9412-179f069cf080" containerName="istio-proxy"
Oct 31 12:55:01.276501 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:01.276450    5060 cpu_manager.go:414] "containerMap: removing container" podUID="17e10309-cb84-4a27-b994-55863b40f305" containerName="revoke-keys"
Oct 31 12:55:01.276501 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:01.276459    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17e10309-cb84-4a27-b994-55863b40f305" containerName="revoke-keys"
Oct 31 12:55:01.276501 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:01.276476    5060 cpu_manager.go:414] "containerMap: removing container" podUID="346f4c45-c06a-4ae5-9fa0-c426467700f9" containerName="istio-proxy"
Oct 31 12:55:01.276886 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:01.276506    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="346f4c45-c06a-4ae5-9fa0-c426467700f9" containerName="istio-proxy"
Oct 31 12:55:01.276886 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:01.276680    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe33e47d-4eb8-411b-8320-21135401155b" containerName="istio-proxy"
Oct 31 12:55:01.276886 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:01.276699    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="df05433b-b6c1-4bd0-88a1-11e769461d69" containerName="istio-proxy"
Oct 31 12:55:01.276886 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:01.276716    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="c86d1b8b-6bc6-45ff-9412-179f069cf080" containerName="istio-proxy"
Oct 31 12:55:01.276886 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:01.276729    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="c86d1b8b-6bc6-45ff-9412-179f069cf080" containerName="istio-proxy"
Oct 31 12:55:01.276886 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:01.276740    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="17e10309-cb84-4a27-b994-55863b40f305" containerName="revoke-keys"
Oct 31 12:55:01.276886 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:01.276753    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="346f4c45-c06a-4ae5-9fa0-c426467700f9" containerName="istio-proxy"
Oct 31 12:55:01.276886 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:01.276771    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e300ab1-6fe3-4548-90b3-0cbb8099e114" containerName="istio-proxy"
Oct 31 12:55:01.276886 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:01.276781    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e300ab1-6fe3-4548-90b3-0cbb8099e114" containerName="istio-proxy"
Oct 31 12:55:01.282159 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:01.282138    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e6c79d-kh295"
Oct 31 12:55:01.287922 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:01.287547    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 12:55:01.288264 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:01.288202    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7wsmb\""
Oct 31 12:55:01.293335 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:01.293308    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e6c79d-kh295"]
Oct 31 12:55:01.328057 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:01.326825    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e849745c-8575-4bf1-81d9-f032ddd5e9fe-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-e6c79d-kh295\" (UID: \"e849745c-8575-4bf1-81d9-f032ddd5e9fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e6c79d-kh295"
Oct 31 12:55:01.328057 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:01.327116    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xw84b\" (UniqueName: \"kubernetes.io/projected/e849745c-8575-4bf1-81d9-f032ddd5e9fe-kube-api-access-xw84b\") pod \"maas-api-revoke-keys-e2e-worker-w2-e6c79d-kh295\" (UID: \"e849745c-8575-4bf1-81d9-f032ddd5e9fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e6c79d-kh295"
Oct 31 12:55:01.328057 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:01.327238    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e849745c-8575-4bf1-81d9-f032ddd5e9fe-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-e6c79d-kh295\" (UID: \"e849745c-8575-4bf1-81d9-f032ddd5e9fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e6c79d-kh295"
Oct 31 12:55:01.429002 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:01.428961    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e849745c-8575-4bf1-81d9-f032ddd5e9fe-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-e6c79d-kh295\" (UID: \"e849745c-8575-4bf1-81d9-f032ddd5e9fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e6c79d-kh295"
Oct 31 12:55:01.429257 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:01.429235    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xw84b\" (UniqueName: \"kubernetes.io/projected/e849745c-8575-4bf1-81d9-f032ddd5e9fe-kube-api-access-xw84b\") pod \"maas-api-revoke-keys-e2e-worker-w2-e6c79d-kh295\" (UID: \"e849745c-8575-4bf1-81d9-f032ddd5e9fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e6c79d-kh295"
Oct 31 12:55:01.429374 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:01.429356    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e849745c-8575-4bf1-81d9-f032ddd5e9fe-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-e6c79d-kh295\" (UID: \"e849745c-8575-4bf1-81d9-f032ddd5e9fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e6c79d-kh295"
Oct 31 12:55:01.430228 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:01.430196    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e849745c-8575-4bf1-81d9-f032ddd5e9fe-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-e6c79d-kh295\" (UID: \"e849745c-8575-4bf1-81d9-f032ddd5e9fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e6c79d-kh295"
Oct 31 12:55:01.431667 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:01.431637    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e849745c-8575-4bf1-81d9-f032ddd5e9fe-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-e6c79d-kh295\" (UID: \"e849745c-8575-4bf1-81d9-f032ddd5e9fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e6c79d-kh295"
Oct 31 12:55:01.442685 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:01.442654    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xw84b\" (UniqueName: \"kubernetes.io/projected/e849745c-8575-4bf1-81d9-f032ddd5e9fe-kube-api-access-xw84b\") pod \"maas-api-revoke-keys-e2e-worker-w2-e6c79d-kh295\" (UID: \"e849745c-8575-4bf1-81d9-f032ddd5e9fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e6c79d-kh295"
Oct 31 12:55:01.642124 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:01.642034    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e6c79d-kh295"
Oct 31 12:55:01.961017 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:01.960987    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e6c79d-kh295"]
Oct 31 12:55:01.963551 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:55:01.963517    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode849745c_8575_4bf1_81d9_f032ddd5e9fe.slice/crio-14d8ce116b8edc01e6ba583b1396187c02143d9444e93700a866b057560a9ccc WatchSource:0}: Error finding container 14d8ce116b8edc01e6ba583b1396187c02143d9444e93700a866b057560a9ccc: Status 404 returned error can't find the container with id 14d8ce116b8edc01e6ba583b1396187c02143d9444e93700a866b057560a9ccc
Oct 31 12:55:02.903176 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:02.903139    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e6c79d-kh295" event={"ID":"e849745c-8575-4bf1-81d9-f032ddd5e9fe","Type":"ContainerStarted","Data":"24c14e673ccf5d91faeafd6d8df3d67096da5b59c3817940bf602dfa94c7af6f"}
Oct 31 12:55:02.903556 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:02.903535    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e6c79d-kh295" event={"ID":"e849745c-8575-4bf1-81d9-f032ddd5e9fe","Type":"ContainerStarted","Data":"14d8ce116b8edc01e6ba583b1396187c02143d9444e93700a866b057560a9ccc"}
Oct 31 12:55:02.934152 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:02.934097    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e6c79d-kh295" podStartSLOduration=1.934080539 podStartE2EDuration="1.934080539s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:55:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:55:02.927194014 +0000 UTC m=+1666.030261728" watchObservedRunningTime="2026-10-31 12:55:02.934080539 +0000 UTC m=+1666.037148252"
Oct 31 12:55:03.666180 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:03.666136    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9a5be90-mtmc2"]
Oct 31 12:55:03.666925 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:03.666904    5060 cpu_manager.go:414] "containerMap: removing container" podUID="c86d1b8b-6bc6-45ff-9412-179f069cf080" containerName="istio-proxy"
Oct 31 12:55:03.666925 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:03.666922    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c86d1b8b-6bc6-45ff-9412-179f069cf080" containerName="istio-proxy"
Oct 31 12:55:03.669593 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:03.669569    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9a5be90-mtmc2"
Oct 31 12:55:03.686342 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:03.686308    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9a5be90-mtmc2"]
Oct 31 12:55:03.794922 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:03.794873    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jspg2\" (UniqueName: \"kubernetes.io/projected/6f4d9298-c434-4443-8014-a1175d8ddb08-kube-api-access-jspg2\") pod \"maas-api-revoke-sub-e9a5be90-mtmc2\" (UID: \"6f4d9298-c434-4443-8014-a1175d8ddb08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9a5be90-mtmc2"
Oct 31 12:55:03.795107 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:03.794932    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f4d9298-c434-4443-8014-a1175d8ddb08-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e9a5be90-mtmc2\" (UID: \"6f4d9298-c434-4443-8014-a1175d8ddb08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9a5be90-mtmc2"
Oct 31 12:55:03.795209 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:03.795184    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f4d9298-c434-4443-8014-a1175d8ddb08-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e9a5be90-mtmc2\" (UID: \"6f4d9298-c434-4443-8014-a1175d8ddb08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9a5be90-mtmc2"
Oct 31 12:55:03.897200 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:03.897161    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f4d9298-c434-4443-8014-a1175d8ddb08-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e9a5be90-mtmc2\" (UID: \"6f4d9298-c434-4443-8014-a1175d8ddb08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9a5be90-mtmc2"
Oct 31 12:55:03.897410 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:03.897288    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jspg2\" (UniqueName: \"kubernetes.io/projected/6f4d9298-c434-4443-8014-a1175d8ddb08-kube-api-access-jspg2\") pod \"maas-api-revoke-sub-e9a5be90-mtmc2\" (UID: \"6f4d9298-c434-4443-8014-a1175d8ddb08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9a5be90-mtmc2"
Oct 31 12:55:03.897410 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:03.897310    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f4d9298-c434-4443-8014-a1175d8ddb08-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e9a5be90-mtmc2\" (UID: \"6f4d9298-c434-4443-8014-a1175d8ddb08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9a5be90-mtmc2"
Oct 31 12:55:03.897884 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:03.897845    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f4d9298-c434-4443-8014-a1175d8ddb08-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e9a5be90-mtmc2\" (UID: \"6f4d9298-c434-4443-8014-a1175d8ddb08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9a5be90-mtmc2"
Oct 31 12:55:03.899822 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:03.899794    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f4d9298-c434-4443-8014-a1175d8ddb08-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e9a5be90-mtmc2\" (UID: \"6f4d9298-c434-4443-8014-a1175d8ddb08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9a5be90-mtmc2"
Oct 31 12:55:03.910336 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:03.910303    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jspg2\" (UniqueName: \"kubernetes.io/projected/6f4d9298-c434-4443-8014-a1175d8ddb08-kube-api-access-jspg2\") pod \"maas-api-revoke-sub-e9a5be90-mtmc2\" (UID: \"6f4d9298-c434-4443-8014-a1175d8ddb08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9a5be90-mtmc2"
Oct 31 12:55:03.918037 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:03.917964    5060 generic.go:356] "Generic (PLEG): container finished" podID="e849745c-8575-4bf1-81d9-f032ddd5e9fe" containerID="24c14e673ccf5d91faeafd6d8df3d67096da5b59c3817940bf602dfa94c7af6f" exitCode=0
Oct 31 12:55:03.918175 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:03.918044    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e6c79d-kh295" event={"ID":"e849745c-8575-4bf1-81d9-f032ddd5e9fe","Type":"ContainerDied","Data":"24c14e673ccf5d91faeafd6d8df3d67096da5b59c3817940bf602dfa94c7af6f"}
Oct 31 12:55:03.989483 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:03.989441    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9a5be90-mtmc2"
Oct 31 12:55:04.309653 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:55:04.309610    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f4d9298_c434_4443_8014_a1175d8ddb08.slice/crio-697c8fb5e7369f5e8df7a5b64fc338638a97bb16a05493a7ac3624774df7b7f1 WatchSource:0}: Error finding container 697c8fb5e7369f5e8df7a5b64fc338638a97bb16a05493a7ac3624774df7b7f1: Status 404 returned error can't find the container with id 697c8fb5e7369f5e8df7a5b64fc338638a97bb16a05493a7ac3624774df7b7f1
Oct 31 12:55:04.310714 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:04.310683    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9a5be90-mtmc2"]
Oct 31 12:55:04.931582 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:04.931546    5060 generic.go:356] "Generic (PLEG): container finished" podID="32a2e0ec-5eaa-4cce-a7c0-87512fe277ed" containerID="7d49e71e687504e7a0bd13babd59f3c3abded0d6427501c69d03fa9f1cdad5e2" exitCode=0
Oct 31 12:55:04.931937 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:04.931831    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2c393f-openshift-default-6cb59888bf-xrph2" event={"ID":"32a2e0ec-5eaa-4cce-a7c0-87512fe277ed","Type":"ContainerDied","Data":"7d49e71e687504e7a0bd13babd59f3c3abded0d6427501c69d03fa9f1cdad5e2"}
Oct 31 12:55:04.932345 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:04.932326    5060 scope.go:122] "RemoveContainer" containerID="7d49e71e687504e7a0bd13babd59f3c3abded0d6427501c69d03fa9f1cdad5e2"
Oct 31 12:55:04.934587 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:04.934482    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:55:04.934725 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:04.934621    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:55:04.934725 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:04.934684    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:55:04.938145 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:04.938117    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9a5be90-mtmc2" event={"ID":"6f4d9298-c434-4443-8014-a1175d8ddb08","Type":"ContainerStarted","Data":"76055c9723c7a18e8a7e5da4e772dcfdb52e9d700b7affd448a76e0913e2c078"}
Oct 31 12:55:04.938289 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:04.938169    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9a5be90-mtmc2" event={"ID":"6f4d9298-c434-4443-8014-a1175d8ddb08","Type":"ContainerStarted","Data":"697c8fb5e7369f5e8df7a5b64fc338638a97bb16a05493a7ac3624774df7b7f1"}
Oct 31 12:55:04.987903 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:04.987842    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9a5be90-mtmc2" podStartSLOduration=1.987823325 podStartE2EDuration="1.987823325s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:55:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:55:04.986262796 +0000 UTC m=+1668.089330509" watchObservedRunningTime="2026-10-31 12:55:04.987823325 +0000 UTC m=+1668.090891039"
Oct 31 12:55:05.268362 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:05.268010    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e6c79d-kh295"
Oct 31 12:55:05.325704 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:05.325677    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e849745c-8575-4bf1-81d9-f032ddd5e9fe-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e849745c-8575-4bf1-81d9-f032ddd5e9fe-maas-api-xks-ca\") pod \"e849745c-8575-4bf1-81d9-f032ddd5e9fe\" (UID: \"e849745c-8575-4bf1-81d9-f032ddd5e9fe\") "
Oct 31 12:55:05.327167 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:05.327138    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e849745c-8575-4bf1-81d9-f032ddd5e9fe-kube-api-access-xw84b\" (UniqueName: \"kubernetes.io/projected/e849745c-8575-4bf1-81d9-f032ddd5e9fe-kube-api-access-xw84b\") pod \"e849745c-8575-4bf1-81d9-f032ddd5e9fe\" (UID: \"e849745c-8575-4bf1-81d9-f032ddd5e9fe\") "
Oct 31 12:55:05.328328 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:05.328271    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e849745c-8575-4bf1-81d9-f032ddd5e9fe-maas-api-openshift-ca" pod "e849745c-8575-4bf1-81d9-f032ddd5e9fe" (UID: "e849745c-8575-4bf1-81d9-f032ddd5e9fe"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:55:05.328606 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:05.328470    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e849745c-8575-4bf1-81d9-f032ddd5e9fe-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e849745c-8575-4bf1-81d9-f032ddd5e9fe-maas-api-openshift-ca\") pod \"e849745c-8575-4bf1-81d9-f032ddd5e9fe\" (UID: \"e849745c-8575-4bf1-81d9-f032ddd5e9fe\") "
Oct 31 12:55:05.328881 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:05.328643    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e849745c-8575-4bf1-81d9-f032ddd5e9fe-maas-api-xks-ca" pod "e849745c-8575-4bf1-81d9-f032ddd5e9fe" (UID: "e849745c-8575-4bf1-81d9-f032ddd5e9fe"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:55:05.332387 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:05.332356    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e849745c-8575-4bf1-81d9-f032ddd5e9fe-kube-api-access-xw84b" pod "e849745c-8575-4bf1-81d9-f032ddd5e9fe" (UID: "e849745c-8575-4bf1-81d9-f032ddd5e9fe"). InnerVolumeSpecName "kube-api-access-xw84b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:55:05.333560 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:05.333536    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e849745c-8575-4bf1-81d9-f032ddd5e9fe-maas-api-xks-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:55:05.334613 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:05.334593    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xw84b\" (UniqueName: \"kubernetes.io/projected/e849745c-8575-4bf1-81d9-f032ddd5e9fe-kube-api-access-xw84b\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:55:05.334738 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:05.334725    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e849745c-8575-4bf1-81d9-f032ddd5e9fe-maas-api-openshift-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:55:05.448592 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:05.448379    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-28t2l"
Oct 31 12:55:05.537373 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:05.537054    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-2fn6t"
Oct 31 12:55:05.542104 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:05.540556    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66295870-6cfc-4f48-b353-566fe73a6bf0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66295870-6cfc-4f48-b353-566fe73a6bf0-istiod-ca-cert\") pod \"66295870-6cfc-4f48-b353-566fe73a6bf0\" (UID: \"66295870-6cfc-4f48-b353-566fe73a6bf0\") "
Oct 31 12:55:05.542104 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:05.540696    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66295870-6cfc-4f48-b353-566fe73a6bf0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66295870-6cfc-4f48-b353-566fe73a6bf0-workload-socket\") pod \"66295870-6cfc-4f48-b353-566fe73a6bf0\" (UID: \"66295870-6cfc-4f48-b353-566fe73a6bf0\") "
Oct 31 12:55:05.542104 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:05.540800    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66295870-6cfc-4f48-b353-566fe73a6bf0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66295870-6cfc-4f48-b353-566fe73a6bf0-istio-envoy\") pod \"66295870-6cfc-4f48-b353-566fe73a6bf0\" (UID: \"66295870-6cfc-4f48-b353-566fe73a6bf0\") "
Oct 31 12:55:05.542104 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:05.540866    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66295870-6cfc-4f48-b353-566fe73a6bf0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66295870-6cfc-4f48-b353-566fe73a6bf0-credential-socket\") pod \"66295870-6cfc-4f48-b353-566fe73a6bf0\" (UID: \"66295870-6cfc-4f48-b353-566fe73a6bf0\") "
Oct 31 12:55:05.542104 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:05.540901    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66295870-6cfc-4f48-b353-566fe73a6bf0-istio-token\" (UniqueName: \"kubernetes.io/projected/66295870-6cfc-4f48-b353-566fe73a6bf0-istio-token\") pod \"66295870-6cfc-4f48-b353-566fe73a6bf0\" (UID: \"66295870-6cfc-4f48-b353-566fe73a6bf0\") "
Oct 31 12:55:05.542104 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:05.540935    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66295870-6cfc-4f48-b353-566fe73a6bf0-kube-api-access-fgjp7\" (UniqueName: \"kubernetes.io/projected/66295870-6cfc-4f48-b353-566fe73a6bf0-kube-api-access-fgjp7\") pod \"66295870-6cfc-4f48-b353-566fe73a6bf0\" (UID: \"66295870-6cfc-4f48-b353-566fe73a6bf0\") "
Oct 31 12:55:05.542104 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:05.541004    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66295870-6cfc-4f48-b353-566fe73a6bf0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66295870-6cfc-4f48-b353-566fe73a6bf0-istio-data\") pod \"66295870-6cfc-4f48-b353-566fe73a6bf0\" (UID: \"66295870-6cfc-4f48-b353-566fe73a6bf0\") "
Oct 31 12:55:05.542104 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:05.541147    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66295870-6cfc-4f48-b353-566fe73a6bf0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66295870-6cfc-4f48-b353-566fe73a6bf0-workload-certs\") pod \"66295870-6cfc-4f48-b353-566fe73a6bf0\" (UID: \"66295870-6cfc-4f48-b353-566fe73a6bf0\") "
Oct 31 12:55:05.542104 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:05.541188    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/66295870-6cfc-4f48-b353-566fe73a6bf0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66295870-6cfc-4f48-b353-566fe73a6bf0-istio-podinfo\") pod \"66295870-6cfc-4f48-b353-566fe73a6bf0\" (UID: \"66295870-6cfc-4f48-b353-566fe73a6bf0\") "
Oct 31 12:55:05.542784 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:05.542260    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66295870-6cfc-4f48-b353-566fe73a6bf0-workload-socket" pod "66295870-6cfc-4f48-b353-566fe73a6bf0" (UID: "66295870-6cfc-4f48-b353-566fe73a6bf0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:55:05.542784 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:05.542548    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66295870-6cfc-4f48-b353-566fe73a6bf0-istiod-ca-cert" pod "66295870-6cfc-4f48-b353-566fe73a6bf0" (UID: "66295870-6cfc-4f48-b353-566fe73a6bf0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:55:05.543316 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:05.543282    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66295870-6cfc-4f48-b353-566fe73a6bf0-workload-certs" pod "66295870-6cfc-4f48-b353-566fe73a6bf0" (UID: "66295870-6cfc-4f48-b353-566fe73a6bf0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:55:05.543757 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:05.543729    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66295870-6cfc-4f48-b353-566fe73a6bf0-istio-data" pod "66295870-6cfc-4f48-b353-566fe73a6bf0" (UID: "66295870-6cfc-4f48-b353-566fe73a6bf0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:55:05.545123 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:05.545103    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66295870-6cfc-4f48-b353-566fe73a6bf0-credential-socket" pod "66295870-6cfc-4f48-b353-566fe73a6bf0" (UID: "66295870-6cfc-4f48-b353-566fe73a6bf0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:55:05.545344 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:05.545305    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66295870-6cfc-4f48-b353-566fe73a6bf0-istio-envoy" pod "66295870-6cfc-4f48-b353-566fe73a6bf0" (UID: "66295870-6cfc-4f48-b353-566fe73a6bf0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:55:05.547321 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:05.547293    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66295870-6cfc-4f48-b353-566fe73a6bf0-kube-api-access-fgjp7" pod "66295870-6cfc-4f48-b353-566fe73a6bf0" (UID: "66295870-6cfc-4f48-b353-566fe73a6bf0"). InnerVolumeSpecName "kube-api-access-fgjp7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:55:05.547784 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:05.547758    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66295870-6cfc-4f48-b353-566fe73a6bf0-istio-token" pod "66295870-6cfc-4f48-b353-566fe73a6bf0" (UID: "66295870-6cfc-4f48-b353-566fe73a6bf0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:55:05.548204 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:05.548182    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/66295870-6cfc-4f48-b353-566fe73a6bf0-istio-podinfo" pod "66295870-6cfc-4f48-b353-566fe73a6bf0" (UID: "66295870-6cfc-4f48-b353-566fe73a6bf0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 12:55:05.642534 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:05.642472    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60d395e9-e4e9-4c68-ab10-9ef5371df1b1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60d395e9-e4e9-4c68-ab10-9ef5371df1b1-istiod-ca-cert\") pod \"60d395e9-e4e9-4c68-ab10-9ef5371df1b1\" (UID: \"60d395e9-e4e9-4c68-ab10-9ef5371df1b1\") "
Oct 31 12:55:05.642748 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:05.642563    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60d395e9-e4e9-4c68-ab10-9ef5371df1b1-istio-token\" (UniqueName: \"kubernetes.io/projected/60d395e9-e4e9-4c68-ab10-9ef5371df1b1-istio-token\") pod \"60d395e9-e4e9-4c68-ab10-9ef5371df1b1\" (UID: \"60d395e9-e4e9-4c68-ab10-9ef5371df1b1\") "
Oct 31 12:55:05.642748 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:05.642619    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60d395e9-e4e9-4c68-ab10-9ef5371df1b1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60d395e9-e4e9-4c68-ab10-9ef5371df1b1-credential-socket\") pod \"60d395e9-e4e9-4c68-ab10-9ef5371df1b1\" (UID: \"60d395e9-e4e9-4c68-ab10-9ef5371df1b1\") "
Oct 31 12:55:05.642748 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:05.642703    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60d395e9-e4e9-4c68-ab10-9ef5371df1b1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60d395e9-e4e9-4c68-ab10-9ef5371df1b1-workload-certs\") pod \"60d395e9-e4e9-4c68-ab10-9ef5371df1b1\" (UID: \"60d395e9-e4e9-4c68-ab10-9ef5371df1b1\") "
Oct 31 12:55:05.642941 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:05.642755    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60d395e9-e4e9-4c68-ab10-9ef5371df1b1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60d395e9-e4e9-4c68-ab10-9ef5371df1b1-istio-envoy\") pod \"60d395e9-e4e9-4c68-ab10-9ef5371df1b1\" (UID: \"60d395e9-e4e9-4c68-ab10-9ef5371df1b1\") "
Oct 31 12:55:05.642941 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:05.642794    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60d395e9-e4e9-4c68-ab10-9ef5371df1b1-kube-api-access-4bzqq\" (UniqueName: \"kubernetes.io/projected/60d395e9-e4e9-4c68-ab10-9ef5371df1b1-kube-api-access-4bzqq\") pod \"60d395e9-e4e9-4c68-ab10-9ef5371df1b1\" (UID: \"60d395e9-e4e9-4c68-ab10-9ef5371df1b1\") "
Oct 31 12:55:05.642941 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:05.642855    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60d395e9-e4e9-4c68-ab10-9ef5371df1b1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60d395e9-e4e9-4c68-ab10-9ef5371df1b1-workload-socket\") pod \"60d395e9-e4e9-4c68-ab10-9ef5371df1b1\" (UID: \"60d395e9-e4e9-4c68-ab10-9ef5371df1b1\") "
Oct 31 12:55:05.642941 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:05.642907    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60d395e9-e4e9-4c68-ab10-9ef5371df1b1-istiod-ca-cert" pod "60d395e9-e4e9-4c68-ab10-9ef5371df1b1" (UID: "60d395e9-e4e9-4c68-ab10-9ef5371df1b1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:55:05.643167 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:05.642992    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/60d395e9-e4e9-4c68-ab10-9ef5371df1b1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60d395e9-e4e9-4c68-ab10-9ef5371df1b1-istio-podinfo\") pod \"60d395e9-e4e9-4c68-ab10-9ef5371df1b1\" (UID: \"60d395e9-e4e9-4c68-ab10-9ef5371df1b1\") "
Oct 31 12:55:05.643167 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:05.643076    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60d395e9-e4e9-4c68-ab10-9ef5371df1b1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60d395e9-e4e9-4c68-ab10-9ef5371df1b1-istio-data\") pod \"60d395e9-e4e9-4c68-ab10-9ef5371df1b1\" (UID: \"60d395e9-e4e9-4c68-ab10-9ef5371df1b1\") "
Oct 31 12:55:05.643286 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:05.643191    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60d395e9-e4e9-4c68-ab10-9ef5371df1b1-workload-socket" pod "60d395e9-e4e9-4c68-ab10-9ef5371df1b1" (UID: "60d395e9-e4e9-4c68-ab10-9ef5371df1b1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:55:05.644132 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:05.643793    5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66295870-6cfc-4f48-b353-566fe73a6bf0-workload-certs\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:55:05.644132 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:05.643821    5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66295870-6cfc-4f48-b353-566fe73a6bf0-istio-podinfo\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:55:05.644132 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:05.643838    5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66295870-6cfc-4f48-b353-566fe73a6bf0-istiod-ca-cert\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:55:05.644132 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:05.643853    5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60d395e9-e4e9-4c68-ab10-9ef5371df1b1-istiod-ca-cert\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:55:05.644132 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:05.643867    5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66295870-6cfc-4f48-b353-566fe73a6bf0-workload-socket\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:55:05.644132 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:05.643881    5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66295870-6cfc-4f48-b353-566fe73a6bf0-istio-envoy\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:55:05.644132 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:05.643894    5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66295870-6cfc-4f48-b353-566fe73a6bf0-credential-socket\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:55:05.644132 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:05.643908    5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66295870-6cfc-4f48-b353-566fe73a6bf0-istio-token\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:55:05.644132 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:05.643922    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fgjp7\" (UniqueName: \"kubernetes.io/projected/66295870-6cfc-4f48-b353-566fe73a6bf0-kube-api-access-fgjp7\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:55:05.644132 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:05.643937    5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60d395e9-e4e9-4c68-ab10-9ef5371df1b1-workload-socket\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:55:05.644132 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:05.643950    5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66295870-6cfc-4f48-b353-566fe73a6bf0-istio-data\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:55:05.644868 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:05.644377    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60d395e9-e4e9-4c68-ab10-9ef5371df1b1-credential-socket" pod "60d395e9-e4e9-4c68-ab10-9ef5371df1b1" (UID: "60d395e9-e4e9-4c68-ab10-9ef5371df1b1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:55:05.644868 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:05.644613    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60d395e9-e4e9-4c68-ab10-9ef5371df1b1-workload-certs" pod "60d395e9-e4e9-4c68-ab10-9ef5371df1b1" (UID: "60d395e9-e4e9-4c68-ab10-9ef5371df1b1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:55:05.645076 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:05.645041    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60d395e9-e4e9-4c68-ab10-9ef5371df1b1-istio-data" pod "60d395e9-e4e9-4c68-ab10-9ef5371df1b1" (UID: "60d395e9-e4e9-4c68-ab10-9ef5371df1b1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:55:05.646190 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:05.646162    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60d395e9-e4e9-4c68-ab10-9ef5371df1b1-istio-token" pod "60d395e9-e4e9-4c68-ab10-9ef5371df1b1" (UID: "60d395e9-e4e9-4c68-ab10-9ef5371df1b1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:55:05.647337 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:05.647296    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60d395e9-e4e9-4c68-ab10-9ef5371df1b1-kube-api-access-4bzqq" pod "60d395e9-e4e9-4c68-ab10-9ef5371df1b1" (UID: "60d395e9-e4e9-4c68-ab10-9ef5371df1b1"). InnerVolumeSpecName "kube-api-access-4bzqq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:55:05.647448 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:05.647361    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60d395e9-e4e9-4c68-ab10-9ef5371df1b1-istio-envoy" pod "60d395e9-e4e9-4c68-ab10-9ef5371df1b1" (UID: "60d395e9-e4e9-4c68-ab10-9ef5371df1b1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:55:05.647756 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:05.647722    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/60d395e9-e4e9-4c68-ab10-9ef5371df1b1-istio-podinfo" pod "60d395e9-e4e9-4c68-ab10-9ef5371df1b1" (UID: "60d395e9-e4e9-4c68-ab10-9ef5371df1b1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 12:55:05.744976 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:05.744943    5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60d395e9-e4e9-4c68-ab10-9ef5371df1b1-istio-podinfo\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:55:05.744976 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:05.744973    5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60d395e9-e4e9-4c68-ab10-9ef5371df1b1-istio-data\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:55:05.744976 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:05.744984    5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60d395e9-e4e9-4c68-ab10-9ef5371df1b1-istio-token\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:55:05.745216 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:05.744993    5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60d395e9-e4e9-4c68-ab10-9ef5371df1b1-credential-socket\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:55:05.745216 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:05.745003    5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60d395e9-e4e9-4c68-ab10-9ef5371df1b1-workload-certs\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:55:05.745216 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:05.745011    5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60d395e9-e4e9-4c68-ab10-9ef5371df1b1-istio-envoy\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:55:05.745216 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:05.745025    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4bzqq\" (UniqueName: \"kubernetes.io/projected/60d395e9-e4e9-4c68-ab10-9ef5371df1b1-kube-api-access-4bzqq\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:55:05.950727 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:05.950631    5060 generic.go:356] "Generic (PLEG): container finished" podID="66295870-6cfc-4f48-b353-566fe73a6bf0" containerID="4b22833c781d46e9c40b427b1113815ec8395e35c2c4ba287f4237abcc240d0b" exitCode=0
Oct 31 12:55:05.950727 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:05.950679    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-28t2l"
Oct 31 12:55:05.951271 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:05.950722    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-28t2l" event={"ID":"66295870-6cfc-4f48-b353-566fe73a6bf0","Type":"ContainerDied","Data":"4b22833c781d46e9c40b427b1113815ec8395e35c2c4ba287f4237abcc240d0b"}
Oct 31 12:55:05.951271 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:05.950803    5060 scope.go:122] "RemoveContainer" containerID="4b22833c781d46e9c40b427b1113815ec8395e35c2c4ba287f4237abcc240d0b"
Oct 31 12:55:05.951523 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:05.950788    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-28t2l" event={"ID":"66295870-6cfc-4f48-b353-566fe73a6bf0","Type":"ContainerDied","Data":"11d442b5712724b7964af9edfab30d6c919ee982ae95393440c8c3ac0df988e0"}
Oct 31 12:55:05.984014 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:05.980892    5060 generic.go:356] "Generic (PLEG): container finished" podID="6f4d9298-c434-4443-8014-a1175d8ddb08" containerID="76055c9723c7a18e8a7e5da4e772dcfdb52e9d700b7affd448a76e0913e2c078" exitCode=0
Oct 31 12:55:05.984014 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:05.981207    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9a5be90-mtmc2" event={"ID":"6f4d9298-c434-4443-8014-a1175d8ddb08","Type":"ContainerDied","Data":"76055c9723c7a18e8a7e5da4e772dcfdb52e9d700b7affd448a76e0913e2c078"}
Oct 31 12:55:05.990830 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:05.990794    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2c393f-openshift-default-6cb59888bf-xrph2" event={"ID":"32a2e0ec-5eaa-4cce-a7c0-87512fe277ed","Type":"ContainerStarted","Data":"a91d050ca52a5d0fa0283a6c2e6d723c813e093ed0c7f6f6d077a58c1334ab52"}
Oct 31 12:55:05.994621 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:05.994582    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e6c79d-kh295" event={"ID":"e849745c-8575-4bf1-81d9-f032ddd5e9fe","Type":"ContainerDied","Data":"14d8ce116b8edc01e6ba583b1396187c02143d9444e93700a866b057560a9ccc"}
Oct 31 12:55:05.994821 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:05.994645    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="14d8ce116b8edc01e6ba583b1396187c02143d9444e93700a866b057560a9ccc"
Oct 31 12:55:05.994821 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:05.994754    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e6c79d-kh295"
Oct 31 12:55:05.997691 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:05.997655    5060 generic.go:356] "Generic (PLEG): container finished" podID="60d395e9-e4e9-4c68-ab10-9ef5371df1b1" containerID="6576cc6e6a88037c75e3d89b906c3e21c090af7d8a2f8e801ef2a27753a51791" exitCode=0
Oct 31 12:55:05.997934 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:05.997914    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-2fn6t" event={"ID":"60d395e9-e4e9-4c68-ab10-9ef5371df1b1","Type":"ContainerDied","Data":"6576cc6e6a88037c75e3d89b906c3e21c090af7d8a2f8e801ef2a27753a51791"}
Oct 31 12:55:05.998005 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:05.997968    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-2fn6t" event={"ID":"60d395e9-e4e9-4c68-ab10-9ef5371df1b1","Type":"ContainerDied","Data":"ac559542ac0e20afb2af7ca9f91ee86a5d4ecab6810e5f14d5526e6f6ba0ab59"}
Oct 31 12:55:05.998095 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:05.998080    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-2fn6t"
Oct 31 12:55:06.032136 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:06.032108    5060 scope.go:122] "RemoveContainer" containerID="4b22833c781d46e9c40b427b1113815ec8395e35c2c4ba287f4237abcc240d0b"
Oct 31 12:55:06.034739 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:55:06.034561    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b22833c781d46e9c40b427b1113815ec8395e35c2c4ba287f4237abcc240d0b\": container with ID starting with 4b22833c781d46e9c40b427b1113815ec8395e35c2c4ba287f4237abcc240d0b not found: ID does not exist" containerID="4b22833c781d46e9c40b427b1113815ec8395e35c2c4ba287f4237abcc240d0b"
Oct 31 12:55:06.034739 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:06.034604    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b22833c781d46e9c40b427b1113815ec8395e35c2c4ba287f4237abcc240d0b"} err="failed to get container status \"4b22833c781d46e9c40b427b1113815ec8395e35c2c4ba287f4237abcc240d0b\": rpc error: code = NotFound desc = could not find container \"4b22833c781d46e9c40b427b1113815ec8395e35c2c4ba287f4237abcc240d0b\": container with ID starting with 4b22833c781d46e9c40b427b1113815ec8395e35c2c4ba287f4237abcc240d0b not found: ID does not exist"
Oct 31 12:55:06.034739 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:06.034627    5060 scope.go:122] "RemoveContainer" containerID="6576cc6e6a88037c75e3d89b906c3e21c090af7d8a2f8e801ef2a27753a51791"
Oct 31 12:55:06.099294 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:06.099237    5060 scope.go:122] "RemoveContainer" containerID="6576cc6e6a88037c75e3d89b906c3e21c090af7d8a2f8e801ef2a27753a51791"
Oct 31 12:55:06.099906 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:55:06.099816    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6576cc6e6a88037c75e3d89b906c3e21c090af7d8a2f8e801ef2a27753a51791\": container with ID starting with 6576cc6e6a88037c75e3d89b906c3e21c090af7d8a2f8e801ef2a27753a51791 not found: ID does not exist" containerID="6576cc6e6a88037c75e3d89b906c3e21c090af7d8a2f8e801ef2a27753a51791"
Oct 31 12:55:06.099906 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:06.099867    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6576cc6e6a88037c75e3d89b906c3e21c090af7d8a2f8e801ef2a27753a51791"} err="failed to get container status \"6576cc6e6a88037c75e3d89b906c3e21c090af7d8a2f8e801ef2a27753a51791\": rpc error: code = NotFound desc = could not find container \"6576cc6e6a88037c75e3d89b906c3e21c090af7d8a2f8e801ef2a27753a51791\": container with ID starting with 6576cc6e6a88037c75e3d89b906c3e21c090af7d8a2f8e801ef2a27753a51791 not found: ID does not exist"
Oct 31 12:55:06.106079 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:06.106045    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-2fn6t"]
Oct 31 12:55:06.115806 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:06.114244    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-e6c79d/premium-w2-e6c79d-kserve-6468bf46d4-h66f8"]
Oct 31 12:55:06.120540 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:06.120514    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-2fn6t"]
Oct 31 12:55:06.126346 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:06.126313    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-28t2l"]
Oct 31 12:55:06.130649 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:06.130624    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-28t2l"]
Oct 31 12:55:06.813535 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:06.813481    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-2c393f-openshift-default-6cb59888bf-xrph2"
Oct 31 12:55:06.814642 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:06.814611    5060 patch_prober.go:29] interesting pod/e2e-inf-b-2c393f-openshift-default-6cb59888bf-xrph2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" start-of-body=
Oct 31 12:55:06.814803 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:06.814666    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2c393f-openshift-default-6cb59888bf-xrph2" podUID="32a2e0ec-5eaa-4cce-a7c0-87512fe277ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused"
Oct 31 12:55:07.051613 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:07.051581    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60d395e9-e4e9-4c68-ab10-9ef5371df1b1" path="/var/lib/kubelet/pods/60d395e9-e4e9-4c68-ab10-9ef5371df1b1/volumes"
Oct 31 12:55:07.052307 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:07.052287    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66295870-6cfc-4f48-b353-566fe73a6bf0" path="/var/lib/kubelet/pods/66295870-6cfc-4f48-b353-566fe73a6bf0/volumes"
Oct 31 12:55:07.303263 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:07.303234    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9a5be90-mtmc2"
Oct 31 12:55:07.403528 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:07.403432    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f4d9298-c434-4443-8014-a1175d8ddb08-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f4d9298-c434-4443-8014-a1175d8ddb08-maas-api-openshift-ca\") pod \"6f4d9298-c434-4443-8014-a1175d8ddb08\" (UID: \"6f4d9298-c434-4443-8014-a1175d8ddb08\") "
Oct 31 12:55:07.403797 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:07.403781    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6f4d9298-c434-4443-8014-a1175d8ddb08-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f4d9298-c434-4443-8014-a1175d8ddb08-maas-api-xks-ca\") pod \"6f4d9298-c434-4443-8014-a1175d8ddb08\" (UID: \"6f4d9298-c434-4443-8014-a1175d8ddb08\") "
Oct 31 12:55:07.403918 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:07.403907    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f4d9298-c434-4443-8014-a1175d8ddb08-kube-api-access-jspg2\" (UniqueName: \"kubernetes.io/projected/6f4d9298-c434-4443-8014-a1175d8ddb08-kube-api-access-jspg2\") pod \"6f4d9298-c434-4443-8014-a1175d8ddb08\" (UID: \"6f4d9298-c434-4443-8014-a1175d8ddb08\") "
Oct 31 12:55:07.404747 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:07.404718    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f4d9298-c434-4443-8014-a1175d8ddb08-maas-api-openshift-ca" pod "6f4d9298-c434-4443-8014-a1175d8ddb08" (UID: "6f4d9298-c434-4443-8014-a1175d8ddb08"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:55:07.407108 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:07.407084    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f4d9298-c434-4443-8014-a1175d8ddb08-kube-api-access-jspg2" pod "6f4d9298-c434-4443-8014-a1175d8ddb08" (UID: "6f4d9298-c434-4443-8014-a1175d8ddb08"). InnerVolumeSpecName "kube-api-access-jspg2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:55:07.407245 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:07.407225    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f4d9298-c434-4443-8014-a1175d8ddb08-maas-api-xks-ca" pod "6f4d9298-c434-4443-8014-a1175d8ddb08" (UID: "6f4d9298-c434-4443-8014-a1175d8ddb08"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:55:07.505829 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:07.505782    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f4d9298-c434-4443-8014-a1175d8ddb08-maas-api-xks-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:55:07.505829 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:07.505830    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jspg2\" (UniqueName: \"kubernetes.io/projected/6f4d9298-c434-4443-8014-a1175d8ddb08-kube-api-access-jspg2\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:55:07.505829 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:07.505851    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f4d9298-c434-4443-8014-a1175d8ddb08-maas-api-openshift-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:55:07.816955 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:07.816898    5060 patch_prober.go:29] interesting pod/e2e-inf-b-2c393f-openshift-default-6cb59888bf-xrph2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" start-of-body=
Oct 31 12:55:07.817293 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:07.817241    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2c393f-openshift-default-6cb59888bf-xrph2" podUID="32a2e0ec-5eaa-4cce-a7c0-87512fe277ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused"
Oct 31 12:55:08.049687 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:08.049651    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9a5be90-mtmc2"
Oct 31 12:55:08.049687 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:08.049677    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9a5be90-mtmc2" event={"ID":"6f4d9298-c434-4443-8014-a1175d8ddb08","Type":"ContainerDied","Data":"697c8fb5e7369f5e8df7a5b64fc338638a97bb16a05493a7ac3624774df7b7f1"}
Oct 31 12:55:08.049887 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:08.049728    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="697c8fb5e7369f5e8df7a5b64fc338638a97bb16a05493a7ac3624774df7b7f1"
Oct 31 12:55:08.338110 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:08.338033    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-2c393f-openshift-default-6cb59888bf-xrph2"
Oct 31 12:55:08.813046 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:08.813007    5060 patch_prober.go:29] interesting pod/e2e-inf-b-2c393f-openshift-default-6cb59888bf-xrph2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" start-of-body=
Oct 31 12:55:08.813223 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:08.813067    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2c393f-openshift-default-6cb59888bf-xrph2" podUID="32a2e0ec-5eaa-4cce-a7c0-87512fe277ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused"
Oct 31 12:55:09.813165 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:09.813123    5060 patch_prober.go:29] interesting pod/e2e-inf-b-2c393f-openshift-default-6cb59888bf-xrph2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" start-of-body=
Oct 31 12:55:09.813591 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:09.813182    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2c393f-openshift-default-6cb59888bf-xrph2" podUID="32a2e0ec-5eaa-4cce-a7c0-87512fe277ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused"
Oct 31 12:55:10.812983 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:10.812945    5060 patch_prober.go:29] interesting pod/e2e-inf-b-2c393f-openshift-default-6cb59888bf-xrph2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" start-of-body=
Oct 31 12:55:10.813148 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:10.813005    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2c393f-openshift-default-6cb59888bf-xrph2" podUID="32a2e0ec-5eaa-4cce-a7c0-87512fe277ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused"
Oct 31 12:55:11.417432 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:11.417389    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-e6c79d-7f6dbd5c9-psn2f"]
Oct 31 12:55:11.843515 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:11.842788    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-2c393f-openshift-default-6cb59888bf-xrph2"
Oct 31 12:55:11.852512 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:11.846374    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-2c393f-openshift-default-6cb59888bf-xrph2"
Oct 31 12:55:13.347406 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:13.347374    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-55360e0c-vhnxl"]
Oct 31 12:55:13.348084 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:13.348068    5060 cpu_manager.go:414] "containerMap: removing container" podUID="6f4d9298-c434-4443-8014-a1175d8ddb08" containerName="revoke-keys"
Oct 31 12:55:13.348142 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:13.348085    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f4d9298-c434-4443-8014-a1175d8ddb08" containerName="revoke-keys"
Oct 31 12:55:13.348142 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:13.348102    5060 cpu_manager.go:414] "containerMap: removing container" podUID="60d395e9-e4e9-4c68-ab10-9ef5371df1b1" containerName="istio-proxy"
Oct 31 12:55:13.348142 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:13.348109    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60d395e9-e4e9-4c68-ab10-9ef5371df1b1" containerName="istio-proxy"
Oct 31 12:55:13.348242 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:13.348157    5060 cpu_manager.go:414] "containerMap: removing container" podUID="66295870-6cfc-4f48-b353-566fe73a6bf0" containerName="istio-proxy"
Oct 31 12:55:13.348242 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:13.348166    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66295870-6cfc-4f48-b353-566fe73a6bf0" containerName="istio-proxy"
Oct 31 12:55:13.348242 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:13.348201    5060 cpu_manager.go:414] "containerMap: removing container" podUID="e849745c-8575-4bf1-81d9-f032ddd5e9fe" containerName="revoke-keys"
Oct 31 12:55:13.348242 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:13.348206    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e849745c-8575-4bf1-81d9-f032ddd5e9fe" containerName="revoke-keys"
Oct 31 12:55:13.348367 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:13.348307    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="e849745c-8575-4bf1-81d9-f032ddd5e9fe" containerName="revoke-keys"
Oct 31 12:55:13.348367 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:13.348317    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f4d9298-c434-4443-8014-a1175d8ddb08" containerName="revoke-keys"
Oct 31 12:55:13.348367 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:13.348324    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="60d395e9-e4e9-4c68-ab10-9ef5371df1b1" containerName="istio-proxy"
Oct 31 12:55:13.348367 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:13.348335    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="66295870-6cfc-4f48-b353-566fe73a6bf0" containerName="istio-proxy"
Oct 31 12:55:13.352587 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:13.352564    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f231812-xgbw7"]
Oct 31 12:55:13.352724 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:13.352665    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55360e0c-vhnxl"
Oct 31 12:55:13.366124 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:13.365281    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 12:55:13.380727 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:13.380120    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7wsmb\""
Oct 31 12:55:13.385780 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:13.385748    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-55360e0c-vhnxl"]
Oct 31 12:55:13.385922 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:13.385903    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f231812-xgbw7"
Oct 31 12:55:13.400439 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:13.400411    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f231812-xgbw7"]
Oct 31 12:55:13.459183 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:13.459148    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8mgx\" (UniqueName: \"kubernetes.io/projected/00572ed5-fa0b-4297-8087-24d4d632f5fb-kube-api-access-w8mgx\") pod \"maas-api-revoke-sub-55360e0c-vhnxl\" (UID: \"00572ed5-fa0b-4297-8087-24d4d632f5fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55360e0c-vhnxl"
Oct 31 12:55:13.459378 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:13.459275    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00572ed5-fa0b-4297-8087-24d4d632f5fb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-55360e0c-vhnxl\" (UID: \"00572ed5-fa0b-4297-8087-24d4d632f5fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55360e0c-vhnxl"
Oct 31 12:55:13.459378 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:13.459310    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00572ed5-fa0b-4297-8087-24d4d632f5fb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-55360e0c-vhnxl\" (UID: \"00572ed5-fa0b-4297-8087-24d4d632f5fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55360e0c-vhnxl"
Oct 31 12:55:13.560803 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:13.560763    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e616d24-af13-4f8f-93f0-bae103fceca8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6f231812-xgbw7\" (UID: \"3e616d24-af13-4f8f-93f0-bae103fceca8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f231812-xgbw7"
Oct 31 12:55:13.561011 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:13.560893    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e616d24-af13-4f8f-93f0-bae103fceca8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6f231812-xgbw7\" (UID: \"3e616d24-af13-4f8f-93f0-bae103fceca8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f231812-xgbw7"
Oct 31 12:55:13.561067 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:13.561025    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqn7r\" (UniqueName: \"kubernetes.io/projected/3e616d24-af13-4f8f-93f0-bae103fceca8-kube-api-access-qqn7r\") pod \"maas-api-revoke-sub-6f231812-xgbw7\" (UID: \"3e616d24-af13-4f8f-93f0-bae103fceca8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f231812-xgbw7"
Oct 31 12:55:13.561200 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:13.561175    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8mgx\" (UniqueName: \"kubernetes.io/projected/00572ed5-fa0b-4297-8087-24d4d632f5fb-kube-api-access-w8mgx\") pod \"maas-api-revoke-sub-55360e0c-vhnxl\" (UID: \"00572ed5-fa0b-4297-8087-24d4d632f5fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55360e0c-vhnxl"
Oct 31 12:55:13.561283 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:13.561267    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00572ed5-fa0b-4297-8087-24d4d632f5fb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-55360e0c-vhnxl\" (UID: \"00572ed5-fa0b-4297-8087-24d4d632f5fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55360e0c-vhnxl"
Oct 31 12:55:13.561345 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:13.561298    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00572ed5-fa0b-4297-8087-24d4d632f5fb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-55360e0c-vhnxl\" (UID: \"00572ed5-fa0b-4297-8087-24d4d632f5fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55360e0c-vhnxl"
Oct 31 12:55:13.561930 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:13.561908    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00572ed5-fa0b-4297-8087-24d4d632f5fb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-55360e0c-vhnxl\" (UID: \"00572ed5-fa0b-4297-8087-24d4d632f5fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55360e0c-vhnxl"
Oct 31 12:55:13.563887 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:13.563860    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00572ed5-fa0b-4297-8087-24d4d632f5fb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-55360e0c-vhnxl\" (UID: \"00572ed5-fa0b-4297-8087-24d4d632f5fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55360e0c-vhnxl"
Oct 31 12:55:13.574685 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:13.574654    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8mgx\" (UniqueName: \"kubernetes.io/projected/00572ed5-fa0b-4297-8087-24d4d632f5fb-kube-api-access-w8mgx\") pod \"maas-api-revoke-sub-55360e0c-vhnxl\" (UID: \"00572ed5-fa0b-4297-8087-24d4d632f5fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55360e0c-vhnxl"
Oct 31 12:55:13.691553 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:13.689305    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqn7r\" (UniqueName: \"kubernetes.io/projected/3e616d24-af13-4f8f-93f0-bae103fceca8-kube-api-access-qqn7r\") pod \"maas-api-revoke-sub-6f231812-xgbw7\" (UID: \"3e616d24-af13-4f8f-93f0-bae103fceca8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f231812-xgbw7"
Oct 31 12:55:13.691553 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:13.689679    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e616d24-af13-4f8f-93f0-bae103fceca8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6f231812-xgbw7\" (UID: \"3e616d24-af13-4f8f-93f0-bae103fceca8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f231812-xgbw7"
Oct 31 12:55:13.691553 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:13.689803    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e616d24-af13-4f8f-93f0-bae103fceca8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6f231812-xgbw7\" (UID: \"3e616d24-af13-4f8f-93f0-bae103fceca8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f231812-xgbw7"
Oct 31 12:55:13.691553 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:13.690379    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55360e0c-vhnxl"
Oct 31 12:55:13.707539 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:13.701014    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e616d24-af13-4f8f-93f0-bae103fceca8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6f231812-xgbw7\" (UID: \"3e616d24-af13-4f8f-93f0-bae103fceca8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f231812-xgbw7"
Oct 31 12:55:13.726767 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:13.724787    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e616d24-af13-4f8f-93f0-bae103fceca8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6f231812-xgbw7\" (UID: \"3e616d24-af13-4f8f-93f0-bae103fceca8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f231812-xgbw7"
Oct 31 12:55:13.731306 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:13.731032    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqn7r\" (UniqueName: \"kubernetes.io/projected/3e616d24-af13-4f8f-93f0-bae103fceca8-kube-api-access-qqn7r\") pod \"maas-api-revoke-sub-6f231812-xgbw7\" (UID: \"3e616d24-af13-4f8f-93f0-bae103fceca8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f231812-xgbw7"
Oct 31 12:55:14.002895 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:55:14.002856    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00572ed5_fa0b_4297_8087_24d4d632f5fb.slice/crio-0f9128786f9588a292483253010b139b08f2d8c9fc904ca7ebc2bfd84031614f WatchSource:0}: Error finding container 0f9128786f9588a292483253010b139b08f2d8c9fc904ca7ebc2bfd84031614f: Status 404 returned error can't find the container with id 0f9128786f9588a292483253010b139b08f2d8c9fc904ca7ebc2bfd84031614f
Oct 31 12:55:14.003068 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:14.002913    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-55360e0c-vhnxl"]
Oct 31 12:55:14.014695 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:14.014666    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f231812-xgbw7"
Oct 31 12:55:14.155052 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:14.155013    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55360e0c-vhnxl" event={"ID":"00572ed5-fa0b-4297-8087-24d4d632f5fb","Type":"ContainerStarted","Data":"0f9128786f9588a292483253010b139b08f2d8c9fc904ca7ebc2bfd84031614f"}
Oct 31 12:55:14.312378 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:55:14.312348    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e616d24_af13_4f8f_93f0_bae103fceca8.slice/crio-863d235cd93048e1d1cb3717d0b8407a4e94ca7c9f7ae77b1441f21659d34453 WatchSource:0}: Error finding container 863d235cd93048e1d1cb3717d0b8407a4e94ca7c9f7ae77b1441f21659d34453: Status 404 returned error can't find the container with id 863d235cd93048e1d1cb3717d0b8407a4e94ca7c9f7ae77b1441f21659d34453
Oct 31 12:55:14.312566 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:14.312541    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f231812-xgbw7"]
Oct 31 12:55:15.167846 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:15.167804    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f231812-xgbw7" event={"ID":"3e616d24-af13-4f8f-93f0-bae103fceca8","Type":"ContainerStarted","Data":"25a4c056e2d959b3e485921e82bc2a0f70a23fd8e3658d33626268655eaa0ed6"}
Oct 31 12:55:15.168234 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:15.167872    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f231812-xgbw7" event={"ID":"3e616d24-af13-4f8f-93f0-bae103fceca8","Type":"ContainerStarted","Data":"863d235cd93048e1d1cb3717d0b8407a4e94ca7c9f7ae77b1441f21659d34453"}
Oct 31 12:55:15.169747 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:15.169716    5060 generic.go:356] "Generic (PLEG): container finished" podID="00572ed5-fa0b-4297-8087-24d4d632f5fb" containerID="3779cdba2e95eaba88ec5a09f2dcf6f2bfafa3160cf3f673b512cbb32005ef08" exitCode=0
Oct 31 12:55:15.169873 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:15.169793    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55360e0c-vhnxl" event={"ID":"00572ed5-fa0b-4297-8087-24d4d632f5fb","Type":"ContainerDied","Data":"3779cdba2e95eaba88ec5a09f2dcf6f2bfafa3160cf3f673b512cbb32005ef08"}
Oct 31 12:55:15.196224 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:15.196166    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f231812-xgbw7" podStartSLOduration=2.196152063 podStartE2EDuration="2.196152063s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:55:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:55:15.191329167 +0000 UTC m=+1678.294396883" watchObservedRunningTime="2026-10-31 12:55:15.196152063 +0000 UTC m=+1678.299219794"
Oct 31 12:55:16.181825 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:16.181785    5060 generic.go:356] "Generic (PLEG): container finished" podID="3e616d24-af13-4f8f-93f0-bae103fceca8" containerID="25a4c056e2d959b3e485921e82bc2a0f70a23fd8e3658d33626268655eaa0ed6" exitCode=0
Oct 31 12:55:16.182314 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:16.181868    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f231812-xgbw7" event={"ID":"3e616d24-af13-4f8f-93f0-bae103fceca8","Type":"ContainerDied","Data":"25a4c056e2d959b3e485921e82bc2a0f70a23fd8e3658d33626268655eaa0ed6"}
Oct 31 12:55:16.418565 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:16.418531    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-e6c79d-7f6dbd5c9-psn2f" podUID="3f6d17e0-1d51-41b8-a52c-a386a8fdcc17" containerName="payload-processing" containerID="cri-o://3e3c21381670c8a6131d9ad5ce41c83a85369b90f0088941ea202ce7538723a7" gracePeriod=25
Oct 31 12:55:16.448723 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:16.448660    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55360e0c-vhnxl"
Oct 31 12:55:16.543543 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:16.543502    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00572ed5-fa0b-4297-8087-24d4d632f5fb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00572ed5-fa0b-4297-8087-24d4d632f5fb-maas-api-openshift-ca\") pod \"00572ed5-fa0b-4297-8087-24d4d632f5fb\" (UID: \"00572ed5-fa0b-4297-8087-24d4d632f5fb\") "
Oct 31 12:55:16.543716 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:16.543624    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/00572ed5-fa0b-4297-8087-24d4d632f5fb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00572ed5-fa0b-4297-8087-24d4d632f5fb-maas-api-xks-ca\") pod \"00572ed5-fa0b-4297-8087-24d4d632f5fb\" (UID: \"00572ed5-fa0b-4297-8087-24d4d632f5fb\") "
Oct 31 12:55:16.543806 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:16.543773    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00572ed5-fa0b-4297-8087-24d4d632f5fb-kube-api-access-w8mgx\" (UniqueName: \"kubernetes.io/projected/00572ed5-fa0b-4297-8087-24d4d632f5fb-kube-api-access-w8mgx\") pod \"00572ed5-fa0b-4297-8087-24d4d632f5fb\" (UID: \"00572ed5-fa0b-4297-8087-24d4d632f5fb\") "
Oct 31 12:55:16.543872 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:16.543843    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00572ed5-fa0b-4297-8087-24d4d632f5fb-maas-api-openshift-ca" pod "00572ed5-fa0b-4297-8087-24d4d632f5fb" (UID: "00572ed5-fa0b-4297-8087-24d4d632f5fb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:55:16.544438 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:16.544411    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00572ed5-fa0b-4297-8087-24d4d632f5fb-maas-api-openshift-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:55:16.545806 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:16.545772    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00572ed5-fa0b-4297-8087-24d4d632f5fb-maas-api-xks-ca" pod "00572ed5-fa0b-4297-8087-24d4d632f5fb" (UID: "00572ed5-fa0b-4297-8087-24d4d632f5fb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:55:16.545907 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:16.545861    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00572ed5-fa0b-4297-8087-24d4d632f5fb-kube-api-access-w8mgx" pod "00572ed5-fa0b-4297-8087-24d4d632f5fb" (UID: "00572ed5-fa0b-4297-8087-24d4d632f5fb"). InnerVolumeSpecName "kube-api-access-w8mgx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:55:16.645529 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:16.645484    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8mgx\" (UniqueName: \"kubernetes.io/projected/00572ed5-fa0b-4297-8087-24d4d632f5fb-kube-api-access-w8mgx\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:55:16.645529 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:16.645529    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00572ed5-fa0b-4297-8087-24d4d632f5fb-maas-api-xks-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:55:17.193880 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:17.193849    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55360e0c-vhnxl"
Oct 31 12:55:17.194331 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:17.193881    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55360e0c-vhnxl" event={"ID":"00572ed5-fa0b-4297-8087-24d4d632f5fb","Type":"ContainerDied","Data":"0f9128786f9588a292483253010b139b08f2d8c9fc904ca7ebc2bfd84031614f"}
Oct 31 12:55:17.194331 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:17.193937    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0f9128786f9588a292483253010b139b08f2d8c9fc904ca7ebc2bfd84031614f"
Oct 31 12:55:17.371516 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:17.371474    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f231812-xgbw7"
Oct 31 12:55:17.456342 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:17.456304    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e616d24-af13-4f8f-93f0-bae103fceca8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e616d24-af13-4f8f-93f0-bae103fceca8-maas-api-openshift-ca\") pod \"3e616d24-af13-4f8f-93f0-bae103fceca8\" (UID: \"3e616d24-af13-4f8f-93f0-bae103fceca8\") "
Oct 31 12:55:17.456609 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:17.456553    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e616d24-af13-4f8f-93f0-bae103fceca8-kube-api-access-qqn7r\" (UniqueName: \"kubernetes.io/projected/3e616d24-af13-4f8f-93f0-bae103fceca8-kube-api-access-qqn7r\") pod \"3e616d24-af13-4f8f-93f0-bae103fceca8\" (UID: \"3e616d24-af13-4f8f-93f0-bae103fceca8\") "
Oct 31 12:55:17.456694 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:17.456658    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3e616d24-af13-4f8f-93f0-bae103fceca8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e616d24-af13-4f8f-93f0-bae103fceca8-maas-api-xks-ca\") pod \"3e616d24-af13-4f8f-93f0-bae103fceca8\" (UID: \"3e616d24-af13-4f8f-93f0-bae103fceca8\") "
Oct 31 12:55:17.456752 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:17.456718    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e616d24-af13-4f8f-93f0-bae103fceca8-maas-api-openshift-ca" pod "3e616d24-af13-4f8f-93f0-bae103fceca8" (UID: "3e616d24-af13-4f8f-93f0-bae103fceca8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:55:17.457337 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:17.457309    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e616d24-af13-4f8f-93f0-bae103fceca8-maas-api-openshift-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:55:17.459648 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:17.459599    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e616d24-af13-4f8f-93f0-bae103fceca8-maas-api-xks-ca" pod "3e616d24-af13-4f8f-93f0-bae103fceca8" (UID: "3e616d24-af13-4f8f-93f0-bae103fceca8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:55:17.459648 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:17.459623    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e616d24-af13-4f8f-93f0-bae103fceca8-kube-api-access-qqn7r" pod "3e616d24-af13-4f8f-93f0-bae103fceca8" (UID: "3e616d24-af13-4f8f-93f0-bae103fceca8"). InnerVolumeSpecName "kube-api-access-qqn7r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:55:17.558498 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:17.558451    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqn7r\" (UniqueName: \"kubernetes.io/projected/3e616d24-af13-4f8f-93f0-bae103fceca8-kube-api-access-qqn7r\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:55:17.558498 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:17.558484    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e616d24-af13-4f8f-93f0-bae103fceca8-maas-api-xks-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:55:18.206738 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:18.206703    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f231812-xgbw7"
Oct 31 12:55:18.207130 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:18.206731    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f231812-xgbw7" event={"ID":"3e616d24-af13-4f8f-93f0-bae103fceca8","Type":"ContainerDied","Data":"863d235cd93048e1d1cb3717d0b8407a4e94ca7c9f7ae77b1441f21659d34453"}
Oct 31 12:55:18.207130 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:18.206790    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="863d235cd93048e1d1cb3717d0b8407a4e94ca7c9f7ae77b1441f21659d34453"
Oct 31 12:55:20.284855 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:20.284804    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b30aae92-c2vmq"]
Oct 31 12:55:20.286756 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:20.286723    5060 cpu_manager.go:414] "containerMap: removing container" podUID="3e616d24-af13-4f8f-93f0-bae103fceca8" containerName="revoke-keys"
Oct 31 12:55:20.287007 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:20.286987    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e616d24-af13-4f8f-93f0-bae103fceca8" containerName="revoke-keys"
Oct 31 12:55:20.287217 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:20.287202    5060 cpu_manager.go:414] "containerMap: removing container" podUID="00572ed5-fa0b-4297-8087-24d4d632f5fb" containerName="revoke-keys"
Oct 31 12:55:20.287320 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:20.287307    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00572ed5-fa0b-4297-8087-24d4d632f5fb" containerName="revoke-keys"
Oct 31 12:55:20.287760 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:20.287740    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="00572ed5-fa0b-4297-8087-24d4d632f5fb" containerName="revoke-keys"
Oct 31 12:55:20.287885 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:20.287873    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e616d24-af13-4f8f-93f0-bae103fceca8" containerName="revoke-keys"
Oct 31 12:55:20.294537 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:20.292718    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b30aae92-c2vmq"
Oct 31 12:55:20.299777 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:20.299753    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 12:55:20.300076 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:20.300058    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7wsmb\""
Oct 31 12:55:20.306779 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:20.306749    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b30aae92-c2vmq"]
Oct 31 12:55:20.317622 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:20.317152    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68c38c1d-abc4-44c0-9280-c378cb1b4590-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b30aae92-c2vmq\" (UID: \"68c38c1d-abc4-44c0-9280-c378cb1b4590\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b30aae92-c2vmq"
Oct 31 12:55:20.317622 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:20.317238    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8z55\" (UniqueName: \"kubernetes.io/projected/68c38c1d-abc4-44c0-9280-c378cb1b4590-kube-api-access-z8z55\") pod \"maas-api-revoke-sub-b30aae92-c2vmq\" (UID: \"68c38c1d-abc4-44c0-9280-c378cb1b4590\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b30aae92-c2vmq"
Oct 31 12:55:20.317622 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:20.317276    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68c38c1d-abc4-44c0-9280-c378cb1b4590-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b30aae92-c2vmq\" (UID: \"68c38c1d-abc4-44c0-9280-c378cb1b4590\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b30aae92-c2vmq"
Oct 31 12:55:20.511530 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:20.509089    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68c38c1d-abc4-44c0-9280-c378cb1b4590-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b30aae92-c2vmq\" (UID: \"68c38c1d-abc4-44c0-9280-c378cb1b4590\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b30aae92-c2vmq"
Oct 31 12:55:20.511530 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:20.509185    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8z55\" (UniqueName: \"kubernetes.io/projected/68c38c1d-abc4-44c0-9280-c378cb1b4590-kube-api-access-z8z55\") pod \"maas-api-revoke-sub-b30aae92-c2vmq\" (UID: \"68c38c1d-abc4-44c0-9280-c378cb1b4590\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b30aae92-c2vmq"
Oct 31 12:55:20.511530 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:20.509228    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68c38c1d-abc4-44c0-9280-c378cb1b4590-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b30aae92-c2vmq\" (UID: \"68c38c1d-abc4-44c0-9280-c378cb1b4590\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b30aae92-c2vmq"
Oct 31 12:55:20.557754 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:20.529599    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68c38c1d-abc4-44c0-9280-c378cb1b4590-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b30aae92-c2vmq\" (UID: \"68c38c1d-abc4-44c0-9280-c378cb1b4590\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b30aae92-c2vmq"
Oct 31 12:55:20.557754 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:20.544535    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8z55\" (UniqueName: \"kubernetes.io/projected/68c38c1d-abc4-44c0-9280-c378cb1b4590-kube-api-access-z8z55\") pod \"maas-api-revoke-sub-b30aae92-c2vmq\" (UID: \"68c38c1d-abc4-44c0-9280-c378cb1b4590\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b30aae92-c2vmq"
Oct 31 12:55:20.570518 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:20.560987    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68c38c1d-abc4-44c0-9280-c378cb1b4590-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b30aae92-c2vmq\" (UID: \"68c38c1d-abc4-44c0-9280-c378cb1b4590\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b30aae92-c2vmq"
Oct 31 12:55:20.626521 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:20.625110    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b30aae92-c2vmq"
Oct 31 12:55:21.174521 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:21.174299    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-e6c79d/premium-w2-e6c79d-kserve-6468bf46d4-h66f8" podUID="cd6a266f-8d12-4089-83ee-c75c716bf26c" containerName="main" containerID="cri-o://4d45f66c55ebe797c041b70d4991a2c802448b1a226ac362d630ca349b5f5bf1" gracePeriod=45
Oct 31 12:55:21.195448 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:21.195297    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b30aae92-c2vmq"]
Oct 31 12:55:21.405961 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:21.405864    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b30aae92-c2vmq" event={"ID":"68c38c1d-abc4-44c0-9280-c378cb1b4590","Type":"ContainerStarted","Data":"30940f3b5486e4b3b40591f72e0c2c519c67cd3f6a11577f6aef55853af07e53"}
Oct 31 12:55:21.681750 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:21.681710    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/premium-w2-e6c79d-kserve-6468bf46d4-h66f8" podUID="cd6a266f-8d12-4089-83ee-c75c716bf26c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": dial tcp 10.132.0.80:8000: connect: connection refused"
Oct 31 12:55:22.628543 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:22.628364    5060 generic.go:356] "Generic (PLEG): container finished" podID="cd6a266f-8d12-4089-83ee-c75c716bf26c" containerID="4d45f66c55ebe797c041b70d4991a2c802448b1a226ac362d630ca349b5f5bf1" exitCode=0
Oct 31 12:55:22.629247 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:22.629218    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e6c79d/premium-w2-e6c79d-kserve-6468bf46d4-h66f8" event={"ID":"cd6a266f-8d12-4089-83ee-c75c716bf26c","Type":"ContainerDied","Data":"4d45f66c55ebe797c041b70d4991a2c802448b1a226ac362d630ca349b5f5bf1"}
Oct 31 12:55:22.816532 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:22.816470    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e6c79d/premium-w2-e6c79d-kserve-6468bf46d4-h66f8"
Oct 31 12:55:22.922048 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:22.903387    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd6a266f-8d12-4089-83ee-c75c716bf26c-kube-api-access-7jnzw\" (UniqueName: \"kubernetes.io/projected/cd6a266f-8d12-4089-83ee-c75c716bf26c-kube-api-access-7jnzw\") pod \"cd6a266f-8d12-4089-83ee-c75c716bf26c\" (UID: \"cd6a266f-8d12-4089-83ee-c75c716bf26c\") "
Oct 31 12:55:22.922048 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:22.903455    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd6a266f-8d12-4089-83ee-c75c716bf26c-tls-certs\" (UniqueName: \"kubernetes.io/secret/cd6a266f-8d12-4089-83ee-c75c716bf26c-tls-certs\") pod \"cd6a266f-8d12-4089-83ee-c75c716bf26c\" (UID: \"cd6a266f-8d12-4089-83ee-c75c716bf26c\") "
Oct 31 12:55:22.922048 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:22.903570    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd6a266f-8d12-4089-83ee-c75c716bf26c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd6a266f-8d12-4089-83ee-c75c716bf26c-dshm\") pod \"cd6a266f-8d12-4089-83ee-c75c716bf26c\" (UID: \"cd6a266f-8d12-4089-83ee-c75c716bf26c\") "
Oct 31 12:55:22.922048 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:22.903603    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd6a266f-8d12-4089-83ee-c75c716bf26c-home\" (UniqueName: \"kubernetes.io/empty-dir/cd6a266f-8d12-4089-83ee-c75c716bf26c-home\") pod \"cd6a266f-8d12-4089-83ee-c75c716bf26c\" (UID: \"cd6a266f-8d12-4089-83ee-c75c716bf26c\") "
Oct 31 12:55:22.922048 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:22.903648    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd6a266f-8d12-4089-83ee-c75c716bf26c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd6a266f-8d12-4089-83ee-c75c716bf26c-model-cache\") pod \"cd6a266f-8d12-4089-83ee-c75c716bf26c\" (UID: \"cd6a266f-8d12-4089-83ee-c75c716bf26c\") "
Oct 31 12:55:22.922048 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:22.903740    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd6a266f-8d12-4089-83ee-c75c716bf26c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd6a266f-8d12-4089-83ee-c75c716bf26c-tmp-dir\") pod \"cd6a266f-8d12-4089-83ee-c75c716bf26c\" (UID: \"cd6a266f-8d12-4089-83ee-c75c716bf26c\") "
Oct 31 12:55:22.922048 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:22.907713    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd6a266f-8d12-4089-83ee-c75c716bf26c-tmp-dir" pod "cd6a266f-8d12-4089-83ee-c75c716bf26c" (UID: "cd6a266f-8d12-4089-83ee-c75c716bf26c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:55:22.922048 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:22.911802    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd6a266f-8d12-4089-83ee-c75c716bf26c-kube-api-access-7jnzw" pod "cd6a266f-8d12-4089-83ee-c75c716bf26c" (UID: "cd6a266f-8d12-4089-83ee-c75c716bf26c"). InnerVolumeSpecName "kube-api-access-7jnzw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:55:22.922048 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:22.915472    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd6a266f-8d12-4089-83ee-c75c716bf26c-tls-certs" pod "cd6a266f-8d12-4089-83ee-c75c716bf26c" (UID: "cd6a266f-8d12-4089-83ee-c75c716bf26c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:55:22.922048 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:22.918526    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd6a266f-8d12-4089-83ee-c75c716bf26c-home" pod "cd6a266f-8d12-4089-83ee-c75c716bf26c" (UID: "cd6a266f-8d12-4089-83ee-c75c716bf26c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:55:22.922048 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:22.919072    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd6a266f-8d12-4089-83ee-c75c716bf26c-model-cache" pod "cd6a266f-8d12-4089-83ee-c75c716bf26c" (UID: "cd6a266f-8d12-4089-83ee-c75c716bf26c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:55:22.922048 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:22.919405    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-e6c79d-7f6dbd5c9-psn2f" podUID="3f6d17e0-1d51-41b8-a52c-a386a8fdcc17" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 12:55:22.939585 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:22.939538    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd6a266f-8d12-4089-83ee-c75c716bf26c-dshm" pod "cd6a266f-8d12-4089-83ee-c75c716bf26c" (UID: "cd6a266f-8d12-4089-83ee-c75c716bf26c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:55:23.018301 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:23.018260    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7jnzw\" (UniqueName: \"kubernetes.io/projected/cd6a266f-8d12-4089-83ee-c75c716bf26c-kube-api-access-7jnzw\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:55:23.018565 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:23.018546    5060 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd6a266f-8d12-4089-83ee-c75c716bf26c-tls-certs\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:55:23.018694 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:23.018681    5060 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd6a266f-8d12-4089-83ee-c75c716bf26c-dshm\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:55:23.018828 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:23.018810    5060 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd6a266f-8d12-4089-83ee-c75c716bf26c-home\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:55:23.018918 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:23.018906    5060 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd6a266f-8d12-4089-83ee-c75c716bf26c-model-cache\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:55:23.018997 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:23.018986    5060 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd6a266f-8d12-4089-83ee-c75c716bf26c-tmp-dir\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:55:23.641375 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:23.641295    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e6c79d/premium-w2-e6c79d-kserve-6468bf46d4-h66f8"
Oct 31 12:55:23.641983 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:23.641317    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e6c79d/premium-w2-e6c79d-kserve-6468bf46d4-h66f8" event={"ID":"cd6a266f-8d12-4089-83ee-c75c716bf26c","Type":"ContainerDied","Data":"d044400765c484e767d3a28f77a8ae1428d4fd428f0ae87b9e7e702f03fda464"}
Oct 31 12:55:23.642194 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:23.642177    5060 scope.go:122] "RemoveContainer" containerID="4d45f66c55ebe797c041b70d4991a2c802448b1a226ac362d630ca349b5f5bf1"
Oct 31 12:55:23.644058 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:23.644034    5060 generic.go:356] "Generic (PLEG): container finished" podID="68c38c1d-abc4-44c0-9280-c378cb1b4590" containerID="7c7ae659a71f10768cffcbb8ddebdb945b3a38814ceb9134d99f0a663d784c92" exitCode=0
Oct 31 12:55:23.644165 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:23.644099    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b30aae92-c2vmq" event={"ID":"68c38c1d-abc4-44c0-9280-c378cb1b4590","Type":"ContainerDied","Data":"7c7ae659a71f10768cffcbb8ddebdb945b3a38814ceb9134d99f0a663d784c92"}
Oct 31 12:55:23.691074 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:23.690572    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/premium-w2-e6c79d-kserve-6468bf46d4-h66f8" podUID="cd6a266f-8d12-4089-83ee-c75c716bf26c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:55:23.691074 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:23.690672    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e6c79d/premium-w2-e6c79d-kserve-6468bf46d4-h66f8"
Oct 31 12:55:23.727608 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:23.727574    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-e6c79d/premium-w2-e6c79d-kserve-6468bf46d4-h66f8"]
Oct 31 12:55:23.741406 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:23.741365    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-e6c79d/premium-w2-e6c79d-kserve-6468bf46d4-h66f8"]
Oct 31 12:55:25.143783 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:25.143748    5060 scope.go:122] "RemoveContainer" containerID="241445e5ee669bb9b3c7e968e26e95300e51aa8e9808e0228a5643c1d68276ac"
Oct 31 12:55:25.154312 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:25.154268    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd6a266f-8d12-4089-83ee-c75c716bf26c" path="/var/lib/kubelet/pods/cd6a266f-8d12-4089-83ee-c75c716bf26c/volumes"
Oct 31 12:55:25.242526 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:25.238556    5060 scope.go:122] "RemoveContainer" containerID="2fa664b61741f11c1b98eb0639d5b30fd6365667ab609cc869b5d321b65c5e71"
Oct 31 12:55:25.346838 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:25.346808    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b30aae92-c2vmq"
Oct 31 12:55:25.443792 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:25.443756    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68c38c1d-abc4-44c0-9280-c378cb1b4590-kube-api-access-z8z55\" (UniqueName: \"kubernetes.io/projected/68c38c1d-abc4-44c0-9280-c378cb1b4590-kube-api-access-z8z55\") pod \"68c38c1d-abc4-44c0-9280-c378cb1b4590\" (UID: \"68c38c1d-abc4-44c0-9280-c378cb1b4590\") "
Oct 31 12:55:25.443914 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:25.443847    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/68c38c1d-abc4-44c0-9280-c378cb1b4590-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68c38c1d-abc4-44c0-9280-c378cb1b4590-maas-api-xks-ca\") pod \"68c38c1d-abc4-44c0-9280-c378cb1b4590\" (UID: \"68c38c1d-abc4-44c0-9280-c378cb1b4590\") "
Oct 31 12:55:25.444088 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:25.444071    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68c38c1d-abc4-44c0-9280-c378cb1b4590-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68c38c1d-abc4-44c0-9280-c378cb1b4590-maas-api-openshift-ca\") pod \"68c38c1d-abc4-44c0-9280-c378cb1b4590\" (UID: \"68c38c1d-abc4-44c0-9280-c378cb1b4590\") "
Oct 31 12:55:25.444820 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:25.444795    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68c38c1d-abc4-44c0-9280-c378cb1b4590-maas-api-openshift-ca" pod "68c38c1d-abc4-44c0-9280-c378cb1b4590" (UID: "68c38c1d-abc4-44c0-9280-c378cb1b4590"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:55:25.446973 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:25.446949    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68c38c1d-abc4-44c0-9280-c378cb1b4590-kube-api-access-z8z55" pod "68c38c1d-abc4-44c0-9280-c378cb1b4590" (UID: "68c38c1d-abc4-44c0-9280-c378cb1b4590"). InnerVolumeSpecName "kube-api-access-z8z55". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:55:25.449116 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:25.449097    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68c38c1d-abc4-44c0-9280-c378cb1b4590-maas-api-xks-ca" pod "68c38c1d-abc4-44c0-9280-c378cb1b4590" (UID: "68c38c1d-abc4-44c0-9280-c378cb1b4590"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:55:25.546013 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:25.545890    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68c38c1d-abc4-44c0-9280-c378cb1b4590-maas-api-openshift-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:55:25.546013 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:25.545924    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8z55\" (UniqueName: \"kubernetes.io/projected/68c38c1d-abc4-44c0-9280-c378cb1b4590-kube-api-access-z8z55\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:55:25.546013 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:25.545938    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68c38c1d-abc4-44c0-9280-c378cb1b4590-maas-api-xks-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:55:25.920512 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:25.903266    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b30aae92-c2vmq" event={"ID":"68c38c1d-abc4-44c0-9280-c378cb1b4590","Type":"ContainerDied","Data":"30940f3b5486e4b3b40591f72e0c2c519c67cd3f6a11577f6aef55853af07e53"}
Oct 31 12:55:25.920512 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:25.903335    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="30940f3b5486e4b3b40591f72e0c2c519c67cd3f6a11577f6aef55853af07e53"
Oct 31 12:55:25.920512 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:25.903418    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b30aae92-c2vmq"
Oct 31 12:55:30.001512 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:55:29.984140    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7phb" containerName="extractor"
Oct 31 12:55:31.844563 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:31.840852    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-015701-jdsj9"]
Oct 31 12:55:31.858591 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:31.858076    5060 cpu_manager.go:414] "containerMap: removing container" podUID="cd6a266f-8d12-4089-83ee-c75c716bf26c" containerName="main"
Oct 31 12:55:31.858591 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:31.858110    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd6a266f-8d12-4089-83ee-c75c716bf26c" containerName="main"
Oct 31 12:55:31.858591 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:31.858146    5060 cpu_manager.go:414] "containerMap: removing container" podUID="68c38c1d-abc4-44c0-9280-c378cb1b4590" containerName="revoke-keys"
Oct 31 12:55:31.858591 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:31.858154    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68c38c1d-abc4-44c0-9280-c378cb1b4590" containerName="revoke-keys"
Oct 31 12:55:31.858591 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:31.858346    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="68c38c1d-abc4-44c0-9280-c378cb1b4590" containerName="revoke-keys"
Oct 31 12:55:31.858591 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:31.858369    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd6a266f-8d12-4089-83ee-c75c716bf26c" containerName="main"
Oct 31 12:55:31.929256 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:31.929205    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-015701-jdsj9"]
Oct 31 12:55:31.929646 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:31.929347    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-015701-jdsj9"
Oct 31 12:55:31.933446 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:31.932818    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fcc724c-21ea-4b0e-b540-6293ae707096-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-015701-jdsj9\" (UID: \"8fcc724c-21ea-4b0e-b540-6293ae707096\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-015701-jdsj9"
Oct 31 12:55:31.933446 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:31.932920    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fcc724c-21ea-4b0e-b540-6293ae707096-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-015701-jdsj9\" (UID: \"8fcc724c-21ea-4b0e-b540-6293ae707096\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-015701-jdsj9"
Oct 31 12:55:31.933446 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:31.933131    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9bp2\" (UniqueName: \"kubernetes.io/projected/8fcc724c-21ea-4b0e-b540-6293ae707096-kube-api-access-r9bp2\") pod \"maas-api-revoke-keys-e2e-inf-a-015701-jdsj9\" (UID: \"8fcc724c-21ea-4b0e-b540-6293ae707096\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-015701-jdsj9"
Oct 31 12:55:31.935788 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:31.935761    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 12:55:31.936057 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:31.936035    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7wsmb\""
Oct 31 12:55:32.034234 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:32.034186    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fcc724c-21ea-4b0e-b540-6293ae707096-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-015701-jdsj9\" (UID: \"8fcc724c-21ea-4b0e-b540-6293ae707096\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-015701-jdsj9"
Oct 31 12:55:32.034480 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:32.034368    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9bp2\" (UniqueName: \"kubernetes.io/projected/8fcc724c-21ea-4b0e-b540-6293ae707096-kube-api-access-r9bp2\") pod \"maas-api-revoke-keys-e2e-inf-a-015701-jdsj9\" (UID: \"8fcc724c-21ea-4b0e-b540-6293ae707096\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-015701-jdsj9"
Oct 31 12:55:32.034480 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:32.034457    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fcc724c-21ea-4b0e-b540-6293ae707096-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-015701-jdsj9\" (UID: \"8fcc724c-21ea-4b0e-b540-6293ae707096\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-015701-jdsj9"
Oct 31 12:55:32.034925 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:32.034905    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fcc724c-21ea-4b0e-b540-6293ae707096-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-015701-jdsj9\" (UID: \"8fcc724c-21ea-4b0e-b540-6293ae707096\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-015701-jdsj9"
Oct 31 12:55:32.037185 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:32.037159    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fcc724c-21ea-4b0e-b540-6293ae707096-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-015701-jdsj9\" (UID: \"8fcc724c-21ea-4b0e-b540-6293ae707096\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-015701-jdsj9"
Oct 31 12:55:32.048193 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:32.048158    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9bp2\" (UniqueName: \"kubernetes.io/projected/8fcc724c-21ea-4b0e-b540-6293ae707096-kube-api-access-r9bp2\") pod \"maas-api-revoke-keys-e2e-inf-a-015701-jdsj9\" (UID: \"8fcc724c-21ea-4b0e-b540-6293ae707096\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-015701-jdsj9"
Oct 31 12:55:32.288187 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:32.288152    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-015701-jdsj9"
Oct 31 12:55:32.643400 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:55:32.643363    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fcc724c_21ea_4b0e_b540_6293ae707096.slice/crio-41763978bf2c29d7708335ddf8bf393a451aa7441a3c0092f7907830cfce4b8d WatchSource:0}: Error finding container 41763978bf2c29d7708335ddf8bf393a451aa7441a3c0092f7907830cfce4b8d: Status 404 returned error can't find the container with id 41763978bf2c29d7708335ddf8bf393a451aa7441a3c0092f7907830cfce4b8d
Oct 31 12:55:32.646769 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:32.646740    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-015701-jdsj9"]
Oct 31 12:55:32.900937 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:32.900855    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-e6c79d-7f6dbd5c9-psn2f" podUID="3f6d17e0-1d51-41b8-a52c-a386a8fdcc17" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 12:55:33.030554 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:33.030517    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-015701-jdsj9" event={"ID":"8fcc724c-21ea-4b0e-b540-6293ae707096","Type":"ContainerStarted","Data":"41763978bf2c29d7708335ddf8bf393a451aa7441a3c0092f7907830cfce4b8d"}
Oct 31 12:55:34.735832 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:34.045679    5060 generic.go:356] "Generic (PLEG): container finished" podID="8fcc724c-21ea-4b0e-b540-6293ae707096" containerID="0c73620ff7fd6c14c6f1cc8f744c0dd5935bd0c8fa36f7e8893ecf82a64731a9" exitCode=0
Oct 31 12:55:34.735832 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:34.045734    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-015701-jdsj9" event={"ID":"8fcc724c-21ea-4b0e-b540-6293ae707096","Type":"ContainerDied","Data":"0c73620ff7fd6c14c6f1cc8f744c0dd5935bd0c8fa36f7e8893ecf82a64731a9"}
Oct 31 12:55:34.735832 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:34.303658    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e6c79d-kh295"]
Oct 31 12:55:34.735832 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:34.346134    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e6c79d-kh295"]
Oct 31 12:55:35.067226 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:35.067181    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e849745c-8575-4bf1-81d9-f032ddd5e9fe" path="/var/lib/kubelet/pods/e849745c-8575-4bf1-81d9-f032ddd5e9fe/volumes"
Oct 31 12:55:35.069187 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:35.069154    5060 generic.go:356] "Generic (PLEG): container finished" podID="0c93ff68-7e63-40b2-882b-8b2aed13e982" containerID="55e1b15a513434f3d2d8350ad52e7342327fa7ab784312e346b4241a14ba93ee" exitCode=0
Oct 31 12:55:35.069349 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:35.069229    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-fd988c-openshift-default-96bd4cd5c-g4dgc" event={"ID":"0c93ff68-7e63-40b2-882b-8b2aed13e982","Type":"ContainerDied","Data":"55e1b15a513434f3d2d8350ad52e7342327fa7ab784312e346b4241a14ba93ee"}
Oct 31 12:55:35.069744 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:35.069725    5060 scope.go:122] "RemoveContainer" containerID="55e1b15a513434f3d2d8350ad52e7342327fa7ab784312e346b4241a14ba93ee"
Oct 31 12:55:35.072066 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:35.072027    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:55:35.072194 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:35.072118    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:55:35.072194 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:35.072161    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:55:35.379055 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:35.379011    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-015701-jdsj9"
Oct 31 12:55:35.432046 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:35.431999    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8fcc724c-21ea-4b0e-b540-6293ae707096-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fcc724c-21ea-4b0e-b540-6293ae707096-maas-api-openshift-ca\") pod \"8fcc724c-21ea-4b0e-b540-6293ae707096\" (UID: \"8fcc724c-21ea-4b0e-b540-6293ae707096\") "
Oct 31 12:55:35.432282 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:35.432144    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fcc724c-21ea-4b0e-b540-6293ae707096-kube-api-access-r9bp2\" (UniqueName: \"kubernetes.io/projected/8fcc724c-21ea-4b0e-b540-6293ae707096-kube-api-access-r9bp2\") pod \"8fcc724c-21ea-4b0e-b540-6293ae707096\" (UID: \"8fcc724c-21ea-4b0e-b540-6293ae707096\") "
Oct 31 12:55:35.432282 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:35.432193    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8fcc724c-21ea-4b0e-b540-6293ae707096-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fcc724c-21ea-4b0e-b540-6293ae707096-maas-api-xks-ca\") pod \"8fcc724c-21ea-4b0e-b540-6293ae707096\" (UID: \"8fcc724c-21ea-4b0e-b540-6293ae707096\") "
Oct 31 12:55:35.433094 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:35.433059    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fcc724c-21ea-4b0e-b540-6293ae707096-maas-api-openshift-ca" pod "8fcc724c-21ea-4b0e-b540-6293ae707096" (UID: "8fcc724c-21ea-4b0e-b540-6293ae707096"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:55:35.446956 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:35.446909    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8fcc724c-21ea-4b0e-b540-6293ae707096-maas-api-xks-ca" pod "8fcc724c-21ea-4b0e-b540-6293ae707096" (UID: "8fcc724c-21ea-4b0e-b540-6293ae707096"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:55:35.447091 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:35.447043    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fcc724c-21ea-4b0e-b540-6293ae707096-kube-api-access-r9bp2" pod "8fcc724c-21ea-4b0e-b540-6293ae707096" (UID: "8fcc724c-21ea-4b0e-b540-6293ae707096"). InnerVolumeSpecName "kube-api-access-r9bp2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:55:35.534802 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:35.534725    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r9bp2\" (UniqueName: \"kubernetes.io/projected/8fcc724c-21ea-4b0e-b540-6293ae707096-kube-api-access-r9bp2\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:55:35.534802 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:35.534762    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fcc724c-21ea-4b0e-b540-6293ae707096-maas-api-xks-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:55:35.534802 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:35.534776    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fcc724c-21ea-4b0e-b540-6293ae707096-maas-api-openshift-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:55:36.092942 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:36.092895    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-015701-jdsj9" event={"ID":"8fcc724c-21ea-4b0e-b540-6293ae707096","Type":"ContainerDied","Data":"41763978bf2c29d7708335ddf8bf393a451aa7441a3c0092f7907830cfce4b8d"}
Oct 31 12:55:36.093570 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:36.092968    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="41763978bf2c29d7708335ddf8bf393a451aa7441a3c0092f7907830cfce4b8d"
Oct 31 12:55:36.093570 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:36.093046    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-015701-jdsj9"
Oct 31 12:55:36.100053 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:36.100011    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-fd988c-openshift-default-96bd4cd5c-g4dgc" event={"ID":"0c93ff68-7e63-40b2-882b-8b2aed13e982","Type":"ContainerStarted","Data":"e162b85c3c8d5b2f01ce21d6b80bffca81e78ad33c6852cd87ec34686222ad6d"}
Oct 31 12:55:36.915307 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:36.834400    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-e6c79d-7f6dbd5c9-psn2f"
Oct 31 12:55:36.915307 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:36.853169    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f6d17e0-1d51-41b8-a52c-a386a8fdcc17-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f6d17e0-1d51-41b8-a52c-a386a8fdcc17-plugins-config-volume\") pod \"3f6d17e0-1d51-41b8-a52c-a386a8fdcc17\" (UID: \"3f6d17e0-1d51-41b8-a52c-a386a8fdcc17\") "
Oct 31 12:55:36.915307 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:36.853253    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f6d17e0-1d51-41b8-a52c-a386a8fdcc17-kube-api-access-h4zkd\" (UniqueName: \"kubernetes.io/projected/3f6d17e0-1d51-41b8-a52c-a386a8fdcc17-kube-api-access-h4zkd\") pod \"3f6d17e0-1d51-41b8-a52c-a386a8fdcc17\" (UID: \"3f6d17e0-1d51-41b8-a52c-a386a8fdcc17\") "
Oct 31 12:55:36.915307 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:36.853538    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f6d17e0-1d51-41b8-a52c-a386a8fdcc17-plugins-config-volume" pod "3f6d17e0-1d51-41b8-a52c-a386a8fdcc17" (UID: "3f6d17e0-1d51-41b8-a52c-a386a8fdcc17"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:55:36.915307 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:36.853756    5060 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f6d17e0-1d51-41b8-a52c-a386a8fdcc17-plugins-config-volume\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:55:36.915307 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:36.894669    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f6d17e0-1d51-41b8-a52c-a386a8fdcc17-kube-api-access-h4zkd" pod "3f6d17e0-1d51-41b8-a52c-a386a8fdcc17" (UID: "3f6d17e0-1d51-41b8-a52c-a386a8fdcc17"). InnerVolumeSpecName "kube-api-access-h4zkd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:55:36.954531 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:36.953695    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-fd988c-openshift-default-96bd4cd5c-g4dgc"
Oct 31 12:55:36.962741 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:36.954991    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4zkd\" (UniqueName: \"kubernetes.io/projected/3f6d17e0-1d51-41b8-a52c-a386a8fdcc17-kube-api-access-h4zkd\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:55:36.986612 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:36.982708    5060 patch_prober.go:29] interesting pod/e2e-inf-b-fd988c-openshift-default-96bd4cd5c-g4dgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" start-of-body=
Oct 31 12:55:36.986612 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:36.982783    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-fd988c-openshift-default-96bd4cd5c-g4dgc" podUID="0c93ff68-7e63-40b2-882b-8b2aed13e982" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused"
Oct 31 12:55:37.113747 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:37.113715    5060 generic.go:356] "Generic (PLEG): container finished" podID="25899808-1323-425c-9359-3b56f99ad5cb" containerID="26e44b9d2ea1cc00868da3adafd4ba6e1bbce4e6f6194a2b35f7de329d87b522" exitCode=0
Oct 31 12:55:37.114223 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:37.113783    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-7stx7" event={"ID":"25899808-1323-425c-9359-3b56f99ad5cb","Type":"ContainerDied","Data":"26e44b9d2ea1cc00868da3adafd4ba6e1bbce4e6f6194a2b35f7de329d87b522"}
Oct 31 12:55:37.114308 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:37.114253    5060 scope.go:122] "RemoveContainer" containerID="26e44b9d2ea1cc00868da3adafd4ba6e1bbce4e6f6194a2b35f7de329d87b522"
Oct 31 12:55:37.115594 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:37.115568    5060 generic.go:356] "Generic (PLEG): container finished" podID="3f6d17e0-1d51-41b8-a52c-a386a8fdcc17" containerID="3e3c21381670c8a6131d9ad5ce41c83a85369b90f0088941ea202ce7538723a7" exitCode=0
Oct 31 12:55:37.115772 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:37.115644    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-e6c79d-7f6dbd5c9-psn2f" event={"ID":"3f6d17e0-1d51-41b8-a52c-a386a8fdcc17","Type":"ContainerDied","Data":"3e3c21381670c8a6131d9ad5ce41c83a85369b90f0088941ea202ce7538723a7"}
Oct 31 12:55:37.115772 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:37.115688    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-e6c79d-7f6dbd5c9-psn2f"
Oct 31 12:55:37.115772 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:37.115697    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-e6c79d-7f6dbd5c9-psn2f" event={"ID":"3f6d17e0-1d51-41b8-a52c-a386a8fdcc17","Type":"ContainerDied","Data":"a909bf9bf2d1d13edc96f405b65ec759bcffa0bf05968dffe732368ca47c0420"}
Oct 31 12:55:37.115772 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:37.115729    5060 scope.go:122] "RemoveContainer" containerID="3e3c21381670c8a6131d9ad5ce41c83a85369b90f0088941ea202ce7538723a7"
Oct 31 12:55:37.116856 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:37.116791    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:55:37.117025 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:37.116889    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:55:37.117025 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:37.116937    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:55:37.289736 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:37.283427    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-s9nqq"]
Oct 31 12:55:37.289736 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:37.285142    5060 cpu_manager.go:414] "containerMap: removing container" podUID="8fcc724c-21ea-4b0e-b540-6293ae707096" containerName="revoke-keys"
Oct 31 12:55:37.289736 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:37.285167    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8fcc724c-21ea-4b0e-b540-6293ae707096" containerName="revoke-keys"
Oct 31 12:55:37.289736 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:37.285289    5060 cpu_manager.go:414] "containerMap: removing container" podUID="3f6d17e0-1d51-41b8-a52c-a386a8fdcc17" containerName="payload-processing"
Oct 31 12:55:37.289736 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:37.285298    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f6d17e0-1d51-41b8-a52c-a386a8fdcc17" containerName="payload-processing"
Oct 31 12:55:37.289736 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:37.285621    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f6d17e0-1d51-41b8-a52c-a386a8fdcc17" containerName="payload-processing"
Oct 31 12:55:37.289736 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:37.285648    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="8fcc724c-21ea-4b0e-b540-6293ae707096" containerName="revoke-keys"
Oct 31 12:55:37.316247 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:37.316212    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-s9nqq"
Oct 31 12:55:37.344028 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:37.343999    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-s9nqq"]
Oct 31 12:55:37.350312 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:37.350290    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-e6c79d-7f6dbd5c9-psn2f"]
Oct 31 12:55:37.359168 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:37.359141    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-e6c79d-7f6dbd5c9-psn2f"]
Oct 31 12:55:37.398301 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:37.398184    5060 scope.go:122] "RemoveContainer" containerID="3e3c21381670c8a6131d9ad5ce41c83a85369b90f0088941ea202ce7538723a7"
Oct 31 12:55:37.398691 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:55:37.398664    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e3c21381670c8a6131d9ad5ce41c83a85369b90f0088941ea202ce7538723a7\": container with ID starting with 3e3c21381670c8a6131d9ad5ce41c83a85369b90f0088941ea202ce7538723a7 not found: ID does not exist" containerID="3e3c21381670c8a6131d9ad5ce41c83a85369b90f0088941ea202ce7538723a7"
Oct 31 12:55:37.398814 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:37.398703    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e3c21381670c8a6131d9ad5ce41c83a85369b90f0088941ea202ce7538723a7"} err="failed to get container status \"3e3c21381670c8a6131d9ad5ce41c83a85369b90f0088941ea202ce7538723a7\": rpc error: code = NotFound desc = could not find container \"3e3c21381670c8a6131d9ad5ce41c83a85369b90f0088941ea202ce7538723a7\": container with ID starting with 3e3c21381670c8a6131d9ad5ce41c83a85369b90f0088941ea202ce7538723a7 not found: ID does not exist"
Oct 31 12:55:37.450311 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:37.450277    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebcf940c-4d23-4944-9948-d3dea5ba99f6-workload-socket\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-s9nqq\" (UID: \"ebcf940c-4d23-4944-9948-d3dea5ba99f6\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-s9nqq"
Oct 31 12:55:37.450467 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:37.450440    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebcf940c-4d23-4944-9948-d3dea5ba99f6-credential-socket\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-s9nqq\" (UID: \"ebcf940c-4d23-4944-9948-d3dea5ba99f6\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-s9nqq"
Oct 31 12:55:37.450873 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:37.450549    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebcf940c-4d23-4944-9948-d3dea5ba99f6-istio-podinfo\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-s9nqq\" (UID: \"ebcf940c-4d23-4944-9948-d3dea5ba99f6\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-s9nqq"
Oct 31 12:55:37.450873 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:37.450586    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebcf940c-4d23-4944-9948-d3dea5ba99f6-workload-certs\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-s9nqq\" (UID: \"ebcf940c-4d23-4944-9948-d3dea5ba99f6\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-s9nqq"
Oct 31 12:55:37.450873 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:37.450605    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebcf940c-4d23-4944-9948-d3dea5ba99f6-istiod-ca-cert\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-s9nqq\" (UID: \"ebcf940c-4d23-4944-9948-d3dea5ba99f6\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-s9nqq"
Oct 31 12:55:37.450873 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:37.450752    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebcf940c-4d23-4944-9948-d3dea5ba99f6-istio-token\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-s9nqq\" (UID: \"ebcf940c-4d23-4944-9948-d3dea5ba99f6\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-s9nqq"
Oct 31 12:55:37.450873 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:37.450783    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlg6q\" (UniqueName: \"kubernetes.io/projected/ebcf940c-4d23-4944-9948-d3dea5ba99f6-kube-api-access-zlg6q\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-s9nqq\" (UID: \"ebcf940c-4d23-4944-9948-d3dea5ba99f6\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-s9nqq"
Oct 31 12:55:37.450873 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:37.450821    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebcf940c-4d23-4944-9948-d3dea5ba99f6-istio-envoy\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-s9nqq\" (UID: \"ebcf940c-4d23-4944-9948-d3dea5ba99f6\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-s9nqq"
Oct 31 12:55:37.451136 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:37.450874    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebcf940c-4d23-4944-9948-d3dea5ba99f6-istio-data\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-s9nqq\" (UID: \"ebcf940c-4d23-4944-9948-d3dea5ba99f6\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-s9nqq"
Oct 31 12:55:37.552341 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:37.552308    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebcf940c-4d23-4944-9948-d3dea5ba99f6-credential-socket\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-s9nqq\" (UID: \"ebcf940c-4d23-4944-9948-d3dea5ba99f6\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-s9nqq"
Oct 31 12:55:37.552552 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:37.552351    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebcf940c-4d23-4944-9948-d3dea5ba99f6-istio-podinfo\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-s9nqq\" (UID: \"ebcf940c-4d23-4944-9948-d3dea5ba99f6\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-s9nqq"
Oct 31 12:55:37.552552 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:37.552375    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebcf940c-4d23-4944-9948-d3dea5ba99f6-workload-certs\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-s9nqq\" (UID: \"ebcf940c-4d23-4944-9948-d3dea5ba99f6\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-s9nqq"
Oct 31 12:55:37.552552 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:37.552393    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebcf940c-4d23-4944-9948-d3dea5ba99f6-istiod-ca-cert\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-s9nqq\" (UID: \"ebcf940c-4d23-4944-9948-d3dea5ba99f6\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-s9nqq"
Oct 31 12:55:37.552552 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:37.552446    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebcf940c-4d23-4944-9948-d3dea5ba99f6-istio-token\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-s9nqq\" (UID: \"ebcf940c-4d23-4944-9948-d3dea5ba99f6\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-s9nqq"
Oct 31 12:55:37.552552 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:37.552464    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlg6q\" (UniqueName: \"kubernetes.io/projected/ebcf940c-4d23-4944-9948-d3dea5ba99f6-kube-api-access-zlg6q\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-s9nqq\" (UID: \"ebcf940c-4d23-4944-9948-d3dea5ba99f6\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-s9nqq"
Oct 31 12:55:37.552552 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:37.552506    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebcf940c-4d23-4944-9948-d3dea5ba99f6-istio-envoy\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-s9nqq\" (UID: \"ebcf940c-4d23-4944-9948-d3dea5ba99f6\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-s9nqq"
Oct 31 12:55:37.552552 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:37.552538    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebcf940c-4d23-4944-9948-d3dea5ba99f6-istio-data\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-s9nqq\" (UID: \"ebcf940c-4d23-4944-9948-d3dea5ba99f6\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-s9nqq"
Oct 31 12:55:37.552968 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:37.552611    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebcf940c-4d23-4944-9948-d3dea5ba99f6-workload-socket\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-s9nqq\" (UID: \"ebcf940c-4d23-4944-9948-d3dea5ba99f6\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-s9nqq"
Oct 31 12:55:37.552968 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:37.552783    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebcf940c-4d23-4944-9948-d3dea5ba99f6-credential-socket\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-s9nqq\" (UID: \"ebcf940c-4d23-4944-9948-d3dea5ba99f6\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-s9nqq"
Oct 31 12:55:37.552968 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:37.552896    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebcf940c-4d23-4944-9948-d3dea5ba99f6-workload-socket\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-s9nqq\" (UID: \"ebcf940c-4d23-4944-9948-d3dea5ba99f6\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-s9nqq"
Oct 31 12:55:37.553405 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:37.553375    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebcf940c-4d23-4944-9948-d3dea5ba99f6-workload-certs\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-s9nqq\" (UID: \"ebcf940c-4d23-4944-9948-d3dea5ba99f6\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-s9nqq"
Oct 31 12:55:37.553768 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:37.553737    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebcf940c-4d23-4944-9948-d3dea5ba99f6-istio-data\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-s9nqq\" (UID: \"ebcf940c-4d23-4944-9948-d3dea5ba99f6\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-s9nqq"
Oct 31 12:55:37.553914 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:37.553830    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebcf940c-4d23-4944-9948-d3dea5ba99f6-istiod-ca-cert\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-s9nqq\" (UID: \"ebcf940c-4d23-4944-9948-d3dea5ba99f6\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-s9nqq"
Oct 31 12:55:37.561312 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:37.561286    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebcf940c-4d23-4944-9948-d3dea5ba99f6-istio-podinfo\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-s9nqq\" (UID: \"ebcf940c-4d23-4944-9948-d3dea5ba99f6\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-s9nqq"
Oct 31 12:55:37.561710 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:37.561688    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebcf940c-4d23-4944-9948-d3dea5ba99f6-istio-envoy\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-s9nqq\" (UID: \"ebcf940c-4d23-4944-9948-d3dea5ba99f6\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-s9nqq"
Oct 31 12:55:37.585238 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:37.585206    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlg6q\" (UniqueName: \"kubernetes.io/projected/ebcf940c-4d23-4944-9948-d3dea5ba99f6-kube-api-access-zlg6q\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-s9nqq\" (UID: \"ebcf940c-4d23-4944-9948-d3dea5ba99f6\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-s9nqq"
Oct 31 12:55:37.589623 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:37.588971    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebcf940c-4d23-4944-9948-d3dea5ba99f6-istio-token\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-s9nqq\" (UID: \"ebcf940c-4d23-4944-9948-d3dea5ba99f6\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-s9nqq"
Oct 31 12:55:37.644261 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:37.644214    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-s9nqq"
Oct 31 12:55:37.954152 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:37.954113    5060 patch_prober.go:29] interesting pod/e2e-inf-b-fd988c-openshift-default-96bd4cd5c-g4dgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" start-of-body=
Oct 31 12:55:37.954353 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:37.954180    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-fd988c-openshift-default-96bd4cd5c-g4dgc" podUID="0c93ff68-7e63-40b2-882b-8b2aed13e982" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused"
Oct 31 12:55:38.000392 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:38.000290    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-s9nqq"]
Oct 31 12:55:38.006970 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:55:38.006922    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebcf940c_4d23_4944_9948_d3dea5ba99f6.slice/crio-1e2f6d4fe31786df00accd6aa7be1bea62c2a700341d8fd7fb362d468269bd9a WatchSource:0}: Error finding container 1e2f6d4fe31786df00accd6aa7be1bea62c2a700341d8fd7fb362d468269bd9a: Status 404 returned error can't find the container with id 1e2f6d4fe31786df00accd6aa7be1bea62c2a700341d8fd7fb362d468269bd9a
Oct 31 12:55:38.010528 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:38.010447    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:55:38.010651 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:38.010562    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:55:38.010651 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:38.010614    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:55:38.155296 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:38.155200    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-s9nqq" event={"ID":"ebcf940c-4d23-4944-9948-d3dea5ba99f6","Type":"ContainerStarted","Data":"1e2f6d4fe31786df00accd6aa7be1bea62c2a700341d8fd7fb362d468269bd9a"}
Oct 31 12:55:38.172713 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:38.171464    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-7stx7" event={"ID":"25899808-1323-425c-9359-3b56f99ad5cb","Type":"ContainerStarted","Data":"0e99ef78dd089b9c1a0dba68109675b3b4434a15d41edf16b2cd47366ecb78b9"}
Oct 31 12:55:38.338430 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:38.338392    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-fd988c-openshift-default-96bd4cd5c-g4dgc"
Oct 31 12:55:38.416298 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:38.415845    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-7stx7"
Oct 31 12:55:38.416298 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:38.415978    5060 patch_prober.go:29] interesting pod/e2e-inf-a-d086f3-openshift-default-5c7884ccc-7stx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body=
Oct 31 12:55:38.416298 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:38.416050    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-7stx7" podUID="25899808-1323-425c-9359-3b56f99ad5cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused"
Oct 31 12:55:38.953869 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:38.953834    5060 patch_prober.go:29] interesting pod/e2e-inf-b-fd988c-openshift-default-96bd4cd5c-g4dgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" start-of-body=
Oct 31 12:55:38.954087 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:38.953883    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-fd988c-openshift-default-96bd4cd5c-g4dgc" podUID="0c93ff68-7e63-40b2-882b-8b2aed13e982" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused"
Oct 31 12:55:38.981510 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:38.981112    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-7stx7"
Oct 31 12:55:39.126913 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:39.126878    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f6d17e0-1d51-41b8-a52c-a386a8fdcc17" path="/var/lib/kubelet/pods/3f6d17e0-1d51-41b8-a52c-a386a8fdcc17/volumes"
Oct 31 12:55:39.235636 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:39.235550    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-s9nqq" event={"ID":"ebcf940c-4d23-4944-9948-d3dea5ba99f6","Type":"ContainerStarted","Data":"36995cc9e7366968dd91f78614b082c5381b377037cf4d2be4495625365cb598"}
Oct 31 12:55:39.281516 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:39.279544    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-s9nqq" podStartSLOduration=2.279524361 podStartE2EDuration="2.279524361s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:55:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:55:39.274308366 +0000 UTC m=+1702.377376104" watchObservedRunningTime="2026-10-31 12:55:39.279524361 +0000 UTC m=+1702.382592075"
Oct 31 12:55:39.414943 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:39.414902    5060 patch_prober.go:29] interesting pod/e2e-inf-a-d086f3-openshift-default-5c7884ccc-7stx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body=
Oct 31 12:55:39.415231 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:39.415201    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-7stx7" podUID="25899808-1323-425c-9359-3b56f99ad5cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused"
Oct 31 12:55:39.656245 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:39.656210    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-s9nqq"
Oct 31 12:55:39.657386 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:39.657304    5060 patch_prober.go:29] interesting pod/e2e-inf-a-d086f3-openshift-default-5c7884ccc-s9nqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.245:15021/healthz/ready\": dial tcp 10.132.0.245:15021: connect: connection refused" start-of-body=
Oct 31 12:55:39.657620 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:39.657595    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-s9nqq" podUID="ebcf940c-4d23-4944-9948-d3dea5ba99f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.245:15021/healthz/ready\": dial tcp 10.132.0.245:15021: connect: connection refused"
Oct 31 12:55:40.002574 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:40.001636    5060 patch_prober.go:29] interesting pod/e2e-inf-b-fd988c-openshift-default-96bd4cd5c-g4dgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" start-of-body=
Oct 31 12:55:40.002574 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:40.001703    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-fd988c-openshift-default-96bd4cd5c-g4dgc" podUID="0c93ff68-7e63-40b2-882b-8b2aed13e982" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused"
Oct 31 12:55:40.420581 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:40.416928    5060 patch_prober.go:29] interesting pod/e2e-inf-a-d086f3-openshift-default-5c7884ccc-7stx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body=
Oct 31 12:55:40.420581 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:40.416983    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-7stx7" podUID="25899808-1323-425c-9359-3b56f99ad5cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused"
Oct 31 12:55:40.646690 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:40.644849    5060 patch_prober.go:29] interesting pod/e2e-inf-a-d086f3-openshift-default-5c7884ccc-s9nqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.245:15021/healthz/ready\": dial tcp 10.132.0.245:15021: connect: connection refused" start-of-body=
Oct 31 12:55:40.646690 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:40.644914    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-s9nqq" podUID="ebcf940c-4d23-4944-9948-d3dea5ba99f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.245:15021/healthz/ready\": dial tcp 10.132.0.245:15021: connect: connection refused"
Oct 31 12:55:40.965422 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:40.965000    5060 patch_prober.go:29] interesting pod/e2e-inf-b-fd988c-openshift-default-96bd4cd5c-g4dgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" start-of-body=
Oct 31 12:55:40.965422 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:40.965061    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-fd988c-openshift-default-96bd4cd5c-g4dgc" podUID="0c93ff68-7e63-40b2-882b-8b2aed13e982" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused"
Oct 31 12:55:41.430517 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:41.423648    5060 patch_prober.go:29] interesting pod/e2e-inf-a-d086f3-openshift-default-5c7884ccc-7stx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body=
Oct 31 12:55:41.430517 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:41.423715    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-7stx7" podUID="25899808-1323-425c-9359-3b56f99ad5cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused"
Oct 31 12:55:41.647515 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:41.647231    5060 patch_prober.go:29] interesting pod/e2e-inf-a-d086f3-openshift-default-5c7884ccc-s9nqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.245:15021/healthz/ready\": dial tcp 10.132.0.245:15021: connect: connection refused" start-of-body=
Oct 31 12:55:41.647515 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:41.647287    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-s9nqq" podUID="ebcf940c-4d23-4944-9948-d3dea5ba99f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.245:15021/healthz/ready\": dial tcp 10.132.0.245:15021: connect: connection refused"
Oct 31 12:55:41.958508 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:41.958040    5060 patch_prober.go:29] interesting pod/e2e-inf-b-fd988c-openshift-default-96bd4cd5c-g4dgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" start-of-body=
Oct 31 12:55:41.958508 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:41.958105    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-fd988c-openshift-default-96bd4cd5c-g4dgc" podUID="0c93ff68-7e63-40b2-882b-8b2aed13e982" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused"
Oct 31 12:55:42.069513 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:42.053901    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8300aa74-49c5l"]
Oct 31 12:55:42.085524 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:42.083229    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8300aa74-49c5l"]
Oct 31 12:55:42.438534 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:42.437696    5060 patch_prober.go:29] interesting pod/e2e-inf-a-d086f3-openshift-default-5c7884ccc-7stx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body=
Oct 31 12:55:42.438534 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:42.437801    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-7stx7" podUID="25899808-1323-425c-9359-3b56f99ad5cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused"
Oct 31 12:55:42.690567 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:42.646918    5060 patch_prober.go:29] interesting pod/e2e-inf-a-d086f3-openshift-default-5c7884ccc-s9nqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.245:15021/healthz/ready\": dial tcp 10.132.0.245:15021: connect: connection refused" start-of-body=
Oct 31 12:55:42.690567 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:42.646982    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-s9nqq" podUID="ebcf940c-4d23-4944-9948-d3dea5ba99f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.245:15021/healthz/ready\": dial tcp 10.132.0.245:15021: connect: connection refused"
Oct 31 12:55:42.776722 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:42.768733    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-8s2sm"]
Oct 31 12:55:42.776722 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:42.769020    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-8s2sm" podUID="ae6f8c0f-87f7-4ad7-85e4-4d69e29ded3b" containerName="istio-proxy" containerID="cri-o://2354abb3c178a09bbcf0d1781171038f45e135388fb4eaf3ae392c7cd677ef7a" gracePeriod=30
Oct 31 12:55:42.851431 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:42.848565    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-r8l72"]
Oct 31 12:55:42.851431 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:42.848841    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-r8l72" podUID="c765a671-664f-40d2-9b62-67c746c397d0" containerName="istio-proxy" containerID="cri-o://6ff27078d42d2f5f7081688c661ea5d22f9d7670552eddd47fd71785f09598e4" gracePeriod=30
Oct 31 12:55:42.989564 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:42.988890    5060 patch_prober.go:29] interesting pod/e2e-inf-b-fd988c-openshift-default-96bd4cd5c-g4dgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" start-of-body=
Oct 31 12:55:42.989564 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:42.988953    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-fd988c-openshift-default-96bd4cd5c-g4dgc" podUID="0c93ff68-7e63-40b2-882b-8b2aed13e982" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused"
Oct 31 12:55:43.346541 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:43.345566    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d15391ff-6f37-40b8-bdfa-42410ca1074a" path="/var/lib/kubelet/pods/d15391ff-6f37-40b8-bdfa-42410ca1074a/volumes"
Oct 31 12:55:43.346978 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:43.346875    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-9b62a0-openshift-default-6f648468b7-g7tch"]
Oct 31 12:55:43.354447 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:43.354421    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-9b62a0-openshift-default-6f648468b7-g7tch"]
Oct 31 12:55:43.362666 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:43.354584    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9b62a0-openshift-default-6f648468b7-g7tch"
Oct 31 12:55:43.384780 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:43.384746    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-9b62a0-openshift-default-dockercfg-l2m24\""
Oct 31 12:55:43.423616 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:43.423581    5060 patch_prober.go:29] interesting pod/e2e-inf-a-d086f3-openshift-default-5c7884ccc-7stx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body=
Oct 31 12:55:43.423878 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:43.423849    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-7stx7" podUID="25899808-1323-425c-9359-3b56f99ad5cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused"
Oct 31 12:55:43.446005 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:43.445963    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ac598e5-c4d3-4aa5-ac25-3bcbd0b3c968-istio-token\") pod \"e2e-inf-a-9b62a0-openshift-default-6f648468b7-g7tch\" (UID: \"6ac598e5-c4d3-4aa5-ac25-3bcbd0b3c968\") " pod="openshift-ingress/e2e-inf-a-9b62a0-openshift-default-6f648468b7-g7tch"
Oct 31 12:55:43.446451 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:43.446121    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ac598e5-c4d3-4aa5-ac25-3bcbd0b3c968-workload-certs\") pod \"e2e-inf-a-9b62a0-openshift-default-6f648468b7-g7tch\" (UID: \"6ac598e5-c4d3-4aa5-ac25-3bcbd0b3c968\") " pod="openshift-ingress/e2e-inf-a-9b62a0-openshift-default-6f648468b7-g7tch"
Oct 31 12:55:43.446451 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:43.446167    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ac598e5-c4d3-4aa5-ac25-3bcbd0b3c968-workload-socket\") pod \"e2e-inf-a-9b62a0-openshift-default-6f648468b7-g7tch\" (UID: \"6ac598e5-c4d3-4aa5-ac25-3bcbd0b3c968\") " pod="openshift-ingress/e2e-inf-a-9b62a0-openshift-default-6f648468b7-g7tch"
Oct 31 12:55:43.446451 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:43.446233    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ac598e5-c4d3-4aa5-ac25-3bcbd0b3c968-istiod-ca-cert\") pod \"e2e-inf-a-9b62a0-openshift-default-6f648468b7-g7tch\" (UID: \"6ac598e5-c4d3-4aa5-ac25-3bcbd0b3c968\") " pod="openshift-ingress/e2e-inf-a-9b62a0-openshift-default-6f648468b7-g7tch"
Oct 31 12:55:43.446451 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:43.446372    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ac598e5-c4d3-4aa5-ac25-3bcbd0b3c968-istio-podinfo\") pod \"e2e-inf-a-9b62a0-openshift-default-6f648468b7-g7tch\" (UID: \"6ac598e5-c4d3-4aa5-ac25-3bcbd0b3c968\") " pod="openshift-ingress/e2e-inf-a-9b62a0-openshift-default-6f648468b7-g7tch"
Oct 31 12:55:43.446451 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:43.446401    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ac598e5-c4d3-4aa5-ac25-3bcbd0b3c968-istio-data\") pod \"e2e-inf-a-9b62a0-openshift-default-6f648468b7-g7tch\" (UID: \"6ac598e5-c4d3-4aa5-ac25-3bcbd0b3c968\") " pod="openshift-ingress/e2e-inf-a-9b62a0-openshift-default-6f648468b7-g7tch"
Oct 31 12:55:43.446451 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:43.446437    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ac598e5-c4d3-4aa5-ac25-3bcbd0b3c968-istio-envoy\") pod \"e2e-inf-a-9b62a0-openshift-default-6f648468b7-g7tch\" (UID: \"6ac598e5-c4d3-4aa5-ac25-3bcbd0b3c968\") " pod="openshift-ingress/e2e-inf-a-9b62a0-openshift-default-6f648468b7-g7tch"
Oct 31 12:55:43.446724 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:43.446511    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8765f\" (UniqueName: \"kubernetes.io/projected/6ac598e5-c4d3-4aa5-ac25-3bcbd0b3c968-kube-api-access-8765f\") pod \"e2e-inf-a-9b62a0-openshift-default-6f648468b7-g7tch\" (UID: \"6ac598e5-c4d3-4aa5-ac25-3bcbd0b3c968\") " pod="openshift-ingress/e2e-inf-a-9b62a0-openshift-default-6f648468b7-g7tch"
Oct 31 12:55:43.446724 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:43.446545    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ac598e5-c4d3-4aa5-ac25-3bcbd0b3c968-credential-socket\") pod \"e2e-inf-a-9b62a0-openshift-default-6f648468b7-g7tch\" (UID: \"6ac598e5-c4d3-4aa5-ac25-3bcbd0b3c968\") " pod="openshift-ingress/e2e-inf-a-9b62a0-openshift-default-6f648468b7-g7tch"
Oct 31 12:55:43.547513 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:43.547382    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ac598e5-c4d3-4aa5-ac25-3bcbd0b3c968-istio-token\") pod \"e2e-inf-a-9b62a0-openshift-default-6f648468b7-g7tch\" (UID: \"6ac598e5-c4d3-4aa5-ac25-3bcbd0b3c968\") " pod="openshift-ingress/e2e-inf-a-9b62a0-openshift-default-6f648468b7-g7tch"
Oct 31 12:55:43.547513 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:43.547458    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ac598e5-c4d3-4aa5-ac25-3bcbd0b3c968-workload-certs\") pod \"e2e-inf-a-9b62a0-openshift-default-6f648468b7-g7tch\" (UID: \"6ac598e5-c4d3-4aa5-ac25-3bcbd0b3c968\") " pod="openshift-ingress/e2e-inf-a-9b62a0-openshift-default-6f648468b7-g7tch"
Oct 31 12:55:43.547513 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:43.547478    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ac598e5-c4d3-4aa5-ac25-3bcbd0b3c968-workload-socket\") pod \"e2e-inf-a-9b62a0-openshift-default-6f648468b7-g7tch\" (UID: \"6ac598e5-c4d3-4aa5-ac25-3bcbd0b3c968\") " pod="openshift-ingress/e2e-inf-a-9b62a0-openshift-default-6f648468b7-g7tch"
Oct 31 12:55:43.547896 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:43.547567    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ac598e5-c4d3-4aa5-ac25-3bcbd0b3c968-istiod-ca-cert\") pod \"e2e-inf-a-9b62a0-openshift-default-6f648468b7-g7tch\" (UID: \"6ac598e5-c4d3-4aa5-ac25-3bcbd0b3c968\") " pod="openshift-ingress/e2e-inf-a-9b62a0-openshift-default-6f648468b7-g7tch"
Oct 31 12:55:43.547896 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:43.547621    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ac598e5-c4d3-4aa5-ac25-3bcbd0b3c968-istio-podinfo\") pod \"e2e-inf-a-9b62a0-openshift-default-6f648468b7-g7tch\" (UID: \"6ac598e5-c4d3-4aa5-ac25-3bcbd0b3c968\") " pod="openshift-ingress/e2e-inf-a-9b62a0-openshift-default-6f648468b7-g7tch"
Oct 31 12:55:43.547896 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:43.547637    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ac598e5-c4d3-4aa5-ac25-3bcbd0b3c968-istio-data\") pod \"e2e-inf-a-9b62a0-openshift-default-6f648468b7-g7tch\" (UID: \"6ac598e5-c4d3-4aa5-ac25-3bcbd0b3c968\") " pod="openshift-ingress/e2e-inf-a-9b62a0-openshift-default-6f648468b7-g7tch"
Oct 31 12:55:43.547896 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:43.547662    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ac598e5-c4d3-4aa5-ac25-3bcbd0b3c968-istio-envoy\") pod \"e2e-inf-a-9b62a0-openshift-default-6f648468b7-g7tch\" (UID: \"6ac598e5-c4d3-4aa5-ac25-3bcbd0b3c968\") " pod="openshift-ingress/e2e-inf-a-9b62a0-openshift-default-6f648468b7-g7tch"
Oct 31 12:55:43.547896 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:43.547703    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8765f\" (UniqueName: \"kubernetes.io/projected/6ac598e5-c4d3-4aa5-ac25-3bcbd0b3c968-kube-api-access-8765f\") pod \"e2e-inf-a-9b62a0-openshift-default-6f648468b7-g7tch\" (UID: \"6ac598e5-c4d3-4aa5-ac25-3bcbd0b3c968\") " pod="openshift-ingress/e2e-inf-a-9b62a0-openshift-default-6f648468b7-g7tch"
Oct 31 12:55:43.547896 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:43.547728    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ac598e5-c4d3-4aa5-ac25-3bcbd0b3c968-credential-socket\") pod \"e2e-inf-a-9b62a0-openshift-default-6f648468b7-g7tch\" (UID: \"6ac598e5-c4d3-4aa5-ac25-3bcbd0b3c968\") " pod="openshift-ingress/e2e-inf-a-9b62a0-openshift-default-6f648468b7-g7tch"
Oct 31 12:55:43.549329 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:43.548187    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ac598e5-c4d3-4aa5-ac25-3bcbd0b3c968-credential-socket\") pod \"e2e-inf-a-9b62a0-openshift-default-6f648468b7-g7tch\" (UID: \"6ac598e5-c4d3-4aa5-ac25-3bcbd0b3c968\") " pod="openshift-ingress/e2e-inf-a-9b62a0-openshift-default-6f648468b7-g7tch"
Oct 31 12:55:43.549329 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:43.549239    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ac598e5-c4d3-4aa5-ac25-3bcbd0b3c968-workload-certs\") pod \"e2e-inf-a-9b62a0-openshift-default-6f648468b7-g7tch\" (UID: \"6ac598e5-c4d3-4aa5-ac25-3bcbd0b3c968\") " pod="openshift-ingress/e2e-inf-a-9b62a0-openshift-default-6f648468b7-g7tch"
Oct 31 12:55:43.549462 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:43.549432    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ac598e5-c4d3-4aa5-ac25-3bcbd0b3c968-workload-socket\") pod \"e2e-inf-a-9b62a0-openshift-default-6f648468b7-g7tch\" (UID: \"6ac598e5-c4d3-4aa5-ac25-3bcbd0b3c968\") " pod="openshift-ingress/e2e-inf-a-9b62a0-openshift-default-6f648468b7-g7tch"
Oct 31 12:55:43.549910 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:43.549887    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ac598e5-c4d3-4aa5-ac25-3bcbd0b3c968-istiod-ca-cert\") pod \"e2e-inf-a-9b62a0-openshift-default-6f648468b7-g7tch\" (UID: \"6ac598e5-c4d3-4aa5-ac25-3bcbd0b3c968\") " pod="openshift-ingress/e2e-inf-a-9b62a0-openshift-default-6f648468b7-g7tch"
Oct 31 12:55:43.550655 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:43.550626    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ac598e5-c4d3-4aa5-ac25-3bcbd0b3c968-istio-data\") pod \"e2e-inf-a-9b62a0-openshift-default-6f648468b7-g7tch\" (UID: \"6ac598e5-c4d3-4aa5-ac25-3bcbd0b3c968\") " pod="openshift-ingress/e2e-inf-a-9b62a0-openshift-default-6f648468b7-g7tch"
Oct 31 12:55:43.552660 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:43.552586    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ac598e5-c4d3-4aa5-ac25-3bcbd0b3c968-istio-envoy\") pod \"e2e-inf-a-9b62a0-openshift-default-6f648468b7-g7tch\" (UID: \"6ac598e5-c4d3-4aa5-ac25-3bcbd0b3c968\") " pod="openshift-ingress/e2e-inf-a-9b62a0-openshift-default-6f648468b7-g7tch"
Oct 31 12:55:43.552781 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:43.552726    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ac598e5-c4d3-4aa5-ac25-3bcbd0b3c968-istio-podinfo\") pod \"e2e-inf-a-9b62a0-openshift-default-6f648468b7-g7tch\" (UID: \"6ac598e5-c4d3-4aa5-ac25-3bcbd0b3c968\") " pod="openshift-ingress/e2e-inf-a-9b62a0-openshift-default-6f648468b7-g7tch"
Oct 31 12:55:43.572514 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:43.568908    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ac598e5-c4d3-4aa5-ac25-3bcbd0b3c968-istio-token\") pod \"e2e-inf-a-9b62a0-openshift-default-6f648468b7-g7tch\" (UID: \"6ac598e5-c4d3-4aa5-ac25-3bcbd0b3c968\") " pod="openshift-ingress/e2e-inf-a-9b62a0-openshift-default-6f648468b7-g7tch"
Oct 31 12:55:43.587525 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:43.585847    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8765f\" (UniqueName: \"kubernetes.io/projected/6ac598e5-c4d3-4aa5-ac25-3bcbd0b3c968-kube-api-access-8765f\") pod \"e2e-inf-a-9b62a0-openshift-default-6f648468b7-g7tch\" (UID: \"6ac598e5-c4d3-4aa5-ac25-3bcbd0b3c968\") " pod="openshift-ingress/e2e-inf-a-9b62a0-openshift-default-6f648468b7-g7tch"
Oct 31 12:55:43.653575 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:43.650301    5060 patch_prober.go:29] interesting pod/e2e-inf-a-d086f3-openshift-default-5c7884ccc-s9nqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.245:15021/healthz/ready\": dial tcp 10.132.0.245:15021: connect: connection refused" start-of-body=
Oct 31 12:55:43.653575 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:43.650360    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-s9nqq" podUID="ebcf940c-4d23-4944-9948-d3dea5ba99f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.245:15021/healthz/ready\": dial tcp 10.132.0.245:15021: connect: connection refused"
Oct 31 12:55:43.741746 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:43.741375    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9b62a0-openshift-default-6f648468b7-g7tch"
Oct 31 12:55:44.274947 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:44.274819    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-fd988c-openshift-default-96bd4cd5c-g4dgc"
Oct 31 12:55:44.451512 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:44.431098    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-fd988c-openshift-default-96bd4cd5c-g4dgc"
Oct 31 12:55:44.451512 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:44.431222    5060 patch_prober.go:29] interesting pod/e2e-inf-a-d086f3-openshift-default-5c7884ccc-7stx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body=
Oct 31 12:55:44.451512 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:44.431265    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-7stx7" podUID="25899808-1323-425c-9359-3b56f99ad5cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused"
Oct 31 12:55:44.735516 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:44.668285    5060 patch_prober.go:29] interesting pod/e2e-inf-a-d086f3-openshift-default-5c7884ccc-s9nqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.245:15021/healthz/ready\": dial tcp 10.132.0.245:15021: connect: connection refused" start-of-body=
Oct 31 12:55:44.735516 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:44.668343    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-s9nqq" podUID="ebcf940c-4d23-4944-9948-d3dea5ba99f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.245:15021/healthz/ready\": dial tcp 10.132.0.245:15021: connect: connection refused"
Oct 31 12:55:44.740372 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:44.740318    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:55:44.741069 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:44.741039    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:55:44.741261 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:44.741241    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:55:44.767635 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:44.767448    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-9b62a0-openshift-default-6f648468b7-g7tch"]
Oct 31 12:55:45.032795 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:45.032529    5060 patch_prober.go:29] interesting pod/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-r8l72 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 12:55:45.032795 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:45.032594    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-r8l72" podUID="c765a671-664f-40d2-9b62-67c746c397d0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 12:55:45.033374 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:45.033344    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9b62a0-openshift-default-6f648468b7-g7tch" event={"ID":"6ac598e5-c4d3-4aa5-ac25-3bcbd0b3c968","Type":"ContainerStarted","Data":"deb7ea752fa8096606ad6acf96d5af9d1d6c2537f0a2e84b1a4919889acb4306"}
Oct 31 12:55:45.414176 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:45.414069    5060 patch_prober.go:29] interesting pod/e2e-inf-a-d086f3-openshift-default-5c7884ccc-7stx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body=
Oct 31 12:55:45.414176 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:45.414126    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-7stx7" podUID="25899808-1323-425c-9359-3b56f99ad5cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused"
Oct 31 12:55:45.678574 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:45.654625    5060 patch_prober.go:29] interesting pod/e2e-inf-a-d086f3-openshift-default-5c7884ccc-s9nqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.245:15021/healthz/ready\": dial tcp 10.132.0.245:15021: connect: connection refused" start-of-body=
Oct 31 12:55:45.678574 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:45.654690    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-s9nqq" podUID="ebcf940c-4d23-4944-9948-d3dea5ba99f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.245:15021/healthz/ready\": dial tcp 10.132.0.245:15021: connect: connection refused"
Oct 31 12:55:45.903539 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:45.903378    5060 patch_prober.go:29] interesting pod/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-8s2sm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 12:55:45.903539 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:45.903456    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-8s2sm" podUID="ae6f8c0f-87f7-4ad7-85e4-4d69e29ded3b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 12:55:46.101415 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:46.100249    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9b62a0-openshift-default-6f648468b7-g7tch" event={"ID":"6ac598e5-c4d3-4aa5-ac25-3bcbd0b3c968","Type":"ContainerStarted","Data":"fd1d1f5f6aaec977ef4430afe8bb838bcc102f2b3ecd2213540edfbe34e1a1f8"}
Oct 31 12:55:46.139438 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:46.139369    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-9b62a0-openshift-default-6f648468b7-g7tch" podStartSLOduration=4.139348326 podStartE2EDuration="4.139348326s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:55:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:55:46.135206098 +0000 UTC m=+1709.238273806" watchObservedRunningTime="2026-10-31 12:55:46.139348326 +0000 UTC m=+1709.242416039"
Oct 31 12:55:46.419051 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:46.418978    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-7stx7"
Oct 31 12:55:46.420430 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:46.420411    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-7stx7"
Oct 31 12:55:46.644772 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:46.644740    5060 patch_prober.go:29] interesting pod/e2e-inf-a-d086f3-openshift-default-5c7884ccc-s9nqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.245:15021/healthz/ready\": dial tcp 10.132.0.245:15021: connect: connection refused" start-of-body=
Oct 31 12:55:46.644964 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:46.644792    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-s9nqq" podUID="ebcf940c-4d23-4944-9948-d3dea5ba99f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.245:15021/healthz/ready\": dial tcp 10.132.0.245:15021: connect: connection refused"
Oct 31 12:55:46.743399 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:46.743311    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-9b62a0-openshift-default-6f648468b7-g7tch"
Oct 31 12:55:46.744586 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:46.744562    5060 patch_prober.go:29] interesting pod/e2e-inf-a-9b62a0-openshift-default-6f648468b7-g7tch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused" start-of-body=
Oct 31 12:55:46.744690 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:46.744607    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9b62a0-openshift-default-6f648468b7-g7tch" podUID="6ac598e5-c4d3-4aa5-ac25-3bcbd0b3c968" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused"
Oct 31 12:55:47.649562 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:47.649524    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-s9nqq"
Oct 31 12:55:47.649980 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:47.649964    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-s9nqq"
Oct 31 12:55:47.650743 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:47.650714    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-s9nqq"
Oct 31 12:55:47.743823 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:47.743787    5060 patch_prober.go:29] interesting pod/e2e-inf-a-9b62a0-openshift-default-6f648468b7-g7tch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused" start-of-body=
Oct 31 12:55:47.744276 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:47.743865    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9b62a0-openshift-default-6f648468b7-g7tch" podUID="6ac598e5-c4d3-4aa5-ac25-3bcbd0b3c968" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused"
Oct 31 12:55:48.128582 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:48.128538    5060 generic.go:356] "Generic (PLEG): container finished" podID="ae6f8c0f-87f7-4ad7-85e4-4d69e29ded3b" containerID="2354abb3c178a09bbcf0d1781171038f45e135388fb4eaf3ae392c7cd677ef7a" exitCode=0
Oct 31 12:55:48.128842 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:48.128821    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-8s2sm" event={"ID":"ae6f8c0f-87f7-4ad7-85e4-4d69e29ded3b","Type":"ContainerDied","Data":"2354abb3c178a09bbcf0d1781171038f45e135388fb4eaf3ae392c7cd677ef7a"}
Oct 31 12:55:48.130588 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:48.130553    5060 generic.go:356] "Generic (PLEG): container finished" podID="c765a671-664f-40d2-9b62-67c746c397d0" containerID="6ff27078d42d2f5f7081688c661ea5d22f9d7670552eddd47fd71785f09598e4" exitCode=0
Oct 31 12:55:48.132778 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:48.132548    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-r8l72" event={"ID":"c765a671-664f-40d2-9b62-67c746c397d0","Type":"ContainerDied","Data":"6ff27078d42d2f5f7081688c661ea5d22f9d7670552eddd47fd71785f09598e4"}
Oct 31 12:55:48.303444 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:48.303415    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-r8l72"
Oct 31 12:55:48.391313 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:48.389004    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c765a671-664f-40d2-9b62-67c746c397d0-istio-token\" (UniqueName: \"kubernetes.io/projected/c765a671-664f-40d2-9b62-67c746c397d0-istio-token\") pod \"c765a671-664f-40d2-9b62-67c746c397d0\" (UID: \"c765a671-664f-40d2-9b62-67c746c397d0\") "
Oct 31 12:55:48.391313 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:48.389112    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c765a671-664f-40d2-9b62-67c746c397d0-kube-api-access-b5nj7\" (UniqueName: \"kubernetes.io/projected/c765a671-664f-40d2-9b62-67c746c397d0-kube-api-access-b5nj7\") pod \"c765a671-664f-40d2-9b62-67c746c397d0\" (UID: \"c765a671-664f-40d2-9b62-67c746c397d0\") "
Oct 31 12:55:48.391313 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:48.389155    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c765a671-664f-40d2-9b62-67c746c397d0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c765a671-664f-40d2-9b62-67c746c397d0-istio-envoy\") pod \"c765a671-664f-40d2-9b62-67c746c397d0\" (UID: \"c765a671-664f-40d2-9b62-67c746c397d0\") "
Oct 31 12:55:48.391313 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:48.389205    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c765a671-664f-40d2-9b62-67c746c397d0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c765a671-664f-40d2-9b62-67c746c397d0-istiod-ca-cert\") pod \"c765a671-664f-40d2-9b62-67c746c397d0\" (UID: \"c765a671-664f-40d2-9b62-67c746c397d0\") "
Oct 31 12:55:48.391313 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:48.389260    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c765a671-664f-40d2-9b62-67c746c397d0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c765a671-664f-40d2-9b62-67c746c397d0-credential-socket\") pod \"c765a671-664f-40d2-9b62-67c746c397d0\" (UID: \"c765a671-664f-40d2-9b62-67c746c397d0\") "
Oct 31 12:55:48.391313 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:48.389359    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c765a671-664f-40d2-9b62-67c746c397d0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c765a671-664f-40d2-9b62-67c746c397d0-workload-certs\") pod \"c765a671-664f-40d2-9b62-67c746c397d0\" (UID: \"c765a671-664f-40d2-9b62-67c746c397d0\") "
Oct 31 12:55:48.391313 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:48.389424    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c765a671-664f-40d2-9b62-67c746c397d0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c765a671-664f-40d2-9b62-67c746c397d0-istio-data\") pod \"c765a671-664f-40d2-9b62-67c746c397d0\" (UID: \"c765a671-664f-40d2-9b62-67c746c397d0\") "
Oct 31 12:55:48.391313 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:48.389601    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c765a671-664f-40d2-9b62-67c746c397d0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c765a671-664f-40d2-9b62-67c746c397d0-workload-socket\") pod \"c765a671-664f-40d2-9b62-67c746c397d0\" (UID: \"c765a671-664f-40d2-9b62-67c746c397d0\") "
Oct 31 12:55:48.391313 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:48.389648    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c765a671-664f-40d2-9b62-67c746c397d0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c765a671-664f-40d2-9b62-67c746c397d0-istio-podinfo\") pod \"c765a671-664f-40d2-9b62-67c746c397d0\" (UID: \"c765a671-664f-40d2-9b62-67c746c397d0\") "
Oct 31 12:55:48.391313 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:48.389920    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c765a671-664f-40d2-9b62-67c746c397d0-istiod-ca-cert" pod "c765a671-664f-40d2-9b62-67c746c397d0" (UID: "c765a671-664f-40d2-9b62-67c746c397d0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:55:48.391313 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:48.390357    5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c765a671-664f-40d2-9b62-67c746c397d0-istiod-ca-cert\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:55:48.392133 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:48.392011    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c765a671-664f-40d2-9b62-67c746c397d0-workload-certs" pod "c765a671-664f-40d2-9b62-67c746c397d0" (UID: "c765a671-664f-40d2-9b62-67c746c397d0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:55:48.392223 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:48.392193    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c765a671-664f-40d2-9b62-67c746c397d0-credential-socket" pod "c765a671-664f-40d2-9b62-67c746c397d0" (UID: "c765a671-664f-40d2-9b62-67c746c397d0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:55:48.392561 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:48.392537    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c765a671-664f-40d2-9b62-67c746c397d0-istio-data" pod "c765a671-664f-40d2-9b62-67c746c397d0" (UID: "c765a671-664f-40d2-9b62-67c746c397d0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:55:48.392863 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:48.392839    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c765a671-664f-40d2-9b62-67c746c397d0-workload-socket" pod "c765a671-664f-40d2-9b62-67c746c397d0" (UID: "c765a671-664f-40d2-9b62-67c746c397d0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:55:48.406242 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:48.406193    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c765a671-664f-40d2-9b62-67c746c397d0-istio-envoy" pod "c765a671-664f-40d2-9b62-67c746c397d0" (UID: "c765a671-664f-40d2-9b62-67c746c397d0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:55:48.408857 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:48.408825    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c765a671-664f-40d2-9b62-67c746c397d0-kube-api-access-b5nj7" pod "c765a671-664f-40d2-9b62-67c746c397d0" (UID: "c765a671-664f-40d2-9b62-67c746c397d0"). InnerVolumeSpecName "kube-api-access-b5nj7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:55:48.409595 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:48.409555    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c765a671-664f-40d2-9b62-67c746c397d0-istio-token" pod "c765a671-664f-40d2-9b62-67c746c397d0" (UID: "c765a671-664f-40d2-9b62-67c746c397d0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:55:48.410925 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:48.410894    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c765a671-664f-40d2-9b62-67c746c397d0-istio-podinfo" pod "c765a671-664f-40d2-9b62-67c746c397d0" (UID: "c765a671-664f-40d2-9b62-67c746c397d0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 12:55:48.504155 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:48.504068    5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c765a671-664f-40d2-9b62-67c746c397d0-istio-token\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:55:48.504155 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:48.504111    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5nj7\" (UniqueName: \"kubernetes.io/projected/c765a671-664f-40d2-9b62-67c746c397d0-kube-api-access-b5nj7\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:55:48.504155 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:48.504127    5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c765a671-664f-40d2-9b62-67c746c397d0-istio-envoy\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:55:48.504155 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:48.504142    5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c765a671-664f-40d2-9b62-67c746c397d0-credential-socket\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:55:48.504517 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:48.504160    5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c765a671-664f-40d2-9b62-67c746c397d0-workload-certs\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:55:48.504517 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:48.504173    5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c765a671-664f-40d2-9b62-67c746c397d0-istio-data\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:55:48.504517 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:48.504188    5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c765a671-664f-40d2-9b62-67c746c397d0-workload-socket\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:55:48.504517 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:48.504202    5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c765a671-664f-40d2-9b62-67c746c397d0-istio-podinfo\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:55:48.602444 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:48.602411    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-8s2sm"
Oct 31 12:55:48.718993 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:48.718949    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae6f8c0f-87f7-4ad7-85e4-4d69e29ded3b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae6f8c0f-87f7-4ad7-85e4-4d69e29ded3b-workload-socket\") pod \"ae6f8c0f-87f7-4ad7-85e4-4d69e29ded3b\" (UID: \"ae6f8c0f-87f7-4ad7-85e4-4d69e29ded3b\") "
Oct 31 12:55:48.719341 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:48.719320    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae6f8c0f-87f7-4ad7-85e4-4d69e29ded3b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae6f8c0f-87f7-4ad7-85e4-4d69e29ded3b-istio-envoy\") pod \"ae6f8c0f-87f7-4ad7-85e4-4d69e29ded3b\" (UID: \"ae6f8c0f-87f7-4ad7-85e4-4d69e29ded3b\") "
Oct 31 12:55:48.719787 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:48.719760    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae6f8c0f-87f7-4ad7-85e4-4d69e29ded3b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae6f8c0f-87f7-4ad7-85e4-4d69e29ded3b-credential-socket\") pod \"ae6f8c0f-87f7-4ad7-85e4-4d69e29ded3b\" (UID: \"ae6f8c0f-87f7-4ad7-85e4-4d69e29ded3b\") "
Oct 31 12:55:48.720604 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:48.720311    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae6f8c0f-87f7-4ad7-85e4-4d69e29ded3b-credential-socket" pod "ae6f8c0f-87f7-4ad7-85e4-4d69e29ded3b" (UID: "ae6f8c0f-87f7-4ad7-85e4-4d69e29ded3b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:55:48.720604 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:48.720455    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae6f8c0f-87f7-4ad7-85e4-4d69e29ded3b-workload-socket" pod "ae6f8c0f-87f7-4ad7-85e4-4d69e29ded3b" (UID: "ae6f8c0f-87f7-4ad7-85e4-4d69e29ded3b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:55:48.720842 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:48.720818    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae6f8c0f-87f7-4ad7-85e4-4d69e29ded3b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae6f8c0f-87f7-4ad7-85e4-4d69e29ded3b-istio-data\") pod \"ae6f8c0f-87f7-4ad7-85e4-4d69e29ded3b\" (UID: \"ae6f8c0f-87f7-4ad7-85e4-4d69e29ded3b\") "
Oct 31 12:55:48.722365 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:48.722343    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ae6f8c0f-87f7-4ad7-85e4-4d69e29ded3b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae6f8c0f-87f7-4ad7-85e4-4d69e29ded3b-istio-podinfo\") pod \"ae6f8c0f-87f7-4ad7-85e4-4d69e29ded3b\" (UID: \"ae6f8c0f-87f7-4ad7-85e4-4d69e29ded3b\") "
Oct 31 12:55:48.739889 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:48.739858    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae6f8c0f-87f7-4ad7-85e4-4d69e29ded3b-istio-token\" (UniqueName: \"kubernetes.io/projected/ae6f8c0f-87f7-4ad7-85e4-4d69e29ded3b-istio-token\") pod \"ae6f8c0f-87f7-4ad7-85e4-4d69e29ded3b\" (UID: \"ae6f8c0f-87f7-4ad7-85e4-4d69e29ded3b\") "
Oct 31 12:55:48.740180 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:48.740159    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae6f8c0f-87f7-4ad7-85e4-4d69e29ded3b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae6f8c0f-87f7-4ad7-85e4-4d69e29ded3b-workload-certs\") pod \"ae6f8c0f-87f7-4ad7-85e4-4d69e29ded3b\" (UID: \"ae6f8c0f-87f7-4ad7-85e4-4d69e29ded3b\") "
Oct 31 12:55:48.740448 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:48.721186    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae6f8c0f-87f7-4ad7-85e4-4d69e29ded3b-istio-data" pod "ae6f8c0f-87f7-4ad7-85e4-4d69e29ded3b" (UID: "ae6f8c0f-87f7-4ad7-85e4-4d69e29ded3b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:55:48.740713 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:48.740473    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae6f8c0f-87f7-4ad7-85e4-4d69e29ded3b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae6f8c0f-87f7-4ad7-85e4-4d69e29ded3b-istiod-ca-cert\") pod \"ae6f8c0f-87f7-4ad7-85e4-4d69e29ded3b\" (UID: \"ae6f8c0f-87f7-4ad7-85e4-4d69e29ded3b\") "
Oct 31 12:55:48.740713 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:48.740550    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae6f8c0f-87f7-4ad7-85e4-4d69e29ded3b-kube-api-access-zhhd8\" (UniqueName: \"kubernetes.io/projected/ae6f8c0f-87f7-4ad7-85e4-4d69e29ded3b-kube-api-access-zhhd8\") pod \"ae6f8c0f-87f7-4ad7-85e4-4d69e29ded3b\" (UID: \"ae6f8c0f-87f7-4ad7-85e4-4d69e29ded3b\") "
Oct 31 12:55:48.741700 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:48.741136    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae6f8c0f-87f7-4ad7-85e4-4d69e29ded3b-workload-certs" pod "ae6f8c0f-87f7-4ad7-85e4-4d69e29ded3b" (UID: "ae6f8c0f-87f7-4ad7-85e4-4d69e29ded3b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:55:48.741700 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:48.741306    5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae6f8c0f-87f7-4ad7-85e4-4d69e29ded3b-workload-certs\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:55:48.741700 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:48.741332    5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae6f8c0f-87f7-4ad7-85e4-4d69e29ded3b-workload-socket\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:55:48.741700 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:48.741348    5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae6f8c0f-87f7-4ad7-85e4-4d69e29ded3b-credential-socket\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:55:48.741700 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:48.741364    5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae6f8c0f-87f7-4ad7-85e4-4d69e29ded3b-istio-data\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:55:48.742281 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:48.742236    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae6f8c0f-87f7-4ad7-85e4-4d69e29ded3b-istiod-ca-cert" pod "ae6f8c0f-87f7-4ad7-85e4-4d69e29ded3b" (UID: "ae6f8c0f-87f7-4ad7-85e4-4d69e29ded3b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:55:48.758678 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:48.755343    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ae6f8c0f-87f7-4ad7-85e4-4d69e29ded3b-istio-podinfo" pod "ae6f8c0f-87f7-4ad7-85e4-4d69e29ded3b" (UID: "ae6f8c0f-87f7-4ad7-85e4-4d69e29ded3b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 12:55:48.768322 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:48.766597    5060 patch_prober.go:29] interesting pod/e2e-inf-a-9b62a0-openshift-default-6f648468b7-g7tch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused" start-of-body=
Oct 31 12:55:48.768322 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:48.766656    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9b62a0-openshift-default-6f648468b7-g7tch" podUID="6ac598e5-c4d3-4aa5-ac25-3bcbd0b3c968" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused"
Oct 31 12:55:48.797901 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:48.796762    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae6f8c0f-87f7-4ad7-85e4-4d69e29ded3b-istio-envoy" pod "ae6f8c0f-87f7-4ad7-85e4-4d69e29ded3b" (UID: "ae6f8c0f-87f7-4ad7-85e4-4d69e29ded3b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:55:48.797901 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:48.796857    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae6f8c0f-87f7-4ad7-85e4-4d69e29ded3b-kube-api-access-zhhd8" pod "ae6f8c0f-87f7-4ad7-85e4-4d69e29ded3b" (UID: "ae6f8c0f-87f7-4ad7-85e4-4d69e29ded3b"). InnerVolumeSpecName "kube-api-access-zhhd8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:55:48.797901 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:48.797209    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae6f8c0f-87f7-4ad7-85e4-4d69e29ded3b-istio-token" pod "ae6f8c0f-87f7-4ad7-85e4-4d69e29ded3b" (UID: "ae6f8c0f-87f7-4ad7-85e4-4d69e29ded3b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:55:48.843193 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:48.843165    5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae6f8c0f-87f7-4ad7-85e4-4d69e29ded3b-istiod-ca-cert\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:55:48.843437 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:48.843420    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhhd8\" (UniqueName: \"kubernetes.io/projected/ae6f8c0f-87f7-4ad7-85e4-4d69e29ded3b-kube-api-access-zhhd8\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:55:48.843534 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:48.843444    5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae6f8c0f-87f7-4ad7-85e4-4d69e29ded3b-istio-envoy\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:55:48.843534 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:48.843458    5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae6f8c0f-87f7-4ad7-85e4-4d69e29ded3b-istio-podinfo\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:55:48.843534 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:48.843472    5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae6f8c0f-87f7-4ad7-85e4-4d69e29ded3b-istio-token\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:55:49.140791 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:49.140760    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-r8l72"
Oct 31 12:55:49.140952 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:49.140785    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-r8l72" event={"ID":"c765a671-664f-40d2-9b62-67c746c397d0","Type":"ContainerDied","Data":"c436d512f8983338a9fc0597b49cd83845e5eb80ad9e212191b73c8ec5a7fceb"}
Oct 31 12:55:49.140952 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:49.140849    5060 scope.go:122] "RemoveContainer" containerID="6ff27078d42d2f5f7081688c661ea5d22f9d7670552eddd47fd71785f09598e4"
Oct 31 12:55:49.142669 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:49.142645    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-8s2sm"
Oct 31 12:55:49.142805 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:49.142676    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-8s2sm" event={"ID":"ae6f8c0f-87f7-4ad7-85e4-4d69e29ded3b","Type":"ContainerDied","Data":"ba533a83cdc7eaff6671dd3b3e7aa8b2085c529d22d56f5aa016db60c45d7e93"}
Oct 31 12:55:49.185516 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:49.185398    5060 scope.go:122] "RemoveContainer" containerID="2354abb3c178a09bbcf0d1781171038f45e135388fb4eaf3ae392c7cd677ef7a"
Oct 31 12:55:49.189169 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:49.189138    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-r8l72"]
Oct 31 12:55:49.194584 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:49.194560    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-r8l72"]
Oct 31 12:55:49.205580 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:49.205551    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-8s2sm"]
Oct 31 12:55:49.209918 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:49.209895    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-8s2sm"]
Oct 31 12:55:49.744068 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:49.744028    5060 patch_prober.go:29] interesting pod/e2e-inf-a-9b62a0-openshift-default-6f648468b7-g7tch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused" start-of-body=
Oct 31 12:55:49.744254 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:49.744085    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9b62a0-openshift-default-6f648468b7-g7tch" podUID="6ac598e5-c4d3-4aa5-ac25-3bcbd0b3c968" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused"
Oct 31 12:55:50.744282 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:50.744238    5060 patch_prober.go:29] interesting pod/e2e-inf-a-9b62a0-openshift-default-6f648468b7-g7tch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused" start-of-body=
Oct 31 12:55:50.744736 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:50.744323    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9b62a0-openshift-default-6f648468b7-g7tch" podUID="6ac598e5-c4d3-4aa5-ac25-3bcbd0b3c968" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused"
Oct 31 12:55:51.050169 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:51.050135    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae6f8c0f-87f7-4ad7-85e4-4d69e29ded3b" path="/var/lib/kubelet/pods/ae6f8c0f-87f7-4ad7-85e4-4d69e29ded3b/volumes"
Oct 31 12:55:51.050666 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:51.050650    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c765a671-664f-40d2-9b62-67c746c397d0" path="/var/lib/kubelet/pods/c765a671-664f-40d2-9b62-67c746c397d0/volumes"
Oct 31 12:55:51.743827 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:51.743792    5060 patch_prober.go:29] interesting pod/e2e-inf-a-9b62a0-openshift-default-6f648468b7-g7tch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused" start-of-body=
Oct 31 12:55:51.744018 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:51.743847    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9b62a0-openshift-default-6f648468b7-g7tch" podUID="6ac598e5-c4d3-4aa5-ac25-3bcbd0b3c968" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused"
Oct 31 12:55:52.748903 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:52.748869    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-9b62a0-openshift-default-6f648468b7-g7tch"
Oct 31 12:55:52.749551 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:52.749526    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-9b62a0-openshift-default-6f648468b7-g7tch"
Oct 31 12:55:52.750129 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:52.750107    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-9b62a0-openshift-default-6f648468b7-g7tch"
Oct 31 12:55:53.053157 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:53.053119    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-015701-jdsj9"]
Oct 31 12:55:53.132529 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:53.126805    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-015701-jdsj9"]
Oct 31 12:55:55.050180 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:55:55.050136    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8fcc724c-21ea-4b0e-b540-6293ae707096" path="/var/lib/kubelet/pods/8fcc724c-21ea-4b0e-b540-6293ae707096/volumes"
Oct 31 12:55:59.926247 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:55:59.926203    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7phb" containerName="extractor"
Oct 31 12:56:03.240446 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:03.240407    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-015701-openshift-default-86f6dfd4b7-5lsm7"]
Oct 31 12:56:03.240986 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:03.240685    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-015701-openshift-default-86f6dfd4b7-5lsm7" podUID="ce0f9360-f1f6-462c-b4b1-1e0a2232fa7d" containerName="istio-proxy" containerID="cri-o://b4fafc52af107f28606032b647220c133808eda2b08c36ce98b6cdf4487a3fe1" gracePeriod=30
Oct 31 12:56:03.647615 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:03.647572    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ad1af7-tjf99"]
Oct 31 12:56:03.648730 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:03.648706    5060 cpu_manager.go:414] "containerMap: removing container" podUID="ae6f8c0f-87f7-4ad7-85e4-4d69e29ded3b" containerName="istio-proxy"
Oct 31 12:56:03.648833 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:03.648734    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae6f8c0f-87f7-4ad7-85e4-4d69e29ded3b" containerName="istio-proxy"
Oct 31 12:56:03.648833 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:03.648771    5060 cpu_manager.go:414] "containerMap: removing container" podUID="c765a671-664f-40d2-9b62-67c746c397d0" containerName="istio-proxy"
Oct 31 12:56:03.648833 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:03.648779    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c765a671-664f-40d2-9b62-67c746c397d0" containerName="istio-proxy"
Oct 31 12:56:03.648976 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:03.648948    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae6f8c0f-87f7-4ad7-85e4-4d69e29ded3b" containerName="istio-proxy"
Oct 31 12:56:03.648976 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:03.648968    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="c765a671-664f-40d2-9b62-67c746c397d0" containerName="istio-proxy"
Oct 31 12:56:03.662542 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:03.655198    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ad1af7-tjf99"
Oct 31 12:56:03.676694 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:03.676662    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7wsmb\""
Oct 31 12:56:03.678872 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:03.677629    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 12:56:03.702622 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:03.702582    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ad1af7-tjf99"]
Oct 31 12:56:03.752504 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:03.752450    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcb987e6-efb9-49c6-a8f5-b628a57c2cc4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-ad1af7-tjf99\" (UID: \"dcb987e6-efb9-49c6-a8f5-b628a57c2cc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ad1af7-tjf99"
Oct 31 12:56:03.752676 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:03.752570    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lt5q\" (UniqueName: \"kubernetes.io/projected/dcb987e6-efb9-49c6-a8f5-b628a57c2cc4-kube-api-access-6lt5q\") pod \"maas-api-revoke-keys-e2e-inf-b-ad1af7-tjf99\" (UID: \"dcb987e6-efb9-49c6-a8f5-b628a57c2cc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ad1af7-tjf99"
Oct 31 12:56:03.752763 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:03.752737    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcb987e6-efb9-49c6-a8f5-b628a57c2cc4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-ad1af7-tjf99\" (UID: \"dcb987e6-efb9-49c6-a8f5-b628a57c2cc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ad1af7-tjf99"
Oct 31 12:56:03.855950 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:03.855915    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcb987e6-efb9-49c6-a8f5-b628a57c2cc4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-ad1af7-tjf99\" (UID: \"dcb987e6-efb9-49c6-a8f5-b628a57c2cc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ad1af7-tjf99"
Oct 31 12:56:03.875655 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:03.875578    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcb987e6-efb9-49c6-a8f5-b628a57c2cc4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-ad1af7-tjf99\" (UID: \"dcb987e6-efb9-49c6-a8f5-b628a57c2cc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ad1af7-tjf99"
Oct 31 12:56:03.876028 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:03.876005    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lt5q\" (UniqueName: \"kubernetes.io/projected/dcb987e6-efb9-49c6-a8f5-b628a57c2cc4-kube-api-access-6lt5q\") pod \"maas-api-revoke-keys-e2e-inf-b-ad1af7-tjf99\" (UID: \"dcb987e6-efb9-49c6-a8f5-b628a57c2cc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ad1af7-tjf99"
Oct 31 12:56:03.876402 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:03.876374    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcb987e6-efb9-49c6-a8f5-b628a57c2cc4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-ad1af7-tjf99\" (UID: \"dcb987e6-efb9-49c6-a8f5-b628a57c2cc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ad1af7-tjf99"
Oct 31 12:56:03.887740 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:03.887700    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcb987e6-efb9-49c6-a8f5-b628a57c2cc4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-ad1af7-tjf99\" (UID: \"dcb987e6-efb9-49c6-a8f5-b628a57c2cc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ad1af7-tjf99"
Oct 31 12:56:03.890299 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:03.890270    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lt5q\" (UniqueName: \"kubernetes.io/projected/dcb987e6-efb9-49c6-a8f5-b628a57c2cc4-kube-api-access-6lt5q\") pod \"maas-api-revoke-keys-e2e-inf-b-ad1af7-tjf99\" (UID: \"dcb987e6-efb9-49c6-a8f5-b628a57c2cc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ad1af7-tjf99"
Oct 31 12:56:04.009206 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:04.009114    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ad1af7-tjf99"
Oct 31 12:56:04.394233 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:04.394198    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ad1af7-tjf99"]
Oct 31 12:56:04.515299 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:04.515251    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ad1af7-tjf99" event={"ID":"dcb987e6-efb9-49c6-a8f5-b628a57c2cc4","Type":"ContainerStarted","Data":"495dc13c884fd37de82d618c7613b773166881e26a72da206048fb67892b2d0b"}
Oct 31 12:56:05.526450 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:05.526415    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ad1af7-tjf99" event={"ID":"dcb987e6-efb9-49c6-a8f5-b628a57c2cc4","Type":"ContainerStarted","Data":"6a3d83b482969b68a66f6be84990624ebc3f7d126b78b096e032372d8922a38f"}
Oct 31 12:56:05.568510 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:05.566578    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ad1af7-tjf99" podStartSLOduration=2.566560256 podStartE2EDuration="2.566560256s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:56:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:56:05.564777787 +0000 UTC m=+1728.667845500" watchObservedRunningTime="2026-10-31 12:56:05.566560256 +0000 UTC m=+1728.669627969"
Oct 31 12:56:06.015793 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:06.015691    5060 patch_prober.go:29] interesting pod/e2e-inf-a-015701-openshift-default-86f6dfd4b7-5lsm7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 12:56:06.016111 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:06.016076    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-015701-openshift-default-86f6dfd4b7-5lsm7" podUID="ce0f9360-f1f6-462c-b4b1-1e0a2232fa7d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 12:56:06.537873 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:06.537840    5060 generic.go:356] "Generic (PLEG): container finished" podID="dcb987e6-efb9-49c6-a8f5-b628a57c2cc4" containerID="6a3d83b482969b68a66f6be84990624ebc3f7d126b78b096e032372d8922a38f" exitCode=0
Oct 31 12:56:06.538292 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:06.537928    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ad1af7-tjf99" event={"ID":"dcb987e6-efb9-49c6-a8f5-b628a57c2cc4","Type":"ContainerDied","Data":"6a3d83b482969b68a66f6be84990624ebc3f7d126b78b096e032372d8922a38f"}
Oct 31 12:56:07.753536 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:07.753462    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ad1af7-tjf99"
Oct 31 12:56:07.898292 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:07.898144    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dcb987e6-efb9-49c6-a8f5-b628a57c2cc4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcb987e6-efb9-49c6-a8f5-b628a57c2cc4-maas-api-openshift-ca\") pod \"dcb987e6-efb9-49c6-a8f5-b628a57c2cc4\" (UID: \"dcb987e6-efb9-49c6-a8f5-b628a57c2cc4\") "
Oct 31 12:56:07.898292 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:07.898232    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcb987e6-efb9-49c6-a8f5-b628a57c2cc4-kube-api-access-6lt5q\" (UniqueName: \"kubernetes.io/projected/dcb987e6-efb9-49c6-a8f5-b628a57c2cc4-kube-api-access-6lt5q\") pod \"dcb987e6-efb9-49c6-a8f5-b628a57c2cc4\" (UID: \"dcb987e6-efb9-49c6-a8f5-b628a57c2cc4\") "
Oct 31 12:56:07.898970 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:07.898715    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dcb987e6-efb9-49c6-a8f5-b628a57c2cc4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcb987e6-efb9-49c6-a8f5-b628a57c2cc4-maas-api-xks-ca\") pod \"dcb987e6-efb9-49c6-a8f5-b628a57c2cc4\" (UID: \"dcb987e6-efb9-49c6-a8f5-b628a57c2cc4\") "
Oct 31 12:56:07.906322 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:07.899072    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcb987e6-efb9-49c6-a8f5-b628a57c2cc4-maas-api-openshift-ca" pod "dcb987e6-efb9-49c6-a8f5-b628a57c2cc4" (UID: "dcb987e6-efb9-49c6-a8f5-b628a57c2cc4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:56:07.906322 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:07.901318    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dcb987e6-efb9-49c6-a8f5-b628a57c2cc4-maas-api-xks-ca" pod "dcb987e6-efb9-49c6-a8f5-b628a57c2cc4" (UID: "dcb987e6-efb9-49c6-a8f5-b628a57c2cc4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:56:07.906322 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:07.901793    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcb987e6-efb9-49c6-a8f5-b628a57c2cc4-kube-api-access-6lt5q" pod "dcb987e6-efb9-49c6-a8f5-b628a57c2cc4" (UID: "dcb987e6-efb9-49c6-a8f5-b628a57c2cc4"). InnerVolumeSpecName "kube-api-access-6lt5q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:56:07.906322 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:07.906075    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcb987e6-efb9-49c6-a8f5-b628a57c2cc4-maas-api-xks-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:56:07.906322 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:07.906103    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcb987e6-efb9-49c6-a8f5-b628a57c2cc4-maas-api-openshift-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:56:07.906322 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:07.906118    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6lt5q\" (UniqueName: \"kubernetes.io/projected/dcb987e6-efb9-49c6-a8f5-b628a57c2cc4-kube-api-access-6lt5q\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:56:08.533550 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:08.531169    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-015701-openshift-default-86f6dfd4b7-5lsm7"
Oct 31 12:56:08.584999 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:08.584915    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ad1af7-tjf99" event={"ID":"dcb987e6-efb9-49c6-a8f5-b628a57c2cc4","Type":"ContainerDied","Data":"495dc13c884fd37de82d618c7613b773166881e26a72da206048fb67892b2d0b"}
Oct 31 12:56:08.584999 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:08.584983    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="495dc13c884fd37de82d618c7613b773166881e26a72da206048fb67892b2d0b"
Oct 31 12:56:08.585188 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:08.585057    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ad1af7-tjf99"
Oct 31 12:56:08.590725 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:08.590694    5060 generic.go:356] "Generic (PLEG): container finished" podID="ce0f9360-f1f6-462c-b4b1-1e0a2232fa7d" containerID="b4fafc52af107f28606032b647220c133808eda2b08c36ce98b6cdf4487a3fe1" exitCode=0
Oct 31 12:56:08.590842 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:08.590771    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-015701-openshift-default-86f6dfd4b7-5lsm7" event={"ID":"ce0f9360-f1f6-462c-b4b1-1e0a2232fa7d","Type":"ContainerDied","Data":"b4fafc52af107f28606032b647220c133808eda2b08c36ce98b6cdf4487a3fe1"}
Oct 31 12:56:08.590842 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:08.590804    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-015701-openshift-default-86f6dfd4b7-5lsm7" event={"ID":"ce0f9360-f1f6-462c-b4b1-1e0a2232fa7d","Type":"ContainerDied","Data":"454e7423ea7590145bc4e0ebc6d8da13fb4b9e1e7b00ea85a683808ff2d7eb42"}
Oct 31 12:56:08.590842 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:08.590826    5060 scope.go:122] "RemoveContainer" containerID="b4fafc52af107f28606032b647220c133808eda2b08c36ce98b6cdf4487a3fe1"
Oct 31 12:56:08.590958 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:08.590946    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-015701-openshift-default-86f6dfd4b7-5lsm7"
Oct 31 12:56:08.618791 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:08.618134    5060 scope.go:122] "RemoveContainer" containerID="b4fafc52af107f28606032b647220c133808eda2b08c36ce98b6cdf4487a3fe1"
Oct 31 12:56:08.618791 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:56:08.618564    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4fafc52af107f28606032b647220c133808eda2b08c36ce98b6cdf4487a3fe1\": container with ID starting with b4fafc52af107f28606032b647220c133808eda2b08c36ce98b6cdf4487a3fe1 not found: ID does not exist" containerID="b4fafc52af107f28606032b647220c133808eda2b08c36ce98b6cdf4487a3fe1"
Oct 31 12:56:08.618791 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:08.618602    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4fafc52af107f28606032b647220c133808eda2b08c36ce98b6cdf4487a3fe1"} err="failed to get container status \"b4fafc52af107f28606032b647220c133808eda2b08c36ce98b6cdf4487a3fe1\": rpc error: code = NotFound desc = could not find container \"b4fafc52af107f28606032b647220c133808eda2b08c36ce98b6cdf4487a3fe1\": container with ID starting with b4fafc52af107f28606032b647220c133808eda2b08c36ce98b6cdf4487a3fe1 not found: ID does not exist"
Oct 31 12:56:08.719205 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:08.719171    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce0f9360-f1f6-462c-b4b1-1e0a2232fa7d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce0f9360-f1f6-462c-b4b1-1e0a2232fa7d-credential-socket\") pod \"ce0f9360-f1f6-462c-b4b1-1e0a2232fa7d\" (UID: \"ce0f9360-f1f6-462c-b4b1-1e0a2232fa7d\") "
Oct 31 12:56:08.719391 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:08.719302    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce0f9360-f1f6-462c-b4b1-1e0a2232fa7d-kube-api-access-s2w6b\" (UniqueName: \"kubernetes.io/projected/ce0f9360-f1f6-462c-b4b1-1e0a2232fa7d-kube-api-access-s2w6b\") pod \"ce0f9360-f1f6-462c-b4b1-1e0a2232fa7d\" (UID: \"ce0f9360-f1f6-462c-b4b1-1e0a2232fa7d\") "
Oct 31 12:56:08.719391 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:08.719372    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce0f9360-f1f6-462c-b4b1-1e0a2232fa7d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce0f9360-f1f6-462c-b4b1-1e0a2232fa7d-istiod-ca-cert\") pod \"ce0f9360-f1f6-462c-b4b1-1e0a2232fa7d\" (UID: \"ce0f9360-f1f6-462c-b4b1-1e0a2232fa7d\") "
Oct 31 12:56:08.719552 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:08.719427    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce0f9360-f1f6-462c-b4b1-1e0a2232fa7d-istio-token\" (UniqueName: \"kubernetes.io/projected/ce0f9360-f1f6-462c-b4b1-1e0a2232fa7d-istio-token\") pod \"ce0f9360-f1f6-462c-b4b1-1e0a2232fa7d\" (UID: \"ce0f9360-f1f6-462c-b4b1-1e0a2232fa7d\") "
Oct 31 12:56:08.719552 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:08.719467    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce0f9360-f1f6-462c-b4b1-1e0a2232fa7d-credential-socket" pod "ce0f9360-f1f6-462c-b4b1-1e0a2232fa7d" (UID: "ce0f9360-f1f6-462c-b4b1-1e0a2232fa7d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:56:08.719552 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:08.719519    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce0f9360-f1f6-462c-b4b1-1e0a2232fa7d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce0f9360-f1f6-462c-b4b1-1e0a2232fa7d-istio-envoy\") pod \"ce0f9360-f1f6-462c-b4b1-1e0a2232fa7d\" (UID: \"ce0f9360-f1f6-462c-b4b1-1e0a2232fa7d\") "
Oct 31 12:56:08.719722 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:08.719555    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce0f9360-f1f6-462c-b4b1-1e0a2232fa7d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce0f9360-f1f6-462c-b4b1-1e0a2232fa7d-workload-certs\") pod \"ce0f9360-f1f6-462c-b4b1-1e0a2232fa7d\" (UID: \"ce0f9360-f1f6-462c-b4b1-1e0a2232fa7d\") "
Oct 31 12:56:08.719722 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:08.719639    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce0f9360-f1f6-462c-b4b1-1e0a2232fa7d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce0f9360-f1f6-462c-b4b1-1e0a2232fa7d-workload-socket\") pod \"ce0f9360-f1f6-462c-b4b1-1e0a2232fa7d\" (UID: \"ce0f9360-f1f6-462c-b4b1-1e0a2232fa7d\") "
Oct 31 12:56:08.719722 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:08.719712    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce0f9360-f1f6-462c-b4b1-1e0a2232fa7d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce0f9360-f1f6-462c-b4b1-1e0a2232fa7d-istio-data\") pod \"ce0f9360-f1f6-462c-b4b1-1e0a2232fa7d\" (UID: \"ce0f9360-f1f6-462c-b4b1-1e0a2232fa7d\") "
Oct 31 12:56:08.719916 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:08.719802    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ce0f9360-f1f6-462c-b4b1-1e0a2232fa7d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce0f9360-f1f6-462c-b4b1-1e0a2232fa7d-istio-podinfo\") pod \"ce0f9360-f1f6-462c-b4b1-1e0a2232fa7d\" (UID: \"ce0f9360-f1f6-462c-b4b1-1e0a2232fa7d\") "
Oct 31 12:56:08.720003 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:08.719975    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce0f9360-f1f6-462c-b4b1-1e0a2232fa7d-workload-certs" pod "ce0f9360-f1f6-462c-b4b1-1e0a2232fa7d" (UID: "ce0f9360-f1f6-462c-b4b1-1e0a2232fa7d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:56:08.720220 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:08.720200    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce0f9360-f1f6-462c-b4b1-1e0a2232fa7d-istio-data" pod "ce0f9360-f1f6-462c-b4b1-1e0a2232fa7d" (UID: "ce0f9360-f1f6-462c-b4b1-1e0a2232fa7d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:56:08.720461 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:08.720138    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce0f9360-f1f6-462c-b4b1-1e0a2232fa7d-workload-socket" pod "ce0f9360-f1f6-462c-b4b1-1e0a2232fa7d" (UID: "ce0f9360-f1f6-462c-b4b1-1e0a2232fa7d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:56:08.720626 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:08.720604    5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce0f9360-f1f6-462c-b4b1-1e0a2232fa7d-istio-data\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:56:08.720694 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:08.720625    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce0f9360-f1f6-462c-b4b1-1e0a2232fa7d-istiod-ca-cert" pod "ce0f9360-f1f6-462c-b4b1-1e0a2232fa7d" (UID: "ce0f9360-f1f6-462c-b4b1-1e0a2232fa7d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:56:08.720694 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:08.720633    5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce0f9360-f1f6-462c-b4b1-1e0a2232fa7d-credential-socket\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:56:08.720694 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:08.720672    5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce0f9360-f1f6-462c-b4b1-1e0a2232fa7d-workload-certs\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:56:08.727997 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:08.727799    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ce0f9360-f1f6-462c-b4b1-1e0a2232fa7d-istio-podinfo" pod "ce0f9360-f1f6-462c-b4b1-1e0a2232fa7d" (UID: "ce0f9360-f1f6-462c-b4b1-1e0a2232fa7d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 12:56:08.727997 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:08.727815    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce0f9360-f1f6-462c-b4b1-1e0a2232fa7d-istio-envoy" pod "ce0f9360-f1f6-462c-b4b1-1e0a2232fa7d" (UID: "ce0f9360-f1f6-462c-b4b1-1e0a2232fa7d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:56:08.727997 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:08.727837    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce0f9360-f1f6-462c-b4b1-1e0a2232fa7d-kube-api-access-s2w6b" pod "ce0f9360-f1f6-462c-b4b1-1e0a2232fa7d" (UID: "ce0f9360-f1f6-462c-b4b1-1e0a2232fa7d"). InnerVolumeSpecName "kube-api-access-s2w6b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:56:08.727997 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:08.727874    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce0f9360-f1f6-462c-b4b1-1e0a2232fa7d-istio-token" pod "ce0f9360-f1f6-462c-b4b1-1e0a2232fa7d" (UID: "ce0f9360-f1f6-462c-b4b1-1e0a2232fa7d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:56:08.823013 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:08.822971    5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce0f9360-f1f6-462c-b4b1-1e0a2232fa7d-workload-socket\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:56:08.823013 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:08.823011    5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce0f9360-f1f6-462c-b4b1-1e0a2232fa7d-istio-podinfo\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:56:08.823426 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:08.823025    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2w6b\" (UniqueName: \"kubernetes.io/projected/ce0f9360-f1f6-462c-b4b1-1e0a2232fa7d-kube-api-access-s2w6b\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:56:08.823426 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:08.823041    5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce0f9360-f1f6-462c-b4b1-1e0a2232fa7d-istiod-ca-cert\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:56:08.823426 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:08.823056    5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce0f9360-f1f6-462c-b4b1-1e0a2232fa7d-istio-token\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:56:08.823426 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:08.823070    5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce0f9360-f1f6-462c-b4b1-1e0a2232fa7d-istio-envoy\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:56:08.997198 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:08.992613    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-015701-openshift-default-86f6dfd4b7-5lsm7"]
Oct 31 12:56:09.009742 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:09.007231    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-015701-openshift-default-86f6dfd4b7-5lsm7"]
Oct 31 12:56:09.063121 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:09.063081    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce0f9360-f1f6-462c-b4b1-1e0a2232fa7d" path="/var/lib/kubelet/pods/ce0f9360-f1f6-462c-b4b1-1e0a2232fa7d/volumes"
Oct 31 12:56:11.622669 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:11.622580    5060 generic.go:356] "Generic (PLEG): container finished" podID="b3992303-31dd-484d-b3ad-3fececdf57fa" containerID="e33caf28b00f04205fa360aaf1fadccce6cb70b25e057f52a58a026e4d2c2ea6" exitCode=0
Oct 31 12:56:11.622669 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:11.622649    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-71bc05-openshift-default-dd6bc8b77-b65mw" event={"ID":"b3992303-31dd-484d-b3ad-3fececdf57fa","Type":"ContainerDied","Data":"e33caf28b00f04205fa360aaf1fadccce6cb70b25e057f52a58a026e4d2c2ea6"}
Oct 31 12:56:11.623221 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:11.623110    5060 scope.go:122] "RemoveContainer" containerID="e33caf28b00f04205fa360aaf1fadccce6cb70b25e057f52a58a026e4d2c2ea6"
Oct 31 12:56:11.624897 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:11.624870    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:56:11.624990 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:11.624939    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:56:11.624990 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:11.624970    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:56:12.634005 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:12.633963    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-71bc05-openshift-default-dd6bc8b77-b65mw" event={"ID":"b3992303-31dd-484d-b3ad-3fececdf57fa","Type":"ContainerStarted","Data":"f3312ffbe5d00b5a088382db79f0b7a35cab76989dea919ce7a3555a8878a7a8"}
Oct 31 12:56:13.000087 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:12.999995    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-71bc05-openshift-default-dd6bc8b77-b65mw"
Oct 31 12:56:13.001327 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:13.001302    5060 patch_prober.go:29] interesting pod/e2e-inf-b-71bc05-openshift-default-dd6bc8b77-b65mw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" start-of-body=
Oct 31 12:56:13.001462 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:13.001345    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-71bc05-openshift-default-dd6bc8b77-b65mw" podUID="b3992303-31dd-484d-b3ad-3fececdf57fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused"
Oct 31 12:56:14.000128 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:13.999985    5060 patch_prober.go:29] interesting pod/e2e-inf-b-71bc05-openshift-default-dd6bc8b77-b65mw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" start-of-body=
Oct 31 12:56:14.000128 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:14.000037    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-71bc05-openshift-default-dd6bc8b77-b65mw" podUID="b3992303-31dd-484d-b3ad-3fececdf57fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused"
Oct 31 12:56:15.000257 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:15.000223    5060 patch_prober.go:29] interesting pod/e2e-inf-b-71bc05-openshift-default-dd6bc8b77-b65mw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" start-of-body=
Oct 31 12:56:15.000710 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:15.000279    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-71bc05-openshift-default-dd6bc8b77-b65mw" podUID="b3992303-31dd-484d-b3ad-3fececdf57fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused"
Oct 31 12:56:15.653335 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:15.653299    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-71bc05-openshift-default-dd6bc8b77-b65mw"
Oct 31 12:56:16.000097 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:16.000008    5060 patch_prober.go:29] interesting pod/e2e-inf-b-71bc05-openshift-default-dd6bc8b77-b65mw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" start-of-body=
Oct 31 12:56:16.000097 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:16.000076    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-71bc05-openshift-default-dd6bc8b77-b65mw" podUID="b3992303-31dd-484d-b3ad-3fececdf57fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused"
Oct 31 12:56:17.000641 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:17.000604    5060 patch_prober.go:29] interesting pod/e2e-inf-b-71bc05-openshift-default-dd6bc8b77-b65mw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" start-of-body=
Oct 31 12:56:17.001297 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:17.001266    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-71bc05-openshift-default-dd6bc8b77-b65mw" podUID="b3992303-31dd-484d-b3ad-3fececdf57fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused"
Oct 31 12:56:18.000273 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:18.000232    5060 patch_prober.go:29] interesting pod/e2e-inf-b-71bc05-openshift-default-dd6bc8b77-b65mw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" start-of-body=
Oct 31 12:56:18.000477 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:18.000301    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-71bc05-openshift-default-dd6bc8b77-b65mw" podUID="b3992303-31dd-484d-b3ad-3fececdf57fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused"
Oct 31 12:56:19.005547 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:19.005506    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-71bc05-openshift-default-dd6bc8b77-b65mw"
Oct 31 12:56:19.007051 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:19.007026    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-71bc05-openshift-default-dd6bc8b77-b65mw"
Oct 31 12:56:22.334423 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:22.334387    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-nzrl9"]
Oct 31 12:56:22.335274 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:22.335248    5060 cpu_manager.go:414] "containerMap: removing container" podUID="ce0f9360-f1f6-462c-b4b1-1e0a2232fa7d" containerName="istio-proxy"
Oct 31 12:56:22.335274 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:22.335274    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce0f9360-f1f6-462c-b4b1-1e0a2232fa7d" containerName="istio-proxy"
Oct 31 12:56:22.335475 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:22.335324    5060 cpu_manager.go:414] "containerMap: removing container" podUID="dcb987e6-efb9-49c6-a8f5-b628a57c2cc4" containerName="revoke-keys"
Oct 31 12:56:22.335475 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:22.335331    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dcb987e6-efb9-49c6-a8f5-b628a57c2cc4" containerName="revoke-keys"
Oct 31 12:56:22.335475 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:22.335426    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="dcb987e6-efb9-49c6-a8f5-b628a57c2cc4" containerName="revoke-keys"
Oct 31 12:56:22.335475 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:22.335438    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce0f9360-f1f6-462c-b4b1-1e0a2232fa7d" containerName="istio-proxy"
Oct 31 12:56:22.341790 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:22.341765    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-nzrl9"
Oct 31 12:56:22.352823 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:22.352771    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-nzrl9"]
Oct 31 12:56:22.511760 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:22.511722    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1ac43d5-eb68-45aa-9e8e-268d7c6355dd-istiod-ca-cert\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-nzrl9\" (UID: \"b1ac43d5-eb68-45aa-9e8e-268d7c6355dd\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-nzrl9"
Oct 31 12:56:22.511944 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:22.511773    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1ac43d5-eb68-45aa-9e8e-268d7c6355dd-istio-envoy\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-nzrl9\" (UID: \"b1ac43d5-eb68-45aa-9e8e-268d7c6355dd\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-nzrl9"
Oct 31 12:56:22.511944 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:22.511873    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlf96\" (UniqueName: \"kubernetes.io/projected/b1ac43d5-eb68-45aa-9e8e-268d7c6355dd-kube-api-access-nlf96\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-nzrl9\" (UID: \"b1ac43d5-eb68-45aa-9e8e-268d7c6355dd\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-nzrl9"
Oct 31 12:56:22.511944 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:22.511914    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1ac43d5-eb68-45aa-9e8e-268d7c6355dd-istio-token\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-nzrl9\" (UID: \"b1ac43d5-eb68-45aa-9e8e-268d7c6355dd\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-nzrl9"
Oct 31 12:56:22.511944 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:22.511934    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1ac43d5-eb68-45aa-9e8e-268d7c6355dd-workload-certs\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-nzrl9\" (UID: \"b1ac43d5-eb68-45aa-9e8e-268d7c6355dd\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-nzrl9"
Oct 31 12:56:22.512124 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:22.512097    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1ac43d5-eb68-45aa-9e8e-268d7c6355dd-istio-data\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-nzrl9\" (UID: \"b1ac43d5-eb68-45aa-9e8e-268d7c6355dd\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-nzrl9"
Oct 31 12:56:22.512159 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:22.512139    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1ac43d5-eb68-45aa-9e8e-268d7c6355dd-credential-socket\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-nzrl9\" (UID: \"b1ac43d5-eb68-45aa-9e8e-268d7c6355dd\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-nzrl9"
Oct 31 12:56:22.512224 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:22.512213    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1ac43d5-eb68-45aa-9e8e-268d7c6355dd-workload-socket\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-nzrl9\" (UID: \"b1ac43d5-eb68-45aa-9e8e-268d7c6355dd\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-nzrl9"
Oct 31 12:56:22.512274 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:22.512263    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1ac43d5-eb68-45aa-9e8e-268d7c6355dd-istio-podinfo\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-nzrl9\" (UID: \"b1ac43d5-eb68-45aa-9e8e-268d7c6355dd\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-nzrl9"
Oct 31 12:56:22.613120 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:22.613027    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1ac43d5-eb68-45aa-9e8e-268d7c6355dd-istiod-ca-cert\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-nzrl9\" (UID: \"b1ac43d5-eb68-45aa-9e8e-268d7c6355dd\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-nzrl9"
Oct 31 12:56:22.613120 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:22.613072    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1ac43d5-eb68-45aa-9e8e-268d7c6355dd-istio-envoy\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-nzrl9\" (UID: \"b1ac43d5-eb68-45aa-9e8e-268d7c6355dd\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-nzrl9"
Oct 31 12:56:22.613120 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:22.613116    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlf96\" (UniqueName: \"kubernetes.io/projected/b1ac43d5-eb68-45aa-9e8e-268d7c6355dd-kube-api-access-nlf96\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-nzrl9\" (UID: \"b1ac43d5-eb68-45aa-9e8e-268d7c6355dd\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-nzrl9"
Oct 31 12:56:22.613439 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:22.613150    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1ac43d5-eb68-45aa-9e8e-268d7c6355dd-istio-token\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-nzrl9\" (UID: \"b1ac43d5-eb68-45aa-9e8e-268d7c6355dd\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-nzrl9"
Oct 31 12:56:22.613439 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:22.613170    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1ac43d5-eb68-45aa-9e8e-268d7c6355dd-workload-certs\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-nzrl9\" (UID: \"b1ac43d5-eb68-45aa-9e8e-268d7c6355dd\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-nzrl9"
Oct 31 12:56:22.613439 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:22.613277    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1ac43d5-eb68-45aa-9e8e-268d7c6355dd-istio-data\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-nzrl9\" (UID: \"b1ac43d5-eb68-45aa-9e8e-268d7c6355dd\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-nzrl9"
Oct 31 12:56:22.613439 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:22.613357    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1ac43d5-eb68-45aa-9e8e-268d7c6355dd-credential-socket\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-nzrl9\" (UID: \"b1ac43d5-eb68-45aa-9e8e-268d7c6355dd\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-nzrl9"
Oct 31 12:56:22.613439 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:22.613394    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1ac43d5-eb68-45aa-9e8e-268d7c6355dd-workload-socket\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-nzrl9\" (UID: \"b1ac43d5-eb68-45aa-9e8e-268d7c6355dd\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-nzrl9"
Oct 31 12:56:22.613757 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:22.613464    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1ac43d5-eb68-45aa-9e8e-268d7c6355dd-istio-podinfo\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-nzrl9\" (UID: \"b1ac43d5-eb68-45aa-9e8e-268d7c6355dd\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-nzrl9"
Oct 31 12:56:22.613757 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:22.613594    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1ac43d5-eb68-45aa-9e8e-268d7c6355dd-workload-certs\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-nzrl9\" (UID: \"b1ac43d5-eb68-45aa-9e8e-268d7c6355dd\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-nzrl9"
Oct 31 12:56:22.613757 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:22.613650    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1ac43d5-eb68-45aa-9e8e-268d7c6355dd-istio-data\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-nzrl9\" (UID: \"b1ac43d5-eb68-45aa-9e8e-268d7c6355dd\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-nzrl9"
Oct 31 12:56:22.613922 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:22.613821    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1ac43d5-eb68-45aa-9e8e-268d7c6355dd-credential-socket\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-nzrl9\" (UID: \"b1ac43d5-eb68-45aa-9e8e-268d7c6355dd\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-nzrl9"
Oct 31 12:56:22.613922 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:22.613907    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1ac43d5-eb68-45aa-9e8e-268d7c6355dd-istiod-ca-cert\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-nzrl9\" (UID: \"b1ac43d5-eb68-45aa-9e8e-268d7c6355dd\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-nzrl9"
Oct 31 12:56:22.614071 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:22.614047    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1ac43d5-eb68-45aa-9e8e-268d7c6355dd-workload-socket\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-nzrl9\" (UID: \"b1ac43d5-eb68-45aa-9e8e-268d7c6355dd\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-nzrl9"
Oct 31 12:56:22.615614 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:22.615588    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1ac43d5-eb68-45aa-9e8e-268d7c6355dd-istio-envoy\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-nzrl9\" (UID: \"b1ac43d5-eb68-45aa-9e8e-268d7c6355dd\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-nzrl9"
Oct 31 12:56:22.616212 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:22.616190    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1ac43d5-eb68-45aa-9e8e-268d7c6355dd-istio-podinfo\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-nzrl9\" (UID: \"b1ac43d5-eb68-45aa-9e8e-268d7c6355dd\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-nzrl9"
Oct 31 12:56:22.631823 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:22.631790    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1ac43d5-eb68-45aa-9e8e-268d7c6355dd-istio-token\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-nzrl9\" (UID: \"b1ac43d5-eb68-45aa-9e8e-268d7c6355dd\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-nzrl9"
Oct 31 12:56:22.632801 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:22.632770    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlf96\" (UniqueName: \"kubernetes.io/projected/b1ac43d5-eb68-45aa-9e8e-268d7c6355dd-kube-api-access-nlf96\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-nzrl9\" (UID: \"b1ac43d5-eb68-45aa-9e8e-268d7c6355dd\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-nzrl9"
Oct 31 12:56:22.670591 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:22.670563    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-nzrl9"
Oct 31 12:56:22.971367 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:56:22.971319    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1ac43d5_eb68_45aa_9e8e_268d7c6355dd.slice/crio-c3b9f8720655ae094f462bc627c2d623fb6b7393e2146cf3d9c8b5a50c206bdb WatchSource:0}: Error finding container c3b9f8720655ae094f462bc627c2d623fb6b7393e2146cf3d9c8b5a50c206bdb: Status 404 returned error can't find the container with id c3b9f8720655ae094f462bc627c2d623fb6b7393e2146cf3d9c8b5a50c206bdb
Oct 31 12:56:22.973785 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:22.973755    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-nzrl9"]
Oct 31 12:56:22.974009 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:22.973974    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:56:22.974064 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:22.974041    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:56:22.974101 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:22.974069    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:56:23.801446 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:23.801405    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-nzrl9" event={"ID":"b1ac43d5-eb68-45aa-9e8e-268d7c6355dd","Type":"ContainerStarted","Data":"8f91d4f726d41d37d7125a8084d62df37bf466c88fd28b02196b5aeff9951a2e"}
Oct 31 12:56:23.801972 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:23.801474    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-nzrl9" event={"ID":"b1ac43d5-eb68-45aa-9e8e-268d7c6355dd","Type":"ContainerStarted","Data":"c3b9f8720655ae094f462bc627c2d623fb6b7393e2146cf3d9c8b5a50c206bdb"}
Oct 31 12:56:23.835046 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:23.834985    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-nzrl9" podStartSLOduration=1.834970592 podStartE2EDuration="1.834970592s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:56:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:56:23.828122128 +0000 UTC m=+1746.931189840" watchObservedRunningTime="2026-10-31 12:56:23.834970592 +0000 UTC m=+1746.938038303"
Oct 31 12:56:24.695478 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:24.692910    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-nzrl9"
Oct 31 12:56:24.695478 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:24.693142    5060 patch_prober.go:29] interesting pod/e2e-inf-a-d086f3-openshift-default-5c7884ccc-nzrl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" start-of-body=
Oct 31 12:56:24.695478 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:24.693191    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-nzrl9" podUID="b1ac43d5-eb68-45aa-9e8e-268d7c6355dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused"
Oct 31 12:56:25.051358 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:25.046550    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ad1af7-tjf99"]
Oct 31 12:56:25.095543 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:25.095220    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ad1af7-tjf99"]
Oct 31 12:56:25.682514 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:25.674861    5060 patch_prober.go:29] interesting pod/e2e-inf-a-d086f3-openshift-default-5c7884ccc-nzrl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" start-of-body=
Oct 31 12:56:25.682514 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:25.674966    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-nzrl9" podUID="b1ac43d5-eb68-45aa-9e8e-268d7c6355dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused"
Oct 31 12:56:26.672898 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:26.672855    5060 patch_prober.go:29] interesting pod/e2e-inf-a-d086f3-openshift-default-5c7884ccc-nzrl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" start-of-body=
Oct 31 12:56:26.673476 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:26.673442    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-nzrl9" podUID="b1ac43d5-eb68-45aa-9e8e-268d7c6355dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused"
Oct 31 12:56:27.048985 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:27.048948    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dcb987e6-efb9-49c6-a8f5-b628a57c2cc4" path="/var/lib/kubelet/pods/dcb987e6-efb9-49c6-a8f5-b628a57c2cc4/volumes"
Oct 31 12:56:27.672356 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:27.671874    5060 patch_prober.go:29] interesting pod/e2e-inf-a-d086f3-openshift-default-5c7884ccc-nzrl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" start-of-body=
Oct 31 12:56:27.672356 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:27.671963    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-nzrl9" podUID="b1ac43d5-eb68-45aa-9e8e-268d7c6355dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused"
Oct 31 12:56:28.675631 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:28.675584    5060 patch_prober.go:29] interesting pod/e2e-inf-a-d086f3-openshift-default-5c7884ccc-nzrl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" start-of-body=
Oct 31 12:56:28.676262 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:28.676220    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-nzrl9" podUID="b1ac43d5-eb68-45aa-9e8e-268d7c6355dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused"
Oct 31 12:56:29.685874 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:29.685843    5060 patch_prober.go:29] interesting pod/e2e-inf-a-d086f3-openshift-default-5c7884ccc-nzrl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" start-of-body=
Oct 31 12:56:29.686351 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:29.686317    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-nzrl9" podUID="b1ac43d5-eb68-45aa-9e8e-268d7c6355dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused"
Oct 31 12:56:29.923425 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:56:29.923380    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7phb" containerName="extractor"
Oct 31 12:56:30.688718 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:30.688684    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-nzrl9"
Oct 31 12:56:30.691329 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:30.691305    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-nzrl9"
Oct 31 12:56:30.695938 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:30.695913    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-nzrl9"
Oct 31 12:56:33.233000 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:33.232962    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-vqcct"]
Oct 31 12:56:33.233518 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:33.233227    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-vqcct" podUID="b2705ec7-400e-4265-b213-5b58ba42cd4c" containerName="istio-proxy" containerID="cri-o://3975283cd17e0a73b11ef41c996088b132983657497571571be49b0e29d04ee0" gracePeriod=30
Oct 31 12:56:36.803604 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:36.803555    5060 patch_prober.go:29] interesting pod/e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-vqcct container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 12:56:36.803989 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:36.803638    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-vqcct" podUID="b2705ec7-400e-4265-b213-5b58ba42cd4c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 12:56:38.543055 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:38.543021    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-vqcct"
Oct 31 12:56:38.710754 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:38.710722    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2705ec7-400e-4265-b213-5b58ba42cd4c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2705ec7-400e-4265-b213-5b58ba42cd4c-credential-socket\") pod \"b2705ec7-400e-4265-b213-5b58ba42cd4c\" (UID: \"b2705ec7-400e-4265-b213-5b58ba42cd4c\") "
Oct 31 12:56:38.710754 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:38.710759    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2705ec7-400e-4265-b213-5b58ba42cd4c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2705ec7-400e-4265-b213-5b58ba42cd4c-workload-certs\") pod \"b2705ec7-400e-4265-b213-5b58ba42cd4c\" (UID: \"b2705ec7-400e-4265-b213-5b58ba42cd4c\") "
Oct 31 12:56:38.711013 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:38.710786    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2705ec7-400e-4265-b213-5b58ba42cd4c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2705ec7-400e-4265-b213-5b58ba42cd4c-istio-envoy\") pod \"b2705ec7-400e-4265-b213-5b58ba42cd4c\" (UID: \"b2705ec7-400e-4265-b213-5b58ba42cd4c\") "
Oct 31 12:56:38.711013 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:38.710869    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b2705ec7-400e-4265-b213-5b58ba42cd4c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2705ec7-400e-4265-b213-5b58ba42cd4c-istio-podinfo\") pod \"b2705ec7-400e-4265-b213-5b58ba42cd4c\" (UID: \"b2705ec7-400e-4265-b213-5b58ba42cd4c\") "
Oct 31 12:56:38.711013 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:38.710902    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2705ec7-400e-4265-b213-5b58ba42cd4c-istio-token\" (UniqueName: \"kubernetes.io/projected/b2705ec7-400e-4265-b213-5b58ba42cd4c-istio-token\") pod \"b2705ec7-400e-4265-b213-5b58ba42cd4c\" (UID: \"b2705ec7-400e-4265-b213-5b58ba42cd4c\") "
Oct 31 12:56:38.711013 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:38.711000    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2705ec7-400e-4265-b213-5b58ba42cd4c-kube-api-access-p4l6w\" (UniqueName: \"kubernetes.io/projected/b2705ec7-400e-4265-b213-5b58ba42cd4c-kube-api-access-p4l6w\") pod \"b2705ec7-400e-4265-b213-5b58ba42cd4c\" (UID: \"b2705ec7-400e-4265-b213-5b58ba42cd4c\") "
Oct 31 12:56:38.711250 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:38.711037    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2705ec7-400e-4265-b213-5b58ba42cd4c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2705ec7-400e-4265-b213-5b58ba42cd4c-istio-data\") pod \"b2705ec7-400e-4265-b213-5b58ba42cd4c\" (UID: \"b2705ec7-400e-4265-b213-5b58ba42cd4c\") "
Oct 31 12:56:38.711250 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:38.711060    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2705ec7-400e-4265-b213-5b58ba42cd4c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2705ec7-400e-4265-b213-5b58ba42cd4c-workload-socket\") pod \"b2705ec7-400e-4265-b213-5b58ba42cd4c\" (UID: \"b2705ec7-400e-4265-b213-5b58ba42cd4c\") "
Oct 31 12:56:38.711250 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:38.711067    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2705ec7-400e-4265-b213-5b58ba42cd4c-workload-certs" pod "b2705ec7-400e-4265-b213-5b58ba42cd4c" (UID: "b2705ec7-400e-4265-b213-5b58ba42cd4c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:56:38.711250 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:38.711168    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2705ec7-400e-4265-b213-5b58ba42cd4c-credential-socket" pod "b2705ec7-400e-4265-b213-5b58ba42cd4c" (UID: "b2705ec7-400e-4265-b213-5b58ba42cd4c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:56:38.711476 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:38.711266    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2705ec7-400e-4265-b213-5b58ba42cd4c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2705ec7-400e-4265-b213-5b58ba42cd4c-istiod-ca-cert\") pod \"b2705ec7-400e-4265-b213-5b58ba42cd4c\" (UID: \"b2705ec7-400e-4265-b213-5b58ba42cd4c\") "
Oct 31 12:56:38.711476 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:38.711397    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2705ec7-400e-4265-b213-5b58ba42cd4c-istio-data" pod "b2705ec7-400e-4265-b213-5b58ba42cd4c" (UID: "b2705ec7-400e-4265-b213-5b58ba42cd4c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:56:38.711706 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:38.711641    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2705ec7-400e-4265-b213-5b58ba42cd4c-workload-socket" pod "b2705ec7-400e-4265-b213-5b58ba42cd4c" (UID: "b2705ec7-400e-4265-b213-5b58ba42cd4c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:56:38.711916 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:38.711896    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2705ec7-400e-4265-b213-5b58ba42cd4c-istiod-ca-cert" pod "b2705ec7-400e-4265-b213-5b58ba42cd4c" (UID: "b2705ec7-400e-4265-b213-5b58ba42cd4c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:56:38.712086 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:38.712070    5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2705ec7-400e-4265-b213-5b58ba42cd4c-istiod-ca-cert\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:56:38.712086 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:38.712086    5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2705ec7-400e-4265-b213-5b58ba42cd4c-credential-socket\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:56:38.712231 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:38.712096    5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2705ec7-400e-4265-b213-5b58ba42cd4c-workload-certs\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:56:38.712231 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:38.712110    5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2705ec7-400e-4265-b213-5b58ba42cd4c-istio-data\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:56:38.712231 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:38.712119    5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2705ec7-400e-4265-b213-5b58ba42cd4c-workload-socket\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:56:38.714059 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:38.714024    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2705ec7-400e-4265-b213-5b58ba42cd4c-kube-api-access-p4l6w" pod "b2705ec7-400e-4265-b213-5b58ba42cd4c" (UID: "b2705ec7-400e-4265-b213-5b58ba42cd4c"). InnerVolumeSpecName "kube-api-access-p4l6w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:56:38.714164 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:38.714092    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2705ec7-400e-4265-b213-5b58ba42cd4c-istio-token" pod "b2705ec7-400e-4265-b213-5b58ba42cd4c" (UID: "b2705ec7-400e-4265-b213-5b58ba42cd4c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:56:38.714899 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:38.714872    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2705ec7-400e-4265-b213-5b58ba42cd4c-istio-envoy" pod "b2705ec7-400e-4265-b213-5b58ba42cd4c" (UID: "b2705ec7-400e-4265-b213-5b58ba42cd4c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:56:38.715035 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:38.715010    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b2705ec7-400e-4265-b213-5b58ba42cd4c-istio-podinfo" pod "b2705ec7-400e-4265-b213-5b58ba42cd4c" (UID: "b2705ec7-400e-4265-b213-5b58ba42cd4c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 12:56:38.813628 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:38.813593    5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2705ec7-400e-4265-b213-5b58ba42cd4c-istio-envoy\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:56:38.813628 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:38.813621    5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2705ec7-400e-4265-b213-5b58ba42cd4c-istio-podinfo\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:56:38.813628 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:38.813633    5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2705ec7-400e-4265-b213-5b58ba42cd4c-istio-token\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:56:38.813880 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:38.813642    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4l6w\" (UniqueName: \"kubernetes.io/projected/b2705ec7-400e-4265-b213-5b58ba42cd4c-kube-api-access-p4l6w\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:56:39.047041 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:39.046992    5060 generic.go:356] "Generic (PLEG): container finished" podID="b2705ec7-400e-4265-b213-5b58ba42cd4c" containerID="3975283cd17e0a73b11ef41c996088b132983657497571571be49b0e29d04ee0" exitCode=0
Oct 31 12:56:39.047225 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:39.047122    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-vqcct"
Oct 31 12:56:39.049223 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:39.049195    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-vqcct" event={"ID":"b2705ec7-400e-4265-b213-5b58ba42cd4c","Type":"ContainerDied","Data":"3975283cd17e0a73b11ef41c996088b132983657497571571be49b0e29d04ee0"}
Oct 31 12:56:39.049365 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:39.049246    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-vqcct" event={"ID":"b2705ec7-400e-4265-b213-5b58ba42cd4c","Type":"ContainerDied","Data":"9397b449c7a1cb0a61faf30b9fe3db9e6a5a7e467d08d4929cb1534380a29c64"}
Oct 31 12:56:39.049365 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:39.049278    5060 scope.go:122] "RemoveContainer" containerID="3975283cd17e0a73b11ef41c996088b132983657497571571be49b0e29d04ee0"
Oct 31 12:56:39.062634 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:39.062616    5060 scope.go:122] "RemoveContainer" containerID="3975283cd17e0a73b11ef41c996088b132983657497571571be49b0e29d04ee0"
Oct 31 12:56:39.062939 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:56:39.062919    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3975283cd17e0a73b11ef41c996088b132983657497571571be49b0e29d04ee0\": container with ID starting with 3975283cd17e0a73b11ef41c996088b132983657497571571be49b0e29d04ee0 not found: ID does not exist" containerID="3975283cd17e0a73b11ef41c996088b132983657497571571be49b0e29d04ee0"
Oct 31 12:56:39.063013 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:39.062949    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3975283cd17e0a73b11ef41c996088b132983657497571571be49b0e29d04ee0"} err="failed to get container status \"3975283cd17e0a73b11ef41c996088b132983657497571571be49b0e29d04ee0\": rpc error: code = NotFound desc = could not find container \"3975283cd17e0a73b11ef41c996088b132983657497571571be49b0e29d04ee0\": container with ID starting with 3975283cd17e0a73b11ef41c996088b132983657497571571be49b0e29d04ee0 not found: ID does not exist"
Oct 31 12:56:39.077128 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:39.077062    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-vqcct"]
Oct 31 12:56:39.081580 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:39.081556    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-vqcct"]
Oct 31 12:56:41.063332 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:41.063297    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2705ec7-400e-4265-b213-5b58ba42cd4c" path="/var/lib/kubelet/pods/b2705ec7-400e-4265-b213-5b58ba42cd4c/volumes"
Oct 31 12:56:44.047161 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:44.047120    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-fb0343-5d8d4b7fb7-dtkzd"]
Oct 31 12:56:44.072223 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:44.071779    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-fb0343-85b486979f-blth7"]
Oct 31 12:56:49.050303 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:49.050261    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-fb0343-5d8d4b7fb7-dtkzd" podUID="6a041d29-7b54-4d07-be06-2c96f8fc097c" containerName="payload-processing" containerID="cri-o://481c99373b0e0f398f783a1ef91a8e7c1944fa271ed2428a6fb1c0632dde60c2" gracePeriod=25
Oct 31 12:56:49.072938 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:49.072871    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-fb0343-85b486979f-blth7" podUID="5f8db300-b095-4b06-9fa5-1804c2a64df8" containerName="payload-pre-processing" containerID="cri-o://7e07cca37a3d92c5eee144bc04bab214a169ce484dfa655084775f7a977435bc" gracePeriod=25
Oct 31 12:56:50.904535 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:50.904451    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05c99f42-tvtf4"]
Oct 31 12:56:50.905614 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:50.905588    5060 cpu_manager.go:414] "containerMap: removing container" podUID="b2705ec7-400e-4265-b213-5b58ba42cd4c" containerName="istio-proxy"
Oct 31 12:56:50.905614 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:50.905614    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2705ec7-400e-4265-b213-5b58ba42cd4c" containerName="istio-proxy"
Oct 31 12:56:50.905824 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:50.905814    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2705ec7-400e-4265-b213-5b58ba42cd4c" containerName="istio-proxy"
Oct 31 12:56:50.912073 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:50.912049    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05c99f42-tvtf4"
Oct 31 12:56:50.934881 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:50.934849    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05c99f42-tvtf4"]
Oct 31 12:56:50.981190 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:50.981105    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j768n\" (UniqueName: \"kubernetes.io/projected/25f69674-db4a-4ce5-a441-d51b61b91d2f-kube-api-access-j768n\") pod \"maas-api-revoke-sub-05c99f42-tvtf4\" (UID: \"25f69674-db4a-4ce5-a441-d51b61b91d2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05c99f42-tvtf4"
Oct 31 12:56:50.981469 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:50.981434    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25f69674-db4a-4ce5-a441-d51b61b91d2f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-05c99f42-tvtf4\" (UID: \"25f69674-db4a-4ce5-a441-d51b61b91d2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05c99f42-tvtf4"
Oct 31 12:56:50.981665 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:50.981643    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25f69674-db4a-4ce5-a441-d51b61b91d2f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-05c99f42-tvtf4\" (UID: \"25f69674-db4a-4ce5-a441-d51b61b91d2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05c99f42-tvtf4"
Oct 31 12:56:50.996839 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:50.996815    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 12:56:50.996950 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:50.996860    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7wsmb\""
Oct 31 12:56:51.083021 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:51.082985    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25f69674-db4a-4ce5-a441-d51b61b91d2f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-05c99f42-tvtf4\" (UID: \"25f69674-db4a-4ce5-a441-d51b61b91d2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05c99f42-tvtf4"
Oct 31 12:56:51.083225 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:51.083093    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25f69674-db4a-4ce5-a441-d51b61b91d2f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-05c99f42-tvtf4\" (UID: \"25f69674-db4a-4ce5-a441-d51b61b91d2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05c99f42-tvtf4"
Oct 31 12:56:51.083225 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:51.083136    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j768n\" (UniqueName: \"kubernetes.io/projected/25f69674-db4a-4ce5-a441-d51b61b91d2f-kube-api-access-j768n\") pod \"maas-api-revoke-sub-05c99f42-tvtf4\" (UID: \"25f69674-db4a-4ce5-a441-d51b61b91d2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05c99f42-tvtf4"
Oct 31 12:56:51.083776 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:51.083738    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25f69674-db4a-4ce5-a441-d51b61b91d2f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-05c99f42-tvtf4\" (UID: \"25f69674-db4a-4ce5-a441-d51b61b91d2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05c99f42-tvtf4"
Oct 31 12:56:51.085677 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:51.085653    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25f69674-db4a-4ce5-a441-d51b61b91d2f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-05c99f42-tvtf4\" (UID: \"25f69674-db4a-4ce5-a441-d51b61b91d2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05c99f42-tvtf4"
Oct 31 12:56:51.096313 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:51.096272    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j768n\" (UniqueName: \"kubernetes.io/projected/25f69674-db4a-4ce5-a441-d51b61b91d2f-kube-api-access-j768n\") pod \"maas-api-revoke-sub-05c99f42-tvtf4\" (UID: \"25f69674-db4a-4ce5-a441-d51b61b91d2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05c99f42-tvtf4"
Oct 31 12:56:51.241197 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:51.241097    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05c99f42-tvtf4"
Oct 31 12:56:51.560404 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:51.560357    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05c99f42-tvtf4"]
Oct 31 12:56:51.565712 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:56:51.563793    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25f69674_db4a_4ce5_a441_d51b61b91d2f.slice/crio-c18490afe7c04b138121b7a36c6d8da73298d258e1525a7cbdf76181387e9be9 WatchSource:0}: Error finding container c18490afe7c04b138121b7a36c6d8da73298d258e1525a7cbdf76181387e9be9: Status 404 returned error can't find the container with id c18490afe7c04b138121b7a36c6d8da73298d258e1525a7cbdf76181387e9be9
Oct 31 12:56:52.328899 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:52.328856    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05c99f42-tvtf4" event={"ID":"25f69674-db4a-4ce5-a441-d51b61b91d2f","Type":"ContainerStarted","Data":"303bfa7ccec2421ef1e9d0125106744445fef4d0d89db08937c51343313386c1"}
Oct 31 12:56:52.329414 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:52.328924    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05c99f42-tvtf4" event={"ID":"25f69674-db4a-4ce5-a441-d51b61b91d2f","Type":"ContainerStarted","Data":"c18490afe7c04b138121b7a36c6d8da73298d258e1525a7cbdf76181387e9be9"}
Oct 31 12:56:52.351042 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:52.350994    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05c99f42-tvtf4" podStartSLOduration=2.350980087 podStartE2EDuration="2.350980087s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:56:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:56:52.346656506 +0000 UTC m=+1775.449724220" watchObservedRunningTime="2026-10-31 12:56:52.350980087 +0000 UTC m=+1775.454047799"
Oct 31 12:56:53.339749 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:53.339714    5060 generic.go:356] "Generic (PLEG): container finished" podID="25f69674-db4a-4ce5-a441-d51b61b91d2f" containerID="303bfa7ccec2421ef1e9d0125106744445fef4d0d89db08937c51343313386c1" exitCode=0
Oct 31 12:56:53.340152 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:53.339789    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05c99f42-tvtf4" event={"ID":"25f69674-db4a-4ce5-a441-d51b61b91d2f","Type":"ContainerDied","Data":"303bfa7ccec2421ef1e9d0125106744445fef4d0d89db08937c51343313386c1"}
Oct 31 12:56:54.525978 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:54.525954    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05c99f42-tvtf4"
Oct 31 12:56:54.684507 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:54.684405    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25f69674-db4a-4ce5-a441-d51b61b91d2f-kube-api-access-j768n\" (UniqueName: \"kubernetes.io/projected/25f69674-db4a-4ce5-a441-d51b61b91d2f-kube-api-access-j768n\") pod \"25f69674-db4a-4ce5-a441-d51b61b91d2f\" (UID: \"25f69674-db4a-4ce5-a441-d51b61b91d2f\") "
Oct 31 12:56:54.684507 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:54.684461    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25f69674-db4a-4ce5-a441-d51b61b91d2f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25f69674-db4a-4ce5-a441-d51b61b91d2f-maas-api-openshift-ca\") pod \"25f69674-db4a-4ce5-a441-d51b61b91d2f\" (UID: \"25f69674-db4a-4ce5-a441-d51b61b91d2f\") "
Oct 31 12:56:54.684788 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:54.684543    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/25f69674-db4a-4ce5-a441-d51b61b91d2f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25f69674-db4a-4ce5-a441-d51b61b91d2f-maas-api-xks-ca\") pod \"25f69674-db4a-4ce5-a441-d51b61b91d2f\" (UID: \"25f69674-db4a-4ce5-a441-d51b61b91d2f\") "
Oct 31 12:56:54.684987 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:54.684948    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25f69674-db4a-4ce5-a441-d51b61b91d2f-maas-api-openshift-ca" pod "25f69674-db4a-4ce5-a441-d51b61b91d2f" (UID: "25f69674-db4a-4ce5-a441-d51b61b91d2f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:56:54.685235 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:54.685212    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25f69674-db4a-4ce5-a441-d51b61b91d2f-maas-api-openshift-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:56:54.687275 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:54.687245    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25f69674-db4a-4ce5-a441-d51b61b91d2f-kube-api-access-j768n" pod "25f69674-db4a-4ce5-a441-d51b61b91d2f" (UID: "25f69674-db4a-4ce5-a441-d51b61b91d2f"). InnerVolumeSpecName "kube-api-access-j768n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:56:54.687385 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:54.687284    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25f69674-db4a-4ce5-a441-d51b61b91d2f-maas-api-xks-ca" pod "25f69674-db4a-4ce5-a441-d51b61b91d2f" (UID: "25f69674-db4a-4ce5-a441-d51b61b91d2f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:56:54.786898 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:54.786860    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25f69674-db4a-4ce5-a441-d51b61b91d2f-maas-api-xks-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:56:54.786898 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:54.786894    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j768n\" (UniqueName: \"kubernetes.io/projected/25f69674-db4a-4ce5-a441-d51b61b91d2f-kube-api-access-j768n\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:56:55.395573 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:55.386812    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05c99f42-tvtf4" event={"ID":"25f69674-db4a-4ce5-a441-d51b61b91d2f","Type":"ContainerDied","Data":"c18490afe7c04b138121b7a36c6d8da73298d258e1525a7cbdf76181387e9be9"}
Oct 31 12:56:55.395573 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:55.386906    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c18490afe7c04b138121b7a36c6d8da73298d258e1525a7cbdf76181387e9be9"
Oct 31 12:56:55.395573 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:55.386993    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05c99f42-tvtf4"
Oct 31 12:56:58.855699 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:58.855658    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-fb0343-85b486979f-blth7" podUID="5f8db300-b095-4b06-9fa5-1804c2a64df8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 12:56:58.856070 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:56:58.855658    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-fb0343-5d8d4b7fb7-dtkzd" podUID="6a041d29-7b54-4d07-be06-2c96f8fc097c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 12:56:59.922922 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:56:59.922885    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7phb" containerName="extractor"
Oct 31 12:57:08.855916 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:57:08.855813    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-fb0343-85b486979f-blth7" podUID="5f8db300-b095-4b06-9fa5-1804c2a64df8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 12:57:08.855916 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:57:08.855897    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-fb0343-5d8d4b7fb7-dtkzd" podUID="6a041d29-7b54-4d07-be06-2c96f8fc097c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 12:57:09.321311 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:57:09.321027    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-fb0343-5d8d4b7fb7-dtkzd"
Oct 31 12:57:09.407564 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:57:09.407263    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a041d29-7b54-4d07-be06-2c96f8fc097c-kube-api-access-drbkt\" (UniqueName: \"kubernetes.io/projected/6a041d29-7b54-4d07-be06-2c96f8fc097c-kube-api-access-drbkt\") pod \"6a041d29-7b54-4d07-be06-2c96f8fc097c\" (UID: \"6a041d29-7b54-4d07-be06-2c96f8fc097c\") "
Oct 31 12:57:09.407564 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:57:09.407324    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a041d29-7b54-4d07-be06-2c96f8fc097c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a041d29-7b54-4d07-be06-2c96f8fc097c-plugins-config-volume\") pod \"6a041d29-7b54-4d07-be06-2c96f8fc097c\" (UID: \"6a041d29-7b54-4d07-be06-2c96f8fc097c\") "
Oct 31 12:57:09.412181 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:57:09.412115    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a041d29-7b54-4d07-be06-2c96f8fc097c-plugins-config-volume" pod "6a041d29-7b54-4d07-be06-2c96f8fc097c" (UID: "6a041d29-7b54-4d07-be06-2c96f8fc097c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:57:09.419685 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:57:09.419648    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a041d29-7b54-4d07-be06-2c96f8fc097c-kube-api-access-drbkt" pod "6a041d29-7b54-4d07-be06-2c96f8fc097c" (UID: "6a041d29-7b54-4d07-be06-2c96f8fc097c"). InnerVolumeSpecName "kube-api-access-drbkt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:57:09.453894 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:57:09.453867    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-fb0343-85b486979f-blth7"
Oct 31 12:57:09.508776 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:57:09.508739    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f8db300-b095-4b06-9fa5-1804c2a64df8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f8db300-b095-4b06-9fa5-1804c2a64df8-plugins-config-volume\") pod \"5f8db300-b095-4b06-9fa5-1804c2a64df8\" (UID: \"5f8db300-b095-4b06-9fa5-1804c2a64df8\") "
Oct 31 12:57:09.509067 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:57:09.509043    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f8db300-b095-4b06-9fa5-1804c2a64df8-kube-api-access-s4g4k\" (UniqueName: \"kubernetes.io/projected/5f8db300-b095-4b06-9fa5-1804c2a64df8-kube-api-access-s4g4k\") pod \"5f8db300-b095-4b06-9fa5-1804c2a64df8\" (UID: \"5f8db300-b095-4b06-9fa5-1804c2a64df8\") "
Oct 31 12:57:09.509187 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:57:09.509152    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f8db300-b095-4b06-9fa5-1804c2a64df8-plugins-config-volume" pod "5f8db300-b095-4b06-9fa5-1804c2a64df8" (UID: "5f8db300-b095-4b06-9fa5-1804c2a64df8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:57:09.509583 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:57:09.509550    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drbkt\" (UniqueName: \"kubernetes.io/projected/6a041d29-7b54-4d07-be06-2c96f8fc097c-kube-api-access-drbkt\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:57:09.509583 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:57:09.509573    5060 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a041d29-7b54-4d07-be06-2c96f8fc097c-plugins-config-volume\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:57:09.509583 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:57:09.509582    5060 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f8db300-b095-4b06-9fa5-1804c2a64df8-plugins-config-volume\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:57:09.511528 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:57:09.511503    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f8db300-b095-4b06-9fa5-1804c2a64df8-kube-api-access-s4g4k" pod "5f8db300-b095-4b06-9fa5-1804c2a64df8" (UID: "5f8db300-b095-4b06-9fa5-1804c2a64df8"). InnerVolumeSpecName "kube-api-access-s4g4k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:57:09.611504 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:57:09.611460    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s4g4k\" (UniqueName: \"kubernetes.io/projected/5f8db300-b095-4b06-9fa5-1804c2a64df8-kube-api-access-s4g4k\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:57:09.623030 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:57:09.622995    5060 generic.go:356] "Generic (PLEG): container finished" podID="5f8db300-b095-4b06-9fa5-1804c2a64df8" containerID="7e07cca37a3d92c5eee144bc04bab214a169ce484dfa655084775f7a977435bc" exitCode=0
Oct 31 12:57:09.623224 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:57:09.623039    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-fb0343-85b486979f-blth7"
Oct 31 12:57:09.623224 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:57:09.623079    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-fb0343-85b486979f-blth7" event={"ID":"5f8db300-b095-4b06-9fa5-1804c2a64df8","Type":"ContainerDied","Data":"7e07cca37a3d92c5eee144bc04bab214a169ce484dfa655084775f7a977435bc"}
Oct 31 12:57:09.623224 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:57:09.623130    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-fb0343-85b486979f-blth7" event={"ID":"5f8db300-b095-4b06-9fa5-1804c2a64df8","Type":"ContainerDied","Data":"73eb3427f1dbbb16373ba8b90a77f301f6b2659bbafc377770768c044bb357c7"}
Oct 31 12:57:09.623224 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:57:09.623151    5060 scope.go:122] "RemoveContainer" containerID="7e07cca37a3d92c5eee144bc04bab214a169ce484dfa655084775f7a977435bc"
Oct 31 12:57:09.624921 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:57:09.624895    5060 generic.go:356] "Generic (PLEG): container finished" podID="6a041d29-7b54-4d07-be06-2c96f8fc097c" containerID="481c99373b0e0f398f783a1ef91a8e7c1944fa271ed2428a6fb1c0632dde60c2" exitCode=0
Oct 31 12:57:09.625056 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:57:09.624944    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-fb0343-5d8d4b7fb7-dtkzd"
Oct 31 12:57:09.625056 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:57:09.624973    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-fb0343-5d8d4b7fb7-dtkzd" event={"ID":"6a041d29-7b54-4d07-be06-2c96f8fc097c","Type":"ContainerDied","Data":"481c99373b0e0f398f783a1ef91a8e7c1944fa271ed2428a6fb1c0632dde60c2"}
Oct 31 12:57:09.625056 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:57:09.625025    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-fb0343-5d8d4b7fb7-dtkzd" event={"ID":"6a041d29-7b54-4d07-be06-2c96f8fc097c","Type":"ContainerDied","Data":"2e64981718a5452ee7cf6828574cff1103b1a8ad00b2f36732201b956f8541e1"}
Oct 31 12:57:09.635544 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:57:09.635525    5060 scope.go:122] "RemoveContainer" containerID="7e07cca37a3d92c5eee144bc04bab214a169ce484dfa655084775f7a977435bc"
Oct 31 12:57:09.635807 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:57:09.635783    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e07cca37a3d92c5eee144bc04bab214a169ce484dfa655084775f7a977435bc\": container with ID starting with 7e07cca37a3d92c5eee144bc04bab214a169ce484dfa655084775f7a977435bc not found: ID does not exist" containerID="7e07cca37a3d92c5eee144bc04bab214a169ce484dfa655084775f7a977435bc"
Oct 31 12:57:09.635904 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:57:09.635814    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e07cca37a3d92c5eee144bc04bab214a169ce484dfa655084775f7a977435bc"} err="failed to get container status \"7e07cca37a3d92c5eee144bc04bab214a169ce484dfa655084775f7a977435bc\": rpc error: code = NotFound desc = could not find container \"7e07cca37a3d92c5eee144bc04bab214a169ce484dfa655084775f7a977435bc\": container with ID starting with 7e07cca37a3d92c5eee144bc04bab214a169ce484dfa655084775f7a977435bc not found: ID does not exist"
Oct 31 12:57:09.635904 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:57:09.635833    5060 scope.go:122] "RemoveContainer" containerID="481c99373b0e0f398f783a1ef91a8e7c1944fa271ed2428a6fb1c0632dde60c2"
Oct 31 12:57:09.646792 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:57:09.646775    5060 scope.go:122] "RemoveContainer" containerID="481c99373b0e0f398f783a1ef91a8e7c1944fa271ed2428a6fb1c0632dde60c2"
Oct 31 12:57:09.647352 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:57:09.647046    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"481c99373b0e0f398f783a1ef91a8e7c1944fa271ed2428a6fb1c0632dde60c2\": container with ID starting with 481c99373b0e0f398f783a1ef91a8e7c1944fa271ed2428a6fb1c0632dde60c2 not found: ID does not exist" containerID="481c99373b0e0f398f783a1ef91a8e7c1944fa271ed2428a6fb1c0632dde60c2"
Oct 31 12:57:09.647352 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:57:09.647205    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"481c99373b0e0f398f783a1ef91a8e7c1944fa271ed2428a6fb1c0632dde60c2"} err="failed to get container status \"481c99373b0e0f398f783a1ef91a8e7c1944fa271ed2428a6fb1c0632dde60c2\": rpc error: code = NotFound desc = could not find container \"481c99373b0e0f398f783a1ef91a8e7c1944fa271ed2428a6fb1c0632dde60c2\": container with ID starting with 481c99373b0e0f398f783a1ef91a8e7c1944fa271ed2428a6fb1c0632dde60c2 not found: ID does not exist"
Oct 31 12:57:09.652775 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:57:09.652750    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-fb0343-5d8d4b7fb7-dtkzd"]
Oct 31 12:57:09.660795 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:57:09.660736    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-fb0343-5d8d4b7fb7-dtkzd"]
Oct 31 12:57:09.667798 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:57:09.667772    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-fb0343-85b486979f-blth7"]
Oct 31 12:57:09.674016 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:57:09.673990    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-fb0343-85b486979f-blth7"]
Oct 31 12:57:11.049911 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:57:11.049880    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f8db300-b095-4b06-9fa5-1804c2a64df8" path="/var/lib/kubelet/pods/5f8db300-b095-4b06-9fa5-1804c2a64df8/volumes"
Oct 31 12:57:11.050310 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:57:11.050296    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a041d29-7b54-4d07-be06-2c96f8fc097c" path="/var/lib/kubelet/pods/6a041d29-7b54-4d07-be06-2c96f8fc097c/volumes"
Oct 31 12:57:15.748906 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:57:15.748871    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-kmx7z"]
Oct 31 12:57:15.749331 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:57:15.749118    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-kmx7z" podUID="9b8d44ec-1b81-4ca2-9337-6294db29bb5b" containerName="istio-proxy" containerID="cri-o://7f53fd2543e6a9686da79fddfa2f812edc241f2cc4cb638a5678d8679506870a" gracePeriod=30
Oct 31 12:57:15.753562 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:57:15.753534    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-qqhsw"]
Oct 31 12:57:15.753743 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:57:15.753723    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-qqhsw" podUID="d9b854bc-d670-414d-8e11-dd7bd401a59f" containerName="istio-proxy" containerID="cri-o://6cd106dfdae1207201a8ba030192c7573beb0d6a4448189ba398e915b4a179c3" gracePeriod=30
Oct 31 12:57:17.397472 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:57:17.397423    5060 patch_prober.go:29] interesting pod/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-kmx7z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 12:57:17.398047 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:57:17.397484    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-kmx7z" podUID="9b8d44ec-1b81-4ca2-9337-6294db29bb5b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 12:57:17.851302 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:57:17.851268    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pcvvm_f032909d-bc39-4c9e-99b7-f3d800842119/console-operator/2.log"
Oct 31 12:57:17.852140 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:57:17.852119    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pcvvm_f032909d-bc39-4c9e-99b7-f3d800842119/console-operator/2.log"
Oct 31 12:57:21.077756 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:57:21.077724    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-kmx7z"
Oct 31 12:57:21.175778 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:57:21.175702    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-qqhsw"
Oct 31 12:57:21.204102 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:57:21.204058    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b8d44ec-1b81-4ca2-9337-6294db29bb5b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b8d44ec-1b81-4ca2-9337-6294db29bb5b-workload-socket\") pod \"9b8d44ec-1b81-4ca2-9337-6294db29bb5b\" (UID: \"9b8d44ec-1b81-4ca2-9337-6294db29bb5b\") "
Oct 31 12:57:21.204280 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:57:21.204164    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b8d44ec-1b81-4ca2-9337-6294db29bb5b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b8d44ec-1b81-4ca2-9337-6294db29bb5b-istio-envoy\") pod \"9b8d44ec-1b81-4ca2-9337-6294db29bb5b\" (UID: \"9b8d44ec-1b81-4ca2-9337-6294db29bb5b\") "
Oct 31 12:57:21.204324 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:57:21.204309    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b8d44ec-1b81-4ca2-9337-6294db29bb5b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b8d44ec-1b81-4ca2-9337-6294db29bb5b-workload-certs\") pod \"9b8d44ec-1b81-4ca2-9337-6294db29bb5b\" (UID: \"9b8d44ec-1b81-4ca2-9337-6294db29bb5b\") "
Oct 31 12:57:21.204405 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:57:21.204383    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b8d44ec-1b81-4ca2-9337-6294db29bb5b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b8d44ec-1b81-4ca2-9337-6294db29bb5b-istio-data\") pod \"9b8d44ec-1b81-4ca2-9337-6294db29bb5b\" (UID: \"9b8d44ec-1b81-4ca2-9337-6294db29bb5b\") "
Oct 31 12:57:21.205731 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:57:21.205709    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9b8d44ec-1b81-4ca2-9337-6294db29bb5b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b8d44ec-1b81-4ca2-9337-6294db29bb5b-istio-podinfo\") pod \"9b8d44ec-1b81-4ca2-9337-6294db29bb5b\" (UID: \"9b8d44ec-1b81-4ca2-9337-6294db29bb5b\") "
Oct 31 12:57:21.206242 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:57:21.206221    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b8d44ec-1b81-4ca2-9337-6294db29bb5b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b8d44ec-1b81-4ca2-9337-6294db29bb5b-istiod-ca-cert\") pod \"9b8d44ec-1b81-4ca2-9337-6294db29bb5b\" (UID: \"9b8d44ec-1b81-4ca2-9337-6294db29bb5b\") "
Oct 31 12:57:21.207359 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:57:21.207336    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b8d44ec-1b81-4ca2-9337-6294db29bb5b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b8d44ec-1b81-4ca2-9337-6294db29bb5b-credential-socket\") pod \"9b8d44ec-1b81-4ca2-9337-6294db29bb5b\" (UID: \"9b8d44ec-1b81-4ca2-9337-6294db29bb5b\") "
Oct 31 12:57:21.207907 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:57:21.204863    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b8d44ec-1b81-4ca2-9337-6294db29bb5b-istio-data" pod "9b8d44ec-1b81-4ca2-9337-6294db29bb5b" (UID: "9b8d44ec-1b81-4ca2-9337-6294db29bb5b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:57:21.207907 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:57:21.205040    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b8d44ec-1b81-4ca2-9337-6294db29bb5b-workload-socket" pod "9b8d44ec-1b81-4ca2-9337-6294db29bb5b" (UID: "9b8d44ec-1b81-4ca2-9337-6294db29bb5b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:57:21.207907 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:57:21.205649    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b8d44ec-1b81-4ca2-9337-6294db29bb5b-workload-certs" pod "9b8d44ec-1b81-4ca2-9337-6294db29bb5b" (UID: "9b8d44ec-1b81-4ca2-9337-6294db29bb5b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:57:21.207907 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:57:21.207213    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b8d44ec-1b81-4ca2-9337-6294db29bb5b-istiod-ca-cert" pod "9b8d44ec-1b81-4ca2-9337-6294db29bb5b" (UID: "9b8d44ec-1b81-4ca2-9337-6294db29bb5b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:57:21.208204 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:57:21.207923    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b8d44ec-1b81-4ca2-9337-6294db29bb5b-istio-envoy" pod "9b8d44ec-1b81-4ca2-9337-6294db29bb5b" (UID: "9b8d44ec-1b81-4ca2-9337-6294db29bb5b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:57:21.208591 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:57:21.208569    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b8d44ec-1b81-4ca2-9337-6294db29bb5b-istio-token\" (UniqueName: \"kubernetes.io/projected/9b8d44ec-1b81-4ca2-9337-6294db29bb5b-istio-token\") pod \"9b8d44ec-1b81-4ca2-9337-6294db29bb5b\" (UID: \"9b8d44ec-1b81-4ca2-9337-6294db29bb5b\") "
Oct 31 12:57:21.209154 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:57:21.209119    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b8d44ec-1b81-4ca2-9337-6294db29bb5b-kube-api-access-8h5mb\" (UniqueName: \"kubernetes.io/projected/9b8d44ec-1b81-4ca2-9337-6294db29bb5b-kube-api-access-8h5mb\") pod \"9b8d44ec-1b81-4ca2-9337-6294db29bb5b\" (UID: \"9b8d44ec-1b81-4ca2-9337-6294db29bb5b\") "
Oct 31 12:57:21.209467 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:57:21.209429    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b8d44ec-1b81-4ca2-9337-6294db29bb5b-credential-socket" pod "9b8d44ec-1b81-4ca2-9337-6294db29bb5b" (UID: "9b8d44ec-1b81-4ca2-9337-6294db29bb5b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:57:21.209929 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:57:21.209602    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9b8d44ec-1b81-4ca2-9337-6294db29bb5b-istio-podinfo" pod "9b8d44ec-1b81-4ca2-9337-6294db29bb5b" (UID: "9b8d44ec-1b81-4ca2-9337-6294db29bb5b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 12:57:21.211074 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:57:21.211045    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b8d44ec-1b81-4ca2-9337-6294db29bb5b-istio-token" pod "9b8d44ec-1b81-4ca2-9337-6294db29bb5b" (UID: "9b8d44ec-1b81-4ca2-9337-6294db29bb5b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:57:21.211299 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:57:21.211270    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b8d44ec-1b81-4ca2-9337-6294db29bb5b-kube-api-access-8h5mb" pod "9b8d44ec-1b81-4ca2-9337-6294db29bb5b" (UID: "9b8d44ec-1b81-4ca2-9337-6294db29bb5b"). InnerVolumeSpecName "kube-api-access-8h5mb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:57:21.211603 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:57:21.211576    5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b8d44ec-1b81-4ca2-9337-6294db29bb5b-workload-certs\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:57:21.211726 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:57:21.211711    5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b8d44ec-1b81-4ca2-9337-6294db29bb5b-istio-data\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:57:21.211837 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:57:21.211824    5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b8d44ec-1b81-4ca2-9337-6294db29bb5b-istio-podinfo\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:57:21.211933 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:57:21.211920    5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b8d44ec-1b81-4ca2-9337-6294db29bb5b-istiod-ca-cert\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:57:21.212032 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:57:21.212020    5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b8d44ec-1b81-4ca2-9337-6294db29bb5b-credential-socket\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:57:21.212122 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:57:21.212110    5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b8d44ec-1b81-4ca2-9337-6294db29bb5b-workload-socket\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:57:21.212216 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:57:21.212203    5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b8d44ec-1b81-4ca2-9337-6294db29bb5b-istio-envoy\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:57:21.313219 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:57:21.313145    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9b854bc-d670-414d-8e11-dd7bd401a59f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9b854bc-d670-414d-8e11-dd7bd401a59f-istio-envoy\") pod \"d9b854bc-d670-414d-8e11-dd7bd401a59f\" (UID: \"d9b854bc-d670-414d-8e11-dd7bd401a59f\") "
Oct 31 12:57:21.313219 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:57:21.313202    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9b854bc-d670-414d-8e11-dd7bd401a59f-kube-api-access-r5hcq\" (UniqueName: \"kubernetes.io/projected/d9b854bc-d670-414d-8e11-dd7bd401a59f-kube-api-access-r5hcq\") pod \"d9b854bc-d670-414d-8e11-dd7bd401a59f\" (UID: \"d9b854bc-d670-414d-8e11-dd7bd401a59f\") "
Oct 31 12:57:21.313430 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:57:21.313284    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9b854bc-d670-414d-8e11-dd7bd401a59f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9b854bc-d670-414d-8e11-dd7bd401a59f-istiod-ca-cert\") pod \"d9b854bc-d670-414d-8e11-dd7bd401a59f\" (UID: \"d9b854bc-d670-414d-8e11-dd7bd401a59f\") "
Oct 31 12:57:21.313430 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:57:21.313370    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9b854bc-d670-414d-8e11-dd7bd401a59f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9b854bc-d670-414d-8e11-dd7bd401a59f-istio-data\") pod \"d9b854bc-d670-414d-8e11-dd7bd401a59f\" (UID: \"d9b854bc-d670-414d-8e11-dd7bd401a59f\") "
Oct 31 12:57:21.313430 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:57:21.313398    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9b854bc-d670-414d-8e11-dd7bd401a59f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9b854bc-d670-414d-8e11-dd7bd401a59f-workload-certs\") pod \"d9b854bc-d670-414d-8e11-dd7bd401a59f\" (UID: \"d9b854bc-d670-414d-8e11-dd7bd401a59f\") "
Oct 31 12:57:21.313662 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:57:21.313427    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9b854bc-d670-414d-8e11-dd7bd401a59f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9b854bc-d670-414d-8e11-dd7bd401a59f-credential-socket\") pod \"d9b854bc-d670-414d-8e11-dd7bd401a59f\" (UID: \"d9b854bc-d670-414d-8e11-dd7bd401a59f\") "
Oct 31 12:57:21.313662 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:57:21.313463    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d9b854bc-d670-414d-8e11-dd7bd401a59f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9b854bc-d670-414d-8e11-dd7bd401a59f-istio-podinfo\") pod \"d9b854bc-d670-414d-8e11-dd7bd401a59f\" (UID: \"d9b854bc-d670-414d-8e11-dd7bd401a59f\") "
Oct 31 12:57:21.313662 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:57:21.313605    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9b854bc-d670-414d-8e11-dd7bd401a59f-istio-token\" (UniqueName: \"kubernetes.io/projected/d9b854bc-d670-414d-8e11-dd7bd401a59f-istio-token\") pod \"d9b854bc-d670-414d-8e11-dd7bd401a59f\" (UID: \"d9b854bc-d670-414d-8e11-dd7bd401a59f\") "
Oct 31 12:57:21.313839 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:57:21.313658    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9b854bc-d670-414d-8e11-dd7bd401a59f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9b854bc-d670-414d-8e11-dd7bd401a59f-workload-socket\") pod \"d9b854bc-d670-414d-8e11-dd7bd401a59f\" (UID: \"d9b854bc-d670-414d-8e11-dd7bd401a59f\") "
Oct 31 12:57:21.313839 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:57:21.313696    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9b854bc-d670-414d-8e11-dd7bd401a59f-istio-data" pod "d9b854bc-d670-414d-8e11-dd7bd401a59f" (UID: "d9b854bc-d670-414d-8e11-dd7bd401a59f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:57:21.313839 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:57:21.313712    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9b854bc-d670-414d-8e11-dd7bd401a59f-istiod-ca-cert" pod "d9b854bc-d670-414d-8e11-dd7bd401a59f" (UID: "d9b854bc-d670-414d-8e11-dd7bd401a59f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:57:21.313839 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:57:21.313830    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9b854bc-d670-414d-8e11-dd7bd401a59f-credential-socket" pod "d9b854bc-d670-414d-8e11-dd7bd401a59f" (UID: "d9b854bc-d670-414d-8e11-dd7bd401a59f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:57:21.314251 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:57:21.314019    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9b854bc-d670-414d-8e11-dd7bd401a59f-workload-certs" pod "d9b854bc-d670-414d-8e11-dd7bd401a59f" (UID: "d9b854bc-d670-414d-8e11-dd7bd401a59f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:57:21.314251 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:57:21.314044    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9b854bc-d670-414d-8e11-dd7bd401a59f-workload-socket" pod "d9b854bc-d670-414d-8e11-dd7bd401a59f" (UID: "d9b854bc-d670-414d-8e11-dd7bd401a59f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:57:21.314415 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:57:21.314388    5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9b854bc-d670-414d-8e11-dd7bd401a59f-istio-data\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:57:21.314584 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:57:21.314418    5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9b854bc-d670-414d-8e11-dd7bd401a59f-workload-certs\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:57:21.314584 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:57:21.314435    5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9b854bc-d670-414d-8e11-dd7bd401a59f-credential-socket\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:57:21.314584 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:57:21.314450    5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9b854bc-d670-414d-8e11-dd7bd401a59f-workload-socket\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:57:21.314584 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:57:21.314464    5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b8d44ec-1b81-4ca2-9337-6294db29bb5b-istio-token\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:57:21.314584 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:57:21.314478    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8h5mb\" (UniqueName: \"kubernetes.io/projected/9b8d44ec-1b81-4ca2-9337-6294db29bb5b-kube-api-access-8h5mb\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:57:21.314584 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:57:21.314536    5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9b854bc-d670-414d-8e11-dd7bd401a59f-istiod-ca-cert\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:57:21.315957 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:57:21.315931    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9b854bc-d670-414d-8e11-dd7bd401a59f-istio-envoy" pod "d9b854bc-d670-414d-8e11-dd7bd401a59f" (UID: "d9b854bc-d670-414d-8e11-dd7bd401a59f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:57:21.316077 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:57:21.315964    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9b854bc-d670-414d-8e11-dd7bd401a59f-kube-api-access-r5hcq" pod "d9b854bc-d670-414d-8e11-dd7bd401a59f" (UID: "d9b854bc-d670-414d-8e11-dd7bd401a59f"). InnerVolumeSpecName "kube-api-access-r5hcq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:57:21.316281 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:57:21.316263    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9b854bc-d670-414d-8e11-dd7bd401a59f-istio-token" pod "d9b854bc-d670-414d-8e11-dd7bd401a59f" (UID: "d9b854bc-d670-414d-8e11-dd7bd401a59f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:57:21.316331 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:57:21.316264    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d9b854bc-d670-414d-8e11-dd7bd401a59f-istio-podinfo" pod "d9b854bc-d670-414d-8e11-dd7bd401a59f" (UID: "d9b854bc-d670-414d-8e11-dd7bd401a59f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 12:57:21.416059 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:57:21.416029    5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9b854bc-d670-414d-8e11-dd7bd401a59f-istio-podinfo\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:57:21.416224 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:57:21.416069    5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9b854bc-d670-414d-8e11-dd7bd401a59f-istio-token\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:57:21.416224 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:57:21.416079    5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9b854bc-d670-414d-8e11-dd7bd401a59f-istio-envoy\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:57:21.416224 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:57:21.416088    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5hcq\" (UniqueName: \"kubernetes.io/projected/d9b854bc-d670-414d-8e11-dd7bd401a59f-kube-api-access-r5hcq\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:57:21.830684 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:57:21.830649    5060 generic.go:356] "Generic (PLEG): container finished" podID="9b8d44ec-1b81-4ca2-9337-6294db29bb5b" containerID="7f53fd2543e6a9686da79fddfa2f812edc241f2cc4cb638a5678d8679506870a" exitCode=0
Oct 31 12:57:21.830881 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:57:21.830700    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-kmx7z"
Oct 31 12:57:21.830881 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:57:21.830728    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-kmx7z" event={"ID":"9b8d44ec-1b81-4ca2-9337-6294db29bb5b","Type":"ContainerDied","Data":"7f53fd2543e6a9686da79fddfa2f812edc241f2cc4cb638a5678d8679506870a"}
Oct 31 12:57:21.830881 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:57:21.830784    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-kmx7z" event={"ID":"9b8d44ec-1b81-4ca2-9337-6294db29bb5b","Type":"ContainerDied","Data":"c263cb3a737d0fd06a44d32b13b7c9d2df85563729ce11284e8ced59d219bd46"}
Oct 31 12:57:21.830881 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:57:21.830810    5060 scope.go:122] "RemoveContainer" containerID="7f53fd2543e6a9686da79fddfa2f812edc241f2cc4cb638a5678d8679506870a"
Oct 31 12:57:21.832413 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:57:21.832388    5060 generic.go:356] "Generic (PLEG): container finished" podID="d9b854bc-d670-414d-8e11-dd7bd401a59f" containerID="6cd106dfdae1207201a8ba030192c7573beb0d6a4448189ba398e915b4a179c3" exitCode=0
Oct 31 12:57:21.832556 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:57:21.832432    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-qqhsw"
Oct 31 12:57:21.832556 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:57:21.832459    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-qqhsw" event={"ID":"d9b854bc-d670-414d-8e11-dd7bd401a59f","Type":"ContainerDied","Data":"6cd106dfdae1207201a8ba030192c7573beb0d6a4448189ba398e915b4a179c3"}
Oct 31 12:57:21.832556 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:57:21.832545    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-qqhsw" event={"ID":"d9b854bc-d670-414d-8e11-dd7bd401a59f","Type":"ContainerDied","Data":"43a74d71decd2d2e41ae7eaeebf21ddd7236647369dae8ac32329bf45591a915"}
Oct 31 12:57:21.844756 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:57:21.844738    5060 scope.go:122] "RemoveContainer" containerID="7f53fd2543e6a9686da79fddfa2f812edc241f2cc4cb638a5678d8679506870a"
Oct 31 12:57:21.845055 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:57:21.845033    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f53fd2543e6a9686da79fddfa2f812edc241f2cc4cb638a5678d8679506870a\": container with ID starting with 7f53fd2543e6a9686da79fddfa2f812edc241f2cc4cb638a5678d8679506870a not found: ID does not exist" containerID="7f53fd2543e6a9686da79fddfa2f812edc241f2cc4cb638a5678d8679506870a"
Oct 31 12:57:21.845155 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:57:21.845060    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f53fd2543e6a9686da79fddfa2f812edc241f2cc4cb638a5678d8679506870a"} err="failed to get container status \"7f53fd2543e6a9686da79fddfa2f812edc241f2cc4cb638a5678d8679506870a\": rpc error: code = NotFound desc = could not find container \"7f53fd2543e6a9686da79fddfa2f812edc241f2cc4cb638a5678d8679506870a\": container with ID starting with 7f53fd2543e6a9686da79fddfa2f812edc241f2cc4cb638a5678d8679506870a not found: ID does not exist"
Oct 31 12:57:21.845155 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:57:21.845079    5060 scope.go:122] "RemoveContainer" containerID="6cd106dfdae1207201a8ba030192c7573beb0d6a4448189ba398e915b4a179c3"
Oct 31 12:57:21.856468 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:57:21.856446    5060 scope.go:122] "RemoveContainer" containerID="6cd106dfdae1207201a8ba030192c7573beb0d6a4448189ba398e915b4a179c3"
Oct 31 12:57:21.856848 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:57:21.856821    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6cd106dfdae1207201a8ba030192c7573beb0d6a4448189ba398e915b4a179c3\": container with ID starting with 6cd106dfdae1207201a8ba030192c7573beb0d6a4448189ba398e915b4a179c3 not found: ID does not exist" containerID="6cd106dfdae1207201a8ba030192c7573beb0d6a4448189ba398e915b4a179c3"
Oct 31 12:57:21.856947 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:57:21.856858    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6cd106dfdae1207201a8ba030192c7573beb0d6a4448189ba398e915b4a179c3"} err="failed to get container status \"6cd106dfdae1207201a8ba030192c7573beb0d6a4448189ba398e915b4a179c3\": rpc error: code = NotFound desc = could not find container \"6cd106dfdae1207201a8ba030192c7573beb0d6a4448189ba398e915b4a179c3\": container with ID starting with 6cd106dfdae1207201a8ba030192c7573beb0d6a4448189ba398e915b4a179c3 not found: ID does not exist"
Oct 31 12:57:21.861016 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:57:21.860993    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-kmx7z"]
Oct 31 12:57:21.865267 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:57:21.865246    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-kmx7z"]
Oct 31 12:57:21.875761 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:57:21.875741    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-qqhsw"]
Oct 31 12:57:21.882343 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:57:21.882306    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-qqhsw"]
Oct 31 12:57:23.049097 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:57:23.049062    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b8d44ec-1b81-4ca2-9337-6294db29bb5b" path="/var/lib/kubelet/pods/9b8d44ec-1b81-4ca2-9337-6294db29bb5b/volumes"
Oct 31 12:57:23.049724 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:57:23.049707    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9b854bc-d670-414d-8e11-dd7bd401a59f" path="/var/lib/kubelet/pods/d9b854bc-d670-414d-8e11-dd7bd401a59f/volumes"
Oct 31 12:57:26.052693 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:57:26.052659    5060 scope.go:122] "RemoveContainer" containerID="ccfc755f8b94bbd46d89db1fe24611084b3cfd202ff00c49da51659de1e13deb"
Oct 31 12:57:29.922736 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:57:29.922696    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7phb" containerName="extractor"
Oct 31 12:57:59.922818 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:57:59.922737    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7phb" containerName="extractor"
Oct 31 12:58:22.378661 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:22.378620    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f64af0e-chm2q"]
Oct 31 12:58:22.379421 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:22.379397    5060 cpu_manager.go:414] "containerMap: removing container" podUID="9b8d44ec-1b81-4ca2-9337-6294db29bb5b" containerName="istio-proxy"
Oct 31 12:58:22.379421 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:22.379422    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b8d44ec-1b81-4ca2-9337-6294db29bb5b" containerName="istio-proxy"
Oct 31 12:58:22.379608 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:22.379447    5060 cpu_manager.go:414] "containerMap: removing container" podUID="25f69674-db4a-4ce5-a441-d51b61b91d2f" containerName="revoke-keys"
Oct 31 12:58:22.379608 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:22.379455    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25f69674-db4a-4ce5-a441-d51b61b91d2f" containerName="revoke-keys"
Oct 31 12:58:22.379608 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:22.379465    5060 cpu_manager.go:414] "containerMap: removing container" podUID="6a041d29-7b54-4d07-be06-2c96f8fc097c" containerName="payload-processing"
Oct 31 12:58:22.379608 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:22.379473    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a041d29-7b54-4d07-be06-2c96f8fc097c" containerName="payload-processing"
Oct 31 12:58:22.379608 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:22.379539    5060 cpu_manager.go:414] "containerMap: removing container" podUID="d9b854bc-d670-414d-8e11-dd7bd401a59f" containerName="istio-proxy"
Oct 31 12:58:22.379608 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:22.379548    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9b854bc-d670-414d-8e11-dd7bd401a59f" containerName="istio-proxy"
Oct 31 12:58:22.379608 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:22.379563    5060 cpu_manager.go:414] "containerMap: removing container" podUID="5f8db300-b095-4b06-9fa5-1804c2a64df8" containerName="payload-pre-processing"
Oct 31 12:58:22.379608 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:22.379570    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f8db300-b095-4b06-9fa5-1804c2a64df8" containerName="payload-pre-processing"
Oct 31 12:58:22.380032 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:22.379713    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="25f69674-db4a-4ce5-a441-d51b61b91d2f" containerName="revoke-keys"
Oct 31 12:58:22.380032 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:22.379724    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a041d29-7b54-4d07-be06-2c96f8fc097c" containerName="payload-processing"
Oct 31 12:58:22.380032 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:22.379736    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9b854bc-d670-414d-8e11-dd7bd401a59f" containerName="istio-proxy"
Oct 31 12:58:22.380032 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:22.379744    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f8db300-b095-4b06-9fa5-1804c2a64df8" containerName="payload-pre-processing"
Oct 31 12:58:22.380032 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:22.379753    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b8d44ec-1b81-4ca2-9337-6294db29bb5b" containerName="istio-proxy"
Oct 31 12:58:22.383859 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:22.383836    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f64af0e-chm2q"
Oct 31 12:58:22.389980 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:22.389953    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 12:58:22.390140 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:22.390018    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7wsmb\""
Oct 31 12:58:22.407990 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:22.407957    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f64af0e-chm2q"]
Oct 31 12:58:22.457690 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:22.457657    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhncc\" (UniqueName: \"kubernetes.io/projected/4cef8546-3c4a-471a-88cb-f31b837b6f70-kube-api-access-vhncc\") pod \"maas-api-revoke-sub-0f64af0e-chm2q\" (UID: \"4cef8546-3c4a-471a-88cb-f31b837b6f70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f64af0e-chm2q"
Oct 31 12:58:22.457891 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:22.457794    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cef8546-3c4a-471a-88cb-f31b837b6f70-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0f64af0e-chm2q\" (UID: \"4cef8546-3c4a-471a-88cb-f31b837b6f70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f64af0e-chm2q"
Oct 31 12:58:22.457891 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:22.457830    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cef8546-3c4a-471a-88cb-f31b837b6f70-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0f64af0e-chm2q\" (UID: \"4cef8546-3c4a-471a-88cb-f31b837b6f70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f64af0e-chm2q"
Oct 31 12:58:22.561667 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:22.561623    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cef8546-3c4a-471a-88cb-f31b837b6f70-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0f64af0e-chm2q\" (UID: \"4cef8546-3c4a-471a-88cb-f31b837b6f70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f64af0e-chm2q"
Oct 31 12:58:22.581656 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:22.580363    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhncc\" (UniqueName: \"kubernetes.io/projected/4cef8546-3c4a-471a-88cb-f31b837b6f70-kube-api-access-vhncc\") pod \"maas-api-revoke-sub-0f64af0e-chm2q\" (UID: \"4cef8546-3c4a-471a-88cb-f31b837b6f70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f64af0e-chm2q"
Oct 31 12:58:22.581656 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:22.580584    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cef8546-3c4a-471a-88cb-f31b837b6f70-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0f64af0e-chm2q\" (UID: \"4cef8546-3c4a-471a-88cb-f31b837b6f70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f64af0e-chm2q"
Oct 31 12:58:22.581656 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:22.581459    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cef8546-3c4a-471a-88cb-f31b837b6f70-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0f64af0e-chm2q\" (UID: \"4cef8546-3c4a-471a-88cb-f31b837b6f70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f64af0e-chm2q"
Oct 31 12:58:22.587560 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:22.585879    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cef8546-3c4a-471a-88cb-f31b837b6f70-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0f64af0e-chm2q\" (UID: \"4cef8546-3c4a-471a-88cb-f31b837b6f70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f64af0e-chm2q"
Oct 31 12:58:22.593373 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:22.593341    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhncc\" (UniqueName: \"kubernetes.io/projected/4cef8546-3c4a-471a-88cb-f31b837b6f70-kube-api-access-vhncc\") pod \"maas-api-revoke-sub-0f64af0e-chm2q\" (UID: \"4cef8546-3c4a-471a-88cb-f31b837b6f70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f64af0e-chm2q"
Oct 31 12:58:22.708476 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:22.708389    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f64af0e-chm2q"
Oct 31 12:58:22.997953 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:22.997917    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f64af0e-chm2q"]
Oct 31 12:58:22.999112 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:58:22.999085    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cef8546_3c4a_471a_88cb_f31b837b6f70.slice/crio-dbb8d326b7d84c9d38b72834693a75ccc5c659ce859c3699c5e75e5424da60eb WatchSource:0}: Error finding container dbb8d326b7d84c9d38b72834693a75ccc5c659ce859c3699c5e75e5424da60eb: Status 404 returned error can't find the container with id dbb8d326b7d84c9d38b72834693a75ccc5c659ce859c3699c5e75e5424da60eb
Oct 31 12:58:23.538193 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:23.538153    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f64af0e-chm2q" event={"ID":"4cef8546-3c4a-471a-88cb-f31b837b6f70","Type":"ContainerStarted","Data":"a110a99c491380e7629140ccd2fca4d8106c276a5fd4bd342af3443e55b0c9e3"}
Oct 31 12:58:23.538635 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:23.538251    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f64af0e-chm2q" event={"ID":"4cef8546-3c4a-471a-88cb-f31b837b6f70","Type":"ContainerStarted","Data":"dbb8d326b7d84c9d38b72834693a75ccc5c659ce859c3699c5e75e5424da60eb"}
Oct 31 12:58:23.559980 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:23.559923    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f64af0e-chm2q" podStartSLOduration=1.559904833 podStartE2EDuration="1.559904833s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:58:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:58:23.555942646 +0000 UTC m=+1866.659010360" watchObservedRunningTime="2026-10-31 12:58:23.559904833 +0000 UTC m=+1866.662972551"
Oct 31 12:58:24.589115 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:24.589078    5060 generic.go:356] "Generic (PLEG): container finished" podID="4cef8546-3c4a-471a-88cb-f31b837b6f70" containerID="a110a99c491380e7629140ccd2fca4d8106c276a5fd4bd342af3443e55b0c9e3" exitCode=0
Oct 31 12:58:24.589863 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:24.589361    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f64af0e-chm2q" event={"ID":"4cef8546-3c4a-471a-88cb-f31b837b6f70","Type":"ContainerDied","Data":"a110a99c491380e7629140ccd2fca4d8106c276a5fd4bd342af3443e55b0c9e3"}
Oct 31 12:58:25.772891 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:25.772858    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f64af0e-chm2q"
Oct 31 12:58:25.855681 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:25.855599    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4cef8546-3c4a-471a-88cb-f31b837b6f70-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cef8546-3c4a-471a-88cb-f31b837b6f70-maas-api-openshift-ca\") pod \"4cef8546-3c4a-471a-88cb-f31b837b6f70\" (UID: \"4cef8546-3c4a-471a-88cb-f31b837b6f70\") "
Oct 31 12:58:25.855870 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:25.855854    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4cef8546-3c4a-471a-88cb-f31b837b6f70-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cef8546-3c4a-471a-88cb-f31b837b6f70-maas-api-xks-ca\") pod \"4cef8546-3c4a-471a-88cb-f31b837b6f70\" (UID: \"4cef8546-3c4a-471a-88cb-f31b837b6f70\") "
Oct 31 12:58:25.856002 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:25.855972    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4cef8546-3c4a-471a-88cb-f31b837b6f70-kube-api-access-vhncc\" (UniqueName: \"kubernetes.io/projected/4cef8546-3c4a-471a-88cb-f31b837b6f70-kube-api-access-vhncc\") pod \"4cef8546-3c4a-471a-88cb-f31b837b6f70\" (UID: \"4cef8546-3c4a-471a-88cb-f31b837b6f70\") "
Oct 31 12:58:25.856075 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:25.855993    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cef8546-3c4a-471a-88cb-f31b837b6f70-maas-api-openshift-ca" pod "4cef8546-3c4a-471a-88cb-f31b837b6f70" (UID: "4cef8546-3c4a-471a-88cb-f31b837b6f70"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:58:25.856452 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:25.856430    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cef8546-3c4a-471a-88cb-f31b837b6f70-maas-api-openshift-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:58:25.858296 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:25.858269    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cef8546-3c4a-471a-88cb-f31b837b6f70-kube-api-access-vhncc" pod "4cef8546-3c4a-471a-88cb-f31b837b6f70" (UID: "4cef8546-3c4a-471a-88cb-f31b837b6f70"). InnerVolumeSpecName "kube-api-access-vhncc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:58:25.858428 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:25.858411    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4cef8546-3c4a-471a-88cb-f31b837b6f70-maas-api-xks-ca" pod "4cef8546-3c4a-471a-88cb-f31b837b6f70" (UID: "4cef8546-3c4a-471a-88cb-f31b837b6f70"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:58:25.957814 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:25.957779    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhncc\" (UniqueName: \"kubernetes.io/projected/4cef8546-3c4a-471a-88cb-f31b837b6f70-kube-api-access-vhncc\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:58:25.957814 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:25.957808    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cef8546-3c4a-471a-88cb-f31b837b6f70-maas-api-xks-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:58:26.614077 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:26.614038    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f64af0e-chm2q" event={"ID":"4cef8546-3c4a-471a-88cb-f31b837b6f70","Type":"ContainerDied","Data":"dbb8d326b7d84c9d38b72834693a75ccc5c659ce859c3699c5e75e5424da60eb"}
Oct 31 12:58:26.614268 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:26.614103    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dbb8d326b7d84c9d38b72834693a75ccc5c659ce859c3699c5e75e5424da60eb"
Oct 31 12:58:26.614268 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:26.614115    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f64af0e-chm2q"
Oct 31 12:58:29.923251 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:58:29.923209    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7phb" containerName="extractor"
Oct 31 12:58:35.347646 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:35.347609    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-acad0e17-qnx5x"]
Oct 31 12:58:35.348470 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:35.348447    5060 cpu_manager.go:414] "containerMap: removing container" podUID="4cef8546-3c4a-471a-88cb-f31b837b6f70" containerName="revoke-keys"
Oct 31 12:58:35.348470 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:35.348470    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cef8546-3c4a-471a-88cb-f31b837b6f70" containerName="revoke-keys"
Oct 31 12:58:35.348740 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:35.348724    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="4cef8546-3c4a-471a-88cb-f31b837b6f70" containerName="revoke-keys"
Oct 31 12:58:35.353075 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:35.353053    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-acad0e17-qnx5x"
Oct 31 12:58:35.357249 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:35.357203    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 12:58:35.357389 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:35.357280    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7wsmb\""
Oct 31 12:58:35.365144 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:35.365112    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-acad0e17-qnx5x"]
Oct 31 12:58:35.386195 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:35.386154    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8p6m\" (UniqueName: \"kubernetes.io/projected/1d7613f9-bcf1-4226-9249-21f08991f8e1-kube-api-access-z8p6m\") pod \"maas-api-revoke-sub-acad0e17-qnx5x\" (UID: \"1d7613f9-bcf1-4226-9249-21f08991f8e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-acad0e17-qnx5x"
Oct 31 12:58:35.386402 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:35.386324    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d7613f9-bcf1-4226-9249-21f08991f8e1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-acad0e17-qnx5x\" (UID: \"1d7613f9-bcf1-4226-9249-21f08991f8e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-acad0e17-qnx5x"
Oct 31 12:58:35.386657 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:35.386634    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d7613f9-bcf1-4226-9249-21f08991f8e1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-acad0e17-qnx5x\" (UID: \"1d7613f9-bcf1-4226-9249-21f08991f8e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-acad0e17-qnx5x"
Oct 31 12:58:35.487790 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:35.487748    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d7613f9-bcf1-4226-9249-21f08991f8e1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-acad0e17-qnx5x\" (UID: \"1d7613f9-bcf1-4226-9249-21f08991f8e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-acad0e17-qnx5x"
Oct 31 12:58:35.488004 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:35.487814    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8p6m\" (UniqueName: \"kubernetes.io/projected/1d7613f9-bcf1-4226-9249-21f08991f8e1-kube-api-access-z8p6m\") pod \"maas-api-revoke-sub-acad0e17-qnx5x\" (UID: \"1d7613f9-bcf1-4226-9249-21f08991f8e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-acad0e17-qnx5x"
Oct 31 12:58:35.488004 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:35.487870    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d7613f9-bcf1-4226-9249-21f08991f8e1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-acad0e17-qnx5x\" (UID: \"1d7613f9-bcf1-4226-9249-21f08991f8e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-acad0e17-qnx5x"
Oct 31 12:58:35.488605 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:35.488577    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d7613f9-bcf1-4226-9249-21f08991f8e1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-acad0e17-qnx5x\" (UID: \"1d7613f9-bcf1-4226-9249-21f08991f8e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-acad0e17-qnx5x"
Oct 31 12:58:35.490376 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:35.490347    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d7613f9-bcf1-4226-9249-21f08991f8e1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-acad0e17-qnx5x\" (UID: \"1d7613f9-bcf1-4226-9249-21f08991f8e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-acad0e17-qnx5x"
Oct 31 12:58:35.500707 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:35.500675    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8p6m\" (UniqueName: \"kubernetes.io/projected/1d7613f9-bcf1-4226-9249-21f08991f8e1-kube-api-access-z8p6m\") pod \"maas-api-revoke-sub-acad0e17-qnx5x\" (UID: \"1d7613f9-bcf1-4226-9249-21f08991f8e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-acad0e17-qnx5x"
Oct 31 12:58:35.682027 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:35.678717    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-acad0e17-qnx5x"
Oct 31 12:58:35.990834 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:58:35.990798    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d7613f9_bcf1_4226_9249_21f08991f8e1.slice/crio-15700354100aa17f1b5c3c4854412d472d2ac406989226cf5f602d238b4f0818 WatchSource:0}: Error finding container 15700354100aa17f1b5c3c4854412d472d2ac406989226cf5f602d238b4f0818: Status 404 returned error can't find the container with id 15700354100aa17f1b5c3c4854412d472d2ac406989226cf5f602d238b4f0818
Oct 31 12:58:35.993160 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:35.993135    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-acad0e17-qnx5x"]
Oct 31 12:58:36.732973 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:36.732931    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-acad0e17-qnx5x" event={"ID":"1d7613f9-bcf1-4226-9249-21f08991f8e1","Type":"ContainerStarted","Data":"72181aa4d5b599cd120e0f81fe2adb34b0c8640625ac2c6611d87d03b36ffaf0"}
Oct 31 12:58:36.733655 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:36.733619    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-acad0e17-qnx5x" event={"ID":"1d7613f9-bcf1-4226-9249-21f08991f8e1","Type":"ContainerStarted","Data":"15700354100aa17f1b5c3c4854412d472d2ac406989226cf5f602d238b4f0818"}
Oct 31 12:58:36.752642 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:36.752584    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-acad0e17-qnx5x" podStartSLOduration=1.752568482 podStartE2EDuration="1.752568482s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:58:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:58:36.751757721 +0000 UTC m=+1879.854825433" watchObservedRunningTime="2026-10-31 12:58:36.752568482 +0000 UTC m=+1879.855636195"
Oct 31 12:58:37.742563 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:37.742530    5060 generic.go:356] "Generic (PLEG): container finished" podID="1d7613f9-bcf1-4226-9249-21f08991f8e1" containerID="72181aa4d5b599cd120e0f81fe2adb34b0c8640625ac2c6611d87d03b36ffaf0" exitCode=0
Oct 31 12:58:37.742989 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:37.742614    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-acad0e17-qnx5x" event={"ID":"1d7613f9-bcf1-4226-9249-21f08991f8e1","Type":"ContainerDied","Data":"72181aa4d5b599cd120e0f81fe2adb34b0c8640625ac2c6611d87d03b36ffaf0"}
Oct 31 12:58:38.922865 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:38.922839    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-acad0e17-qnx5x"
Oct 31 12:58:38.940761 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:38.940733    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d7613f9-bcf1-4226-9249-21f08991f8e1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d7613f9-bcf1-4226-9249-21f08991f8e1-maas-api-openshift-ca\") pod \"1d7613f9-bcf1-4226-9249-21f08991f8e1\" (UID: \"1d7613f9-bcf1-4226-9249-21f08991f8e1\") "
Oct 31 12:58:38.940964 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:38.940951    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d7613f9-bcf1-4226-9249-21f08991f8e1-kube-api-access-z8p6m\" (UniqueName: \"kubernetes.io/projected/1d7613f9-bcf1-4226-9249-21f08991f8e1-kube-api-access-z8p6m\") pod \"1d7613f9-bcf1-4226-9249-21f08991f8e1\" (UID: \"1d7613f9-bcf1-4226-9249-21f08991f8e1\") "
Oct 31 12:58:38.941044 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:38.941032    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1d7613f9-bcf1-4226-9249-21f08991f8e1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d7613f9-bcf1-4226-9249-21f08991f8e1-maas-api-xks-ca\") pod \"1d7613f9-bcf1-4226-9249-21f08991f8e1\" (UID: \"1d7613f9-bcf1-4226-9249-21f08991f8e1\") "
Oct 31 12:58:38.941085 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:38.941039    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d7613f9-bcf1-4226-9249-21f08991f8e1-maas-api-openshift-ca" pod "1d7613f9-bcf1-4226-9249-21f08991f8e1" (UID: "1d7613f9-bcf1-4226-9249-21f08991f8e1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:58:38.941377 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:38.941363    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d7613f9-bcf1-4226-9249-21f08991f8e1-maas-api-openshift-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:58:38.942949 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:38.942926    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d7613f9-bcf1-4226-9249-21f08991f8e1-maas-api-xks-ca" pod "1d7613f9-bcf1-4226-9249-21f08991f8e1" (UID: "1d7613f9-bcf1-4226-9249-21f08991f8e1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:58:38.943103 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:38.943085    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d7613f9-bcf1-4226-9249-21f08991f8e1-kube-api-access-z8p6m" pod "1d7613f9-bcf1-4226-9249-21f08991f8e1" (UID: "1d7613f9-bcf1-4226-9249-21f08991f8e1"). InnerVolumeSpecName "kube-api-access-z8p6m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:58:39.043152 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:39.043109    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8p6m\" (UniqueName: \"kubernetes.io/projected/1d7613f9-bcf1-4226-9249-21f08991f8e1-kube-api-access-z8p6m\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:58:39.043152 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:39.043145    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d7613f9-bcf1-4226-9249-21f08991f8e1-maas-api-xks-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:58:39.761203 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:39.761159    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-acad0e17-qnx5x" event={"ID":"1d7613f9-bcf1-4226-9249-21f08991f8e1","Type":"ContainerDied","Data":"15700354100aa17f1b5c3c4854412d472d2ac406989226cf5f602d238b4f0818"}
Oct 31 12:58:39.761485 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:39.761223    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="15700354100aa17f1b5c3c4854412d472d2ac406989226cf5f602d238b4f0818"
Oct 31 12:58:39.761485 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:39.761174    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-acad0e17-qnx5x"
Oct 31 12:58:40.380456 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:40.380420    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kj56r"]
Oct 31 12:58:40.380986 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:40.380634    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kj56r" podUID="120c71cc-6b6d-4833-91a1-52bc36d759e8" containerName="discovery" containerID="cri-o://4605fa9aff5b8d0a04e1ea8977eb486b17349d449855c518b62aa6255364cd85" gracePeriod=30
Oct 31 12:58:40.469065 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:40.469031    5060 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-kj56r container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.118:8080/ready\": dial tcp 10.132.0.118:8080: connect: connection refused" start-of-body=
Oct 31 12:58:40.469210 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:40.469124    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kj56r" podUID="120c71cc-6b6d-4833-91a1-52bc36d759e8" containerName="discovery" probeResult="failure" output="Get \"http://10.132.0.118:8080/ready\": dial tcp 10.132.0.118:8080: connect: connection refused"
Oct 31 12:58:40.687997 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:40.687964    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kj56r"
Oct 31 12:58:40.768205 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:40.768167    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/120c71cc-6b6d-4833-91a1-52bc36d759e8-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/120c71cc-6b6d-4833-91a1-52bc36d759e8-istio-kubeconfig\") pod \"120c71cc-6b6d-4833-91a1-52bc36d759e8\" (UID: \"120c71cc-6b6d-4833-91a1-52bc36d759e8\") "
Oct 31 12:58:40.768416 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:40.768269    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/120c71cc-6b6d-4833-91a1-52bc36d759e8-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/120c71cc-6b6d-4833-91a1-52bc36d759e8-local-certs\") pod \"120c71cc-6b6d-4833-91a1-52bc36d759e8\" (UID: \"120c71cc-6b6d-4833-91a1-52bc36d759e8\") "
Oct 31 12:58:40.768416 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:40.768336    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/120c71cc-6b6d-4833-91a1-52bc36d759e8-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/120c71cc-6b6d-4833-91a1-52bc36d759e8-istio-csr-dns-cert\") pod \"120c71cc-6b6d-4833-91a1-52bc36d759e8\" (UID: \"120c71cc-6b6d-4833-91a1-52bc36d759e8\") "
Oct 31 12:58:40.768416 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:40.768378    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/120c71cc-6b6d-4833-91a1-52bc36d759e8-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/120c71cc-6b6d-4833-91a1-52bc36d759e8-istio-csr-ca-configmap\") pod \"120c71cc-6b6d-4833-91a1-52bc36d759e8\" (UID: \"120c71cc-6b6d-4833-91a1-52bc36d759e8\") "
Oct 31 12:58:40.768766 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:40.768531    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/120c71cc-6b6d-4833-91a1-52bc36d759e8-cacerts\" (UniqueName: \"kubernetes.io/secret/120c71cc-6b6d-4833-91a1-52bc36d759e8-cacerts\") pod \"120c71cc-6b6d-4833-91a1-52bc36d759e8\" (UID: \"120c71cc-6b6d-4833-91a1-52bc36d759e8\") "
Oct 31 12:58:40.768766 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:40.768594    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/120c71cc-6b6d-4833-91a1-52bc36d759e8-istio-token\" (UniqueName: \"kubernetes.io/projected/120c71cc-6b6d-4833-91a1-52bc36d759e8-istio-token\") pod \"120c71cc-6b6d-4833-91a1-52bc36d759e8\" (UID: \"120c71cc-6b6d-4833-91a1-52bc36d759e8\") "
Oct 31 12:58:40.768766 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:40.768675    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/120c71cc-6b6d-4833-91a1-52bc36d759e8-kube-api-access-qzhxn\" (UniqueName: \"kubernetes.io/projected/120c71cc-6b6d-4833-91a1-52bc36d759e8-kube-api-access-qzhxn\") pod \"120c71cc-6b6d-4833-91a1-52bc36d759e8\" (UID: \"120c71cc-6b6d-4833-91a1-52bc36d759e8\") "
Oct 31 12:58:40.770295 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:40.770035    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/120c71cc-6b6d-4833-91a1-52bc36d759e8-istio-csr-ca-configmap" pod "120c71cc-6b6d-4833-91a1-52bc36d759e8" (UID: "120c71cc-6b6d-4833-91a1-52bc36d759e8"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:58:40.772877 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:40.772849    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/120c71cc-6b6d-4833-91a1-52bc36d759e8-local-certs" pod "120c71cc-6b6d-4833-91a1-52bc36d759e8" (UID: "120c71cc-6b6d-4833-91a1-52bc36d759e8"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:58:40.773238 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:40.773196    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/120c71cc-6b6d-4833-91a1-52bc36d759e8-kube-api-access-qzhxn" pod "120c71cc-6b6d-4833-91a1-52bc36d759e8" (UID: "120c71cc-6b6d-4833-91a1-52bc36d759e8"). InnerVolumeSpecName "kube-api-access-qzhxn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:58:40.773341 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:40.773269    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/120c71cc-6b6d-4833-91a1-52bc36d759e8-cacerts" pod "120c71cc-6b6d-4833-91a1-52bc36d759e8" (UID: "120c71cc-6b6d-4833-91a1-52bc36d759e8"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:58:40.773569 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:40.773471    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/120c71cc-6b6d-4833-91a1-52bc36d759e8-istio-kubeconfig" pod "120c71cc-6b6d-4833-91a1-52bc36d759e8" (UID: "120c71cc-6b6d-4833-91a1-52bc36d759e8"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:58:40.774620 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:40.774595    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/120c71cc-6b6d-4833-91a1-52bc36d759e8-istio-token" pod "120c71cc-6b6d-4833-91a1-52bc36d759e8" (UID: "120c71cc-6b6d-4833-91a1-52bc36d759e8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:58:40.776454 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:40.776422    5060 generic.go:356] "Generic (PLEG): container finished" podID="120c71cc-6b6d-4833-91a1-52bc36d759e8" containerID="4605fa9aff5b8d0a04e1ea8977eb486b17349d449855c518b62aa6255364cd85" exitCode=0
Oct 31 12:58:40.776606 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:40.776512    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kj56r" event={"ID":"120c71cc-6b6d-4833-91a1-52bc36d759e8","Type":"ContainerDied","Data":"4605fa9aff5b8d0a04e1ea8977eb486b17349d449855c518b62aa6255364cd85"}
Oct 31 12:58:40.776606 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:40.776560    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kj56r" event={"ID":"120c71cc-6b6d-4833-91a1-52bc36d759e8","Type":"ContainerDied","Data":"550c21bcac92762e91ca89e81bec5dfe5441d94125255396e8d09c95b02e9004"}
Oct 31 12:58:40.776606 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:40.776573    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/120c71cc-6b6d-4833-91a1-52bc36d759e8-istio-csr-dns-cert" pod "120c71cc-6b6d-4833-91a1-52bc36d759e8" (UID: "120c71cc-6b6d-4833-91a1-52bc36d759e8"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:58:40.776800 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:40.776586    5060 scope.go:122] "RemoveContainer" containerID="4605fa9aff5b8d0a04e1ea8977eb486b17349d449855c518b62aa6255364cd85"
Oct 31 12:58:40.776800 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:40.776672    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kj56r"
Oct 31 12:58:40.808557 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:40.808529    5060 scope.go:122] "RemoveContainer" containerID="4605fa9aff5b8d0a04e1ea8977eb486b17349d449855c518b62aa6255364cd85"
Oct 31 12:58:40.808890 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:58:40.808860    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4605fa9aff5b8d0a04e1ea8977eb486b17349d449855c518b62aa6255364cd85\": container with ID starting with 4605fa9aff5b8d0a04e1ea8977eb486b17349d449855c518b62aa6255364cd85 not found: ID does not exist" containerID="4605fa9aff5b8d0a04e1ea8977eb486b17349d449855c518b62aa6255364cd85"
Oct 31 12:58:40.809029 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:40.808902    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4605fa9aff5b8d0a04e1ea8977eb486b17349d449855c518b62aa6255364cd85"} err="failed to get container status \"4605fa9aff5b8d0a04e1ea8977eb486b17349d449855c518b62aa6255364cd85\": rpc error: code = NotFound desc = could not find container \"4605fa9aff5b8d0a04e1ea8977eb486b17349d449855c518b62aa6255364cd85\": container with ID starting with 4605fa9aff5b8d0a04e1ea8977eb486b17349d449855c518b62aa6255364cd85 not found: ID does not exist"
Oct 31 12:58:40.823986 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:40.822992    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kj56r"]
Oct 31 12:58:40.831833 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:40.831803    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kj56r"]
Oct 31 12:58:40.870379 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:40.870332    5060 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/120c71cc-6b6d-4833-91a1-52bc36d759e8-istio-csr-dns-cert\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:58:40.870379 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:40.870366    5060 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/120c71cc-6b6d-4833-91a1-52bc36d759e8-istio-csr-ca-configmap\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:58:40.870379 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:40.870384    5060 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/120c71cc-6b6d-4833-91a1-52bc36d759e8-cacerts\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:58:40.870815 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:40.870397    5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/120c71cc-6b6d-4833-91a1-52bc36d759e8-istio-token\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:58:40.870815 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:40.870411    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qzhxn\" (UniqueName: \"kubernetes.io/projected/120c71cc-6b6d-4833-91a1-52bc36d759e8-kube-api-access-qzhxn\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:58:40.870815 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:40.870426    5060 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/120c71cc-6b6d-4833-91a1-52bc36d759e8-istio-kubeconfig\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:58:40.870815 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:40.870441    5060 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/120c71cc-6b6d-4833-91a1-52bc36d759e8-local-certs\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:58:41.049630 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:41.049596    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="120c71cc-6b6d-4833-91a1-52bc36d759e8" path="/var/lib/kubelet/pods/120c71cc-6b6d-4833-91a1-52bc36d759e8/volumes"
Oct 31 12:58:46.009261 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:46.009223    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9b62a0-jrzjw"]
Oct 31 12:58:46.009902 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:46.009883    5060 cpu_manager.go:414] "containerMap: removing container" podUID="120c71cc-6b6d-4833-91a1-52bc36d759e8" containerName="discovery"
Oct 31 12:58:46.009902 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:46.009901    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="120c71cc-6b6d-4833-91a1-52bc36d759e8" containerName="discovery"
Oct 31 12:58:46.010030 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:46.009936    5060 cpu_manager.go:414] "containerMap: removing container" podUID="1d7613f9-bcf1-4226-9249-21f08991f8e1" containerName="revoke-keys"
Oct 31 12:58:46.010030 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:46.009942    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d7613f9-bcf1-4226-9249-21f08991f8e1" containerName="revoke-keys"
Oct 31 12:58:46.010141 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:46.010041    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d7613f9-bcf1-4226-9249-21f08991f8e1" containerName="revoke-keys"
Oct 31 12:58:46.010141 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:46.010054    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="120c71cc-6b6d-4833-91a1-52bc36d759e8" containerName="discovery"
Oct 31 12:58:46.014669 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:46.014639    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9b62a0-jrzjw"
Oct 31 12:58:46.022561 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:46.022067    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7wsmb\""
Oct 31 12:58:46.022772 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:46.022671    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 12:58:46.025531 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:46.025476    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9b62a0-jrzjw"]
Oct 31 12:58:46.060819 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:46.060471    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trm46\" (UniqueName: \"kubernetes.io/projected/1829c1e7-5a72-4204-9115-64702cc89d17-kube-api-access-trm46\") pod \"maas-api-revoke-keys-e2e-inf-a-9b62a0-jrzjw\" (UID: \"1829c1e7-5a72-4204-9115-64702cc89d17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9b62a0-jrzjw"
Oct 31 12:58:46.060819 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:46.060614    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1829c1e7-5a72-4204-9115-64702cc89d17-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-9b62a0-jrzjw\" (UID: \"1829c1e7-5a72-4204-9115-64702cc89d17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9b62a0-jrzjw"
Oct 31 12:58:46.060819 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:46.060666    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1829c1e7-5a72-4204-9115-64702cc89d17-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-9b62a0-jrzjw\" (UID: \"1829c1e7-5a72-4204-9115-64702cc89d17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9b62a0-jrzjw"
Oct 31 12:58:46.161705 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:46.161650    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trm46\" (UniqueName: \"kubernetes.io/projected/1829c1e7-5a72-4204-9115-64702cc89d17-kube-api-access-trm46\") pod \"maas-api-revoke-keys-e2e-inf-a-9b62a0-jrzjw\" (UID: \"1829c1e7-5a72-4204-9115-64702cc89d17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9b62a0-jrzjw"
Oct 31 12:58:46.161947 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:46.161755    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1829c1e7-5a72-4204-9115-64702cc89d17-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-9b62a0-jrzjw\" (UID: \"1829c1e7-5a72-4204-9115-64702cc89d17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9b62a0-jrzjw"
Oct 31 12:58:46.161947 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:46.161793    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1829c1e7-5a72-4204-9115-64702cc89d17-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-9b62a0-jrzjw\" (UID: \"1829c1e7-5a72-4204-9115-64702cc89d17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9b62a0-jrzjw"
Oct 31 12:58:46.162620 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:46.162596    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1829c1e7-5a72-4204-9115-64702cc89d17-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-9b62a0-jrzjw\" (UID: \"1829c1e7-5a72-4204-9115-64702cc89d17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9b62a0-jrzjw"
Oct 31 12:58:46.164409 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:46.164378    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1829c1e7-5a72-4204-9115-64702cc89d17-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-9b62a0-jrzjw\" (UID: \"1829c1e7-5a72-4204-9115-64702cc89d17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9b62a0-jrzjw"
Oct 31 12:58:46.174213 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:46.174187    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-trm46\" (UniqueName: \"kubernetes.io/projected/1829c1e7-5a72-4204-9115-64702cc89d17-kube-api-access-trm46\") pod \"maas-api-revoke-keys-e2e-inf-a-9b62a0-jrzjw\" (UID: \"1829c1e7-5a72-4204-9115-64702cc89d17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9b62a0-jrzjw"
Oct 31 12:58:46.341863 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:46.341826    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9b62a0-jrzjw"
Oct 31 12:58:46.639722 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:46.639688    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9b62a0-jrzjw"]
Oct 31 12:58:46.640211 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:58:46.640188    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1829c1e7_5a72_4204_9115_64702cc89d17.slice/crio-a3280fb950b1a1a1c18d1744be0757f167c45296dcaaaac32687aafd042ca23c WatchSource:0}: Error finding container a3280fb950b1a1a1c18d1744be0757f167c45296dcaaaac32687aafd042ca23c: Status 404 returned error can't find the container with id a3280fb950b1a1a1c18d1744be0757f167c45296dcaaaac32687aafd042ca23c
Oct 31 12:58:46.837198 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:46.837166    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9b62a0-jrzjw" event={"ID":"1829c1e7-5a72-4204-9115-64702cc89d17","Type":"ContainerStarted","Data":"a3280fb950b1a1a1c18d1744be0757f167c45296dcaaaac32687aafd042ca23c"}
Oct 31 12:58:47.847594 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:47.847556    5060 generic.go:356] "Generic (PLEG): container finished" podID="1829c1e7-5a72-4204-9115-64702cc89d17" containerID="e52178036598dbfb506c27779018f4fbda6e60d94ff6f8ba3f97fb2dfb0fadd7" exitCode=0
Oct 31 12:58:47.847997 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:47.847625    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9b62a0-jrzjw" event={"ID":"1829c1e7-5a72-4204-9115-64702cc89d17","Type":"ContainerDied","Data":"e52178036598dbfb506c27779018f4fbda6e60d94ff6f8ba3f97fb2dfb0fadd7"}
Oct 31 12:58:49.043364 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:49.043336    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9b62a0-jrzjw"
Oct 31 12:58:49.112671 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:49.112569    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1829c1e7-5a72-4204-9115-64702cc89d17-kube-api-access-trm46\" (UniqueName: \"kubernetes.io/projected/1829c1e7-5a72-4204-9115-64702cc89d17-kube-api-access-trm46\") pod \"1829c1e7-5a72-4204-9115-64702cc89d17\" (UID: \"1829c1e7-5a72-4204-9115-64702cc89d17\") "
Oct 31 12:58:49.112853 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:49.112714    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1829c1e7-5a72-4204-9115-64702cc89d17-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1829c1e7-5a72-4204-9115-64702cc89d17-maas-api-xks-ca\") pod \"1829c1e7-5a72-4204-9115-64702cc89d17\" (UID: \"1829c1e7-5a72-4204-9115-64702cc89d17\") "
Oct 31 12:58:49.112853 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:49.112781    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1829c1e7-5a72-4204-9115-64702cc89d17-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1829c1e7-5a72-4204-9115-64702cc89d17-maas-api-openshift-ca\") pod \"1829c1e7-5a72-4204-9115-64702cc89d17\" (UID: \"1829c1e7-5a72-4204-9115-64702cc89d17\") "
Oct 31 12:58:49.113185 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:49.113155    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1829c1e7-5a72-4204-9115-64702cc89d17-maas-api-openshift-ca" pod "1829c1e7-5a72-4204-9115-64702cc89d17" (UID: "1829c1e7-5a72-4204-9115-64702cc89d17"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:58:49.113441 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:49.113421    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1829c1e7-5a72-4204-9115-64702cc89d17-maas-api-openshift-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:58:49.115334 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:49.115304    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1829c1e7-5a72-4204-9115-64702cc89d17-kube-api-access-trm46" pod "1829c1e7-5a72-4204-9115-64702cc89d17" (UID: "1829c1e7-5a72-4204-9115-64702cc89d17"). InnerVolumeSpecName "kube-api-access-trm46". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:58:49.115669 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:49.115634    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1829c1e7-5a72-4204-9115-64702cc89d17-maas-api-xks-ca" pod "1829c1e7-5a72-4204-9115-64702cc89d17" (UID: "1829c1e7-5a72-4204-9115-64702cc89d17"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:58:49.215220 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:49.215181    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-trm46\" (UniqueName: \"kubernetes.io/projected/1829c1e7-5a72-4204-9115-64702cc89d17-kube-api-access-trm46\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:58:49.215220 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:49.215220    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1829c1e7-5a72-4204-9115-64702cc89d17-maas-api-xks-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:58:49.871109 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:49.871073    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9b62a0-jrzjw"
Oct 31 12:58:49.871109 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:49.871098    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9b62a0-jrzjw" event={"ID":"1829c1e7-5a72-4204-9115-64702cc89d17","Type":"ContainerDied","Data":"a3280fb950b1a1a1c18d1744be0757f167c45296dcaaaac32687aafd042ca23c"}
Oct 31 12:58:49.871381 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:49.871145    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a3280fb950b1a1a1c18d1744be0757f167c45296dcaaaac32687aafd042ca23c"
Oct 31 12:58:58.746999 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:58.746956    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-9b62a0-8445958dbb-99wdf"]
Oct 31 12:58:58.747528 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:58.747220    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9b62a0-8445958dbb-99wdf" podUID="9aa65f61-54d4-4e45-8f33-51ea2fe5bc56" containerName="maas-api" containerID="cri-o://ad65c2b0b7be6499f4f7a5ce027fdf7cff5ae574e08b2031866792e3c6e77401" gracePeriod=30
Oct 31 12:58:59.060071 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:59.059843    5060 generic.go:356] "Generic (PLEG): container finished" podID="9aa65f61-54d4-4e45-8f33-51ea2fe5bc56" containerID="ad65c2b0b7be6499f4f7a5ce027fdf7cff5ae574e08b2031866792e3c6e77401" exitCode=0
Oct 31 12:58:59.063579 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:59.063543    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9b62a0-8445958dbb-99wdf" event={"ID":"9aa65f61-54d4-4e45-8f33-51ea2fe5bc56","Type":"ContainerDied","Data":"ad65c2b0b7be6499f4f7a5ce027fdf7cff5ae574e08b2031866792e3c6e77401"}
Oct 31 12:58:59.171566 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:59.171542    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9b62a0-jrzjw"]
Oct 31 12:58:59.178698 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:59.178666    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9b62a0-jrzjw"]
Oct 31 12:58:59.195459 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:59.195437    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9b62a0-8445958dbb-99wdf"
Oct 31 12:58:59.304089 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:59.304052    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9aa65f61-54d4-4e45-8f33-51ea2fe5bc56-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9aa65f61-54d4-4e45-8f33-51ea2fe5bc56-openshift-service-ca\") pod \"9aa65f61-54d4-4e45-8f33-51ea2fe5bc56\" (UID: \"9aa65f61-54d4-4e45-8f33-51ea2fe5bc56\") "
Oct 31 12:58:59.304340 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:59.304152    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9aa65f61-54d4-4e45-8f33-51ea2fe5bc56-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9aa65f61-54d4-4e45-8f33-51ea2fe5bc56-metrics-certs\") pod \"9aa65f61-54d4-4e45-8f33-51ea2fe5bc56\" (UID: \"9aa65f61-54d4-4e45-8f33-51ea2fe5bc56\") "
Oct 31 12:58:59.304340 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:59.304208    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9aa65f61-54d4-4e45-8f33-51ea2fe5bc56-kube-api-access-x8vbn\" (UniqueName: \"kubernetes.io/projected/9aa65f61-54d4-4e45-8f33-51ea2fe5bc56-kube-api-access-x8vbn\") pod \"9aa65f61-54d4-4e45-8f33-51ea2fe5bc56\" (UID: \"9aa65f61-54d4-4e45-8f33-51ea2fe5bc56\") "
Oct 31 12:58:59.304514 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:59.304347    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9aa65f61-54d4-4e45-8f33-51ea2fe5bc56-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9aa65f61-54d4-4e45-8f33-51ea2fe5bc56-maas-api-tls\") pod \"9aa65f61-54d4-4e45-8f33-51ea2fe5bc56\" (UID: \"9aa65f61-54d4-4e45-8f33-51ea2fe5bc56\") "
Oct 31 12:58:59.305507 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:59.305458    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9aa65f61-54d4-4e45-8f33-51ea2fe5bc56-openshift-service-ca" pod "9aa65f61-54d4-4e45-8f33-51ea2fe5bc56" (UID: "9aa65f61-54d4-4e45-8f33-51ea2fe5bc56"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:58:59.326963 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:59.326889    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9aa65f61-54d4-4e45-8f33-51ea2fe5bc56-metrics-certs" pod "9aa65f61-54d4-4e45-8f33-51ea2fe5bc56" (UID: "9aa65f61-54d4-4e45-8f33-51ea2fe5bc56"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:58:59.327080 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:59.326951    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9aa65f61-54d4-4e45-8f33-51ea2fe5bc56-kube-api-access-x8vbn" pod "9aa65f61-54d4-4e45-8f33-51ea2fe5bc56" (UID: "9aa65f61-54d4-4e45-8f33-51ea2fe5bc56"). InnerVolumeSpecName "kube-api-access-x8vbn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:58:59.328587 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:59.328561    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9aa65f61-54d4-4e45-8f33-51ea2fe5bc56-maas-api-tls" pod "9aa65f61-54d4-4e45-8f33-51ea2fe5bc56" (UID: "9aa65f61-54d4-4e45-8f33-51ea2fe5bc56"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:58:59.406253 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:59.406212    5060 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9aa65f61-54d4-4e45-8f33-51ea2fe5bc56-metrics-certs\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:58:59.406253 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:59.406250    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x8vbn\" (UniqueName: \"kubernetes.io/projected/9aa65f61-54d4-4e45-8f33-51ea2fe5bc56-kube-api-access-x8vbn\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:58:59.406253 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:59.406260    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9aa65f61-54d4-4e45-8f33-51ea2fe5bc56-maas-api-tls\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:58:59.406253 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:58:59.406269    5060 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9aa65f61-54d4-4e45-8f33-51ea2fe5bc56-openshift-service-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:58:59.921901 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:58:59.921863    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7phb" containerName="extractor"
Oct 31 12:59:00.104376 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:00.104344    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9b62a0-8445958dbb-99wdf"
Oct 31 12:59:00.104785 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:00.104438    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9b62a0-8445958dbb-99wdf" event={"ID":"9aa65f61-54d4-4e45-8f33-51ea2fe5bc56","Type":"ContainerDied","Data":"662c02e3a518a464c7902ab0551f07e2bf9ab4ffd61e1e960b511d4ee6418d60"}
Oct 31 12:59:00.104927 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:00.104832    5060 scope.go:122] "RemoveContainer" containerID="ad65c2b0b7be6499f4f7a5ce027fdf7cff5ae574e08b2031866792e3c6e77401"
Oct 31 12:59:00.138345 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:00.137688    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-9b62a0-8445958dbb-99wdf"]
Oct 31 12:59:00.142272 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:00.142245    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-9b62a0-8445958dbb-99wdf"]
Oct 31 12:59:00.273570 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:00.273536    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d176a5-57f6cd6df5-tzc5c"]
Oct 31 12:59:01.050086 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:01.050035    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1829c1e7-5a72-4204-9115-64702cc89d17" path="/var/lib/kubelet/pods/1829c1e7-5a72-4204-9115-64702cc89d17/volumes"
Oct 31 12:59:01.050613 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:01.050511    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9aa65f61-54d4-4e45-8f33-51ea2fe5bc56" path="/var/lib/kubelet/pods/9aa65f61-54d4-4e45-8f33-51ea2fe5bc56/volumes"
Oct 31 12:59:01.560574 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:01.560540    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d086f3-58b5756f6f-m4jx8"]
Oct 31 12:59:01.560790 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:01.560747    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d086f3-58b5756f6f-m4jx8" podUID="e7971b7e-6267-4ce1-ac72-38be9977a7b4" containerName="maas-api" containerID="cri-o://810ebd8e03ac0a6c7cf27d578dd6cfc5e90a378312a9ea1517ecd99441b26882" gracePeriod=30
Oct 31 12:59:02.034270 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:02.034238    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d086f3-58b5756f6f-m4jx8"
Oct 31 12:59:02.124087 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:02.124047    5060 generic.go:356] "Generic (PLEG): container finished" podID="e7971b7e-6267-4ce1-ac72-38be9977a7b4" containerID="810ebd8e03ac0a6c7cf27d578dd6cfc5e90a378312a9ea1517ecd99441b26882" exitCode=0
Oct 31 12:59:02.124087 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:02.124086    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d086f3-58b5756f6f-m4jx8" event={"ID":"e7971b7e-6267-4ce1-ac72-38be9977a7b4","Type":"ContainerDied","Data":"810ebd8e03ac0a6c7cf27d578dd6cfc5e90a378312a9ea1517ecd99441b26882"}
Oct 31 12:59:02.124793 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:02.124129    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d086f3-58b5756f6f-m4jx8"
Oct 31 12:59:02.124793 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:02.124160    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d086f3-58b5756f6f-m4jx8" event={"ID":"e7971b7e-6267-4ce1-ac72-38be9977a7b4","Type":"ContainerDied","Data":"91c6cc8bce0e150c82ee97f0e25e45f1467fa52263a0af4fdca5de14666d2043"}
Oct 31 12:59:02.124793 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:02.124189    5060 scope.go:122] "RemoveContainer" containerID="810ebd8e03ac0a6c7cf27d578dd6cfc5e90a378312a9ea1517ecd99441b26882"
Oct 31 12:59:02.138621 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:02.138599    5060 scope.go:122] "RemoveContainer" containerID="810ebd8e03ac0a6c7cf27d578dd6cfc5e90a378312a9ea1517ecd99441b26882"
Oct 31 12:59:02.138955 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:59:02.138926    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"810ebd8e03ac0a6c7cf27d578dd6cfc5e90a378312a9ea1517ecd99441b26882\": container with ID starting with 810ebd8e03ac0a6c7cf27d578dd6cfc5e90a378312a9ea1517ecd99441b26882 not found: ID does not exist" containerID="810ebd8e03ac0a6c7cf27d578dd6cfc5e90a378312a9ea1517ecd99441b26882"
Oct 31 12:59:02.139056 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:02.138969    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"810ebd8e03ac0a6c7cf27d578dd6cfc5e90a378312a9ea1517ecd99441b26882"} err="failed to get container status \"810ebd8e03ac0a6c7cf27d578dd6cfc5e90a378312a9ea1517ecd99441b26882\": rpc error: code = NotFound desc = could not find container \"810ebd8e03ac0a6c7cf27d578dd6cfc5e90a378312a9ea1517ecd99441b26882\": container with ID starting with 810ebd8e03ac0a6c7cf27d578dd6cfc5e90a378312a9ea1517ecd99441b26882 not found: ID does not exist"
Oct 31 12:59:02.156214 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:02.156190    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7971b7e-6267-4ce1-ac72-38be9977a7b4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7971b7e-6267-4ce1-ac72-38be9977a7b4-openshift-service-ca\") pod \"e7971b7e-6267-4ce1-ac72-38be9977a7b4\" (UID: \"e7971b7e-6267-4ce1-ac72-38be9977a7b4\") "
Oct 31 12:59:02.156342 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:02.156223    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7971b7e-6267-4ce1-ac72-38be9977a7b4-kube-api-access-dr926\" (UniqueName: \"kubernetes.io/projected/e7971b7e-6267-4ce1-ac72-38be9977a7b4-kube-api-access-dr926\") pod \"e7971b7e-6267-4ce1-ac72-38be9977a7b4\" (UID: \"e7971b7e-6267-4ce1-ac72-38be9977a7b4\") "
Oct 31 12:59:02.156342 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:02.156319    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e7971b7e-6267-4ce1-ac72-38be9977a7b4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7971b7e-6267-4ce1-ac72-38be9977a7b4-metrics-certs\") pod \"e7971b7e-6267-4ce1-ac72-38be9977a7b4\" (UID: \"e7971b7e-6267-4ce1-ac72-38be9977a7b4\") "
Oct 31 12:59:02.156775 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:02.156440    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e7971b7e-6267-4ce1-ac72-38be9977a7b4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e7971b7e-6267-4ce1-ac72-38be9977a7b4-maas-api-tls\") pod \"e7971b7e-6267-4ce1-ac72-38be9977a7b4\" (UID: \"e7971b7e-6267-4ce1-ac72-38be9977a7b4\") "
Oct 31 12:59:02.156775 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:02.156621    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7971b7e-6267-4ce1-ac72-38be9977a7b4-openshift-service-ca" pod "e7971b7e-6267-4ce1-ac72-38be9977a7b4" (UID: "e7971b7e-6267-4ce1-ac72-38be9977a7b4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:59:02.156992 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:02.156969    5060 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7971b7e-6267-4ce1-ac72-38be9977a7b4-openshift-service-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:02.158759 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:02.158733    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7971b7e-6267-4ce1-ac72-38be9977a7b4-metrics-certs" pod "e7971b7e-6267-4ce1-ac72-38be9977a7b4" (UID: "e7971b7e-6267-4ce1-ac72-38be9977a7b4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:59:02.158845 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:02.158793    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7971b7e-6267-4ce1-ac72-38be9977a7b4-maas-api-tls" pod "e7971b7e-6267-4ce1-ac72-38be9977a7b4" (UID: "e7971b7e-6267-4ce1-ac72-38be9977a7b4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:59:02.158942 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:02.158926    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7971b7e-6267-4ce1-ac72-38be9977a7b4-kube-api-access-dr926" pod "e7971b7e-6267-4ce1-ac72-38be9977a7b4" (UID: "e7971b7e-6267-4ce1-ac72-38be9977a7b4"). InnerVolumeSpecName "kube-api-access-dr926". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:59:02.258899 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:02.258814    5060 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7971b7e-6267-4ce1-ac72-38be9977a7b4-metrics-certs\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:02.258899 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:02.258846    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e7971b7e-6267-4ce1-ac72-38be9977a7b4-maas-api-tls\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:02.258899 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:02.258856    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dr926\" (UniqueName: \"kubernetes.io/projected/e7971b7e-6267-4ce1-ac72-38be9977a7b4-kube-api-access-dr926\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:02.451726 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:02.451688    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d086f3-58b5756f6f-m4jx8"]
Oct 31 12:59:02.455802 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:02.455771    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d086f3-58b5756f6f-m4jx8"]
Oct 31 12:59:03.031610 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:03.031574    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1e071a8-vqh87"]
Oct 31 12:59:03.038342 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:03.038309    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d1721ec-dq9xk"]
Oct 31 12:59:03.049005 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:03.048976    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7971b7e-6267-4ce1-ac72-38be9977a7b4" path="/var/lib/kubelet/pods/e7971b7e-6267-4ce1-ac72-38be9977a7b4/volumes"
Oct 31 12:59:03.049317 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:03.049303    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1e071a8-vqh87"]
Oct 31 12:59:03.049359 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:03.049332    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d1721ec-dq9xk"]
Oct 31 12:59:05.066510 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:05.058776    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb1bcc49-26c3-46a9-a134-dffba7b91a58" path="/var/lib/kubelet/pods/bb1bcc49-26c3-46a9-a134-dffba7b91a58/volumes"
Oct 31 12:59:05.066510 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:05.063076    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc885fe1-b05b-48a3-9617-2d9475ec888a" path="/var/lib/kubelet/pods/fc885fe1-b05b-48a3-9617-2d9475ec888a/volumes"
Oct 31 12:59:05.275307 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:05.275261    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d176a5-57f6cd6df5-tzc5c" podUID="40133e45-b954-420a-aa2c-8e5b9c873ef4" containerName="payload-pre-processing" containerID="cri-o://1e156eee88d8e158f3ae297081e63eee06fef2459d71a4f8ade2c3ec2e726d39" gracePeriod=25
Oct 31 12:59:07.170516 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:07.170466    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-9b62a0-openshift-default-6f648468b7-g7tch"]
Oct 31 12:59:07.171092 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:07.170718    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-9b62a0-openshift-default-6f648468b7-g7tch" podUID="6ac598e5-c4d3-4aa5-ac25-3bcbd0b3c968" containerName="istio-proxy" containerID="cri-o://fd1d1f5f6aaec977ef4430afe8bb838bcc102f2b3ecd2213540edfbe34e1a1f8" gracePeriod=30
Oct 31 12:59:07.683614 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:07.683580    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2c393f-gq4fj"]
Oct 31 12:59:07.684208 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:07.684187    5060 cpu_manager.go:414] "containerMap: removing container" podUID="e7971b7e-6267-4ce1-ac72-38be9977a7b4" containerName="maas-api"
Oct 31 12:59:07.684208 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:07.684207    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7971b7e-6267-4ce1-ac72-38be9977a7b4" containerName="maas-api"
Oct 31 12:59:07.684412 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:07.684276    5060 cpu_manager.go:414] "containerMap: removing container" podUID="1829c1e7-5a72-4204-9115-64702cc89d17" containerName="revoke-keys"
Oct 31 12:59:07.684412 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:07.684285    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1829c1e7-5a72-4204-9115-64702cc89d17" containerName="revoke-keys"
Oct 31 12:59:07.684412 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:07.684309    5060 cpu_manager.go:414] "containerMap: removing container" podUID="9aa65f61-54d4-4e45-8f33-51ea2fe5bc56" containerName="maas-api"
Oct 31 12:59:07.684412 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:07.684317    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9aa65f61-54d4-4e45-8f33-51ea2fe5bc56" containerName="maas-api"
Oct 31 12:59:07.684674 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:07.684511    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="9aa65f61-54d4-4e45-8f33-51ea2fe5bc56" containerName="maas-api"
Oct 31 12:59:07.684674 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:07.684534    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7971b7e-6267-4ce1-ac72-38be9977a7b4" containerName="maas-api"
Oct 31 12:59:07.684674 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:07.684555    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="1829c1e7-5a72-4204-9115-64702cc89d17" containerName="revoke-keys"
Oct 31 12:59:07.690351 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:07.690329    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2c393f-gq4fj"
Oct 31 12:59:07.695545 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:07.695519    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 12:59:07.696214 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:07.696178    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7wsmb\""
Oct 31 12:59:07.698188 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:07.698161    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2c393f-gq4fj"]
Oct 31 12:59:07.751786 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:07.751755    5060 patch_prober.go:29] interesting pod/e2e-inf-a-9b62a0-openshift-default-6f648468b7-g7tch container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 12:59:07.751941 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:07.751813    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-9b62a0-openshift-default-6f648468b7-g7tch" podUID="6ac598e5-c4d3-4aa5-ac25-3bcbd0b3c968" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 12:59:07.840182 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:07.840145    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zd72z\" (UniqueName: \"kubernetes.io/projected/545ba362-f42e-48b4-aa04-afe3e25401cb-kube-api-access-zd72z\") pod \"maas-api-revoke-keys-e2e-inf-b-2c393f-gq4fj\" (UID: \"545ba362-f42e-48b4-aa04-afe3e25401cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2c393f-gq4fj"
Oct 31 12:59:07.840412 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:07.840386    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/545ba362-f42e-48b4-aa04-afe3e25401cb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-2c393f-gq4fj\" (UID: \"545ba362-f42e-48b4-aa04-afe3e25401cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2c393f-gq4fj"
Oct 31 12:59:07.840548 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:07.840443    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/545ba362-f42e-48b4-aa04-afe3e25401cb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-2c393f-gq4fj\" (UID: \"545ba362-f42e-48b4-aa04-afe3e25401cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2c393f-gq4fj"
Oct 31 12:59:07.941661 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:07.941577    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/545ba362-f42e-48b4-aa04-afe3e25401cb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-2c393f-gq4fj\" (UID: \"545ba362-f42e-48b4-aa04-afe3e25401cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2c393f-gq4fj"
Oct 31 12:59:07.941661 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:07.941626    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/545ba362-f42e-48b4-aa04-afe3e25401cb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-2c393f-gq4fj\" (UID: \"545ba362-f42e-48b4-aa04-afe3e25401cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2c393f-gq4fj"
Oct 31 12:59:07.941898 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:07.941706    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zd72z\" (UniqueName: \"kubernetes.io/projected/545ba362-f42e-48b4-aa04-afe3e25401cb-kube-api-access-zd72z\") pod \"maas-api-revoke-keys-e2e-inf-b-2c393f-gq4fj\" (UID: \"545ba362-f42e-48b4-aa04-afe3e25401cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2c393f-gq4fj"
Oct 31 12:59:07.942319 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:07.942291    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/545ba362-f42e-48b4-aa04-afe3e25401cb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-2c393f-gq4fj\" (UID: \"545ba362-f42e-48b4-aa04-afe3e25401cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2c393f-gq4fj"
Oct 31 12:59:07.944183 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:07.944151    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/545ba362-f42e-48b4-aa04-afe3e25401cb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-2c393f-gq4fj\" (UID: \"545ba362-f42e-48b4-aa04-afe3e25401cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2c393f-gq4fj"
Oct 31 12:59:07.955087 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:07.955057    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zd72z\" (UniqueName: \"kubernetes.io/projected/545ba362-f42e-48b4-aa04-afe3e25401cb-kube-api-access-zd72z\") pod \"maas-api-revoke-keys-e2e-inf-b-2c393f-gq4fj\" (UID: \"545ba362-f42e-48b4-aa04-afe3e25401cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2c393f-gq4fj"
Oct 31 12:59:08.018529 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:08.018163    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2c393f-gq4fj"
Oct 31 12:59:08.360065 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:59:08.360027    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod545ba362_f42e_48b4_aa04_afe3e25401cb.slice/crio-feb725149bf7dd3a2ba04b4c5eb6967f7e4b6745882f7c089996df7a16bf9058 WatchSource:0}: Error finding container feb725149bf7dd3a2ba04b4c5eb6967f7e4b6745882f7c089996df7a16bf9058: Status 404 returned error can't find the container with id feb725149bf7dd3a2ba04b4c5eb6967f7e4b6745882f7c089996df7a16bf9058
Oct 31 12:59:08.369458 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:08.369423    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2c393f-gq4fj"]
Oct 31 12:59:09.210060 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:09.210023    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2c393f-gq4fj" event={"ID":"545ba362-f42e-48b4-aa04-afe3e25401cb","Type":"ContainerStarted","Data":"628a065de5c8ef62178bcfe61d7da9a579e462ff23678c576911709d365a628e"}
Oct 31 12:59:09.210272 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:09.210087    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2c393f-gq4fj" event={"ID":"545ba362-f42e-48b4-aa04-afe3e25401cb","Type":"ContainerStarted","Data":"feb725149bf7dd3a2ba04b4c5eb6967f7e4b6745882f7c089996df7a16bf9058"}
Oct 31 12:59:09.242828 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:09.242614    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2c393f-gq4fj" podStartSLOduration=2.242589818 podStartE2EDuration="2.242589818s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:59:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:59:09.22902552 +0000 UTC m=+1912.332093232" watchObservedRunningTime="2026-10-31 12:59:09.242589818 +0000 UTC m=+1912.345657531"
Oct 31 12:59:10.124447 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:10.124406    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-pz8cf"]
Oct 31 12:59:10.221151 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:10.221112    5060 generic.go:356] "Generic (PLEG): container finished" podID="545ba362-f42e-48b4-aa04-afe3e25401cb" containerID="628a065de5c8ef62178bcfe61d7da9a579e462ff23678c576911709d365a628e" exitCode=0
Oct 31 12:59:10.221327 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:10.221154    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2c393f-gq4fj" event={"ID":"545ba362-f42e-48b4-aa04-afe3e25401cb","Type":"ContainerDied","Data":"628a065de5c8ef62178bcfe61d7da9a579e462ff23678c576911709d365a628e"}
Oct 31 12:59:10.221575 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:10.221553    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-pz8cf" podUID="326d88b8-e041-4b4f-ab99-f1d2622d76b4" containerName="istio-proxy" containerID="cri-o://adf65d533598e86aae889d3f711cf879259eb58f2663d14215861777e4ff8ad3" gracePeriod=30
Oct 31 12:59:11.101976 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:11.101932    5060 patch_prober.go:29] interesting pod/e2e-worker-w6-67d410-openshift-default-676b978549-pz8cf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 12:59:11.102300 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:11.102013    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-pz8cf" podUID="326d88b8-e041-4b4f-ab99-f1d2622d76b4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 12:59:11.243264 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:11.243167    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-nzrl9"]
Oct 31 12:59:11.243804 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:11.243423    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-nzrl9" podUID="b1ac43d5-eb68-45aa-9e8e-268d7c6355dd" containerName="istio-proxy" containerID="cri-o://8f91d4f726d41d37d7125a8084d62df37bf466c88fd28b02196b5aeff9951a2e" gracePeriod=30
Oct 31 12:59:11.250768 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:11.250714    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-s9nqq"]
Oct 31 12:59:11.252070 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:11.251841    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-s9nqq" podUID="ebcf940c-4d23-4944-9948-d3dea5ba99f6" containerName="istio-proxy" containerID="cri-o://36995cc9e7366968dd91f78614b082c5381b377037cf4d2be4495625365cb598" gracePeriod=30
Oct 31 12:59:11.263850 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:11.263823    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-7stx7"]
Oct 31 12:59:11.264229 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:11.264204    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-7stx7" podUID="25899808-1323-425c-9359-3b56f99ad5cb" containerName="istio-proxy" containerID="cri-o://0e99ef78dd089b9c1a0dba68109675b3b4434a15d41edf16b2cd47366ecb78b9" gracePeriod=30
Oct 31 12:59:11.534064 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:11.534036    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2c393f-gq4fj"
Oct 31 12:59:11.720638 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:11.720559    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/545ba362-f42e-48b4-aa04-afe3e25401cb-kube-api-access-zd72z\" (UniqueName: \"kubernetes.io/projected/545ba362-f42e-48b4-aa04-afe3e25401cb-kube-api-access-zd72z\") pod \"545ba362-f42e-48b4-aa04-afe3e25401cb\" (UID: \"545ba362-f42e-48b4-aa04-afe3e25401cb\") "
Oct 31 12:59:11.720638 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:11.720636    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/545ba362-f42e-48b4-aa04-afe3e25401cb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/545ba362-f42e-48b4-aa04-afe3e25401cb-maas-api-openshift-ca\") pod \"545ba362-f42e-48b4-aa04-afe3e25401cb\" (UID: \"545ba362-f42e-48b4-aa04-afe3e25401cb\") "
Oct 31 12:59:11.720866 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:11.720661    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/545ba362-f42e-48b4-aa04-afe3e25401cb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/545ba362-f42e-48b4-aa04-afe3e25401cb-maas-api-xks-ca\") pod \"545ba362-f42e-48b4-aa04-afe3e25401cb\" (UID: \"545ba362-f42e-48b4-aa04-afe3e25401cb\") "
Oct 31 12:59:11.721155 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:11.721120    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/545ba362-f42e-48b4-aa04-afe3e25401cb-maas-api-openshift-ca" pod "545ba362-f42e-48b4-aa04-afe3e25401cb" (UID: "545ba362-f42e-48b4-aa04-afe3e25401cb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:59:11.723038 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:11.723010    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/545ba362-f42e-48b4-aa04-afe3e25401cb-kube-api-access-zd72z" pod "545ba362-f42e-48b4-aa04-afe3e25401cb" (UID: "545ba362-f42e-48b4-aa04-afe3e25401cb"). InnerVolumeSpecName "kube-api-access-zd72z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:59:11.723038 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:11.723019    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/545ba362-f42e-48b4-aa04-afe3e25401cb-maas-api-xks-ca" pod "545ba362-f42e-48b4-aa04-afe3e25401cb" (UID: "545ba362-f42e-48b4-aa04-afe3e25401cb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:59:11.743004 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:11.742972    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-71bc05-fsl95"]
Oct 31 12:59:11.743627 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:11.743609    5060 cpu_manager.go:414] "containerMap: removing container" podUID="545ba362-f42e-48b4-aa04-afe3e25401cb" containerName="revoke-keys"
Oct 31 12:59:11.743730 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:11.743625    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="545ba362-f42e-48b4-aa04-afe3e25401cb" containerName="revoke-keys"
Oct 31 12:59:11.743792 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:11.743755    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="545ba362-f42e-48b4-aa04-afe3e25401cb" containerName="revoke-keys"
Oct 31 12:59:11.747574 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:11.747556    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-71bc05-fsl95"
Oct 31 12:59:11.768039 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:11.768007    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-71bc05-fsl95"]
Oct 31 12:59:11.822602 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:11.822569    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zd72z\" (UniqueName: \"kubernetes.io/projected/545ba362-f42e-48b4-aa04-afe3e25401cb-kube-api-access-zd72z\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:11.822602 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:11.822600    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/545ba362-f42e-48b4-aa04-afe3e25401cb-maas-api-openshift-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:11.822602 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:11.822610    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/545ba362-f42e-48b4-aa04-afe3e25401cb-maas-api-xks-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:11.924071 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:11.924032    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49c1b7fd-fcf1-494e-b03a-321bb70d4151-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-71bc05-fsl95\" (UID: \"49c1b7fd-fcf1-494e-b03a-321bb70d4151\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-71bc05-fsl95"
Oct 31 12:59:11.924230 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:11.924163    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49c1b7fd-fcf1-494e-b03a-321bb70d4151-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-71bc05-fsl95\" (UID: \"49c1b7fd-fcf1-494e-b03a-321bb70d4151\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-71bc05-fsl95"
Oct 31 12:59:11.924376 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:11.924360    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lx4xt\" (UniqueName: \"kubernetes.io/projected/49c1b7fd-fcf1-494e-b03a-321bb70d4151-kube-api-access-lx4xt\") pod \"maas-api-revoke-keys-e2e-inf-b-71bc05-fsl95\" (UID: \"49c1b7fd-fcf1-494e-b03a-321bb70d4151\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-71bc05-fsl95"
Oct 31 12:59:12.026107 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:12.026070    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lx4xt\" (UniqueName: \"kubernetes.io/projected/49c1b7fd-fcf1-494e-b03a-321bb70d4151-kube-api-access-lx4xt\") pod \"maas-api-revoke-keys-e2e-inf-b-71bc05-fsl95\" (UID: \"49c1b7fd-fcf1-494e-b03a-321bb70d4151\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-71bc05-fsl95"
Oct 31 12:59:12.026268 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:12.026116    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49c1b7fd-fcf1-494e-b03a-321bb70d4151-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-71bc05-fsl95\" (UID: \"49c1b7fd-fcf1-494e-b03a-321bb70d4151\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-71bc05-fsl95"
Oct 31 12:59:12.026268 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:12.026178    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49c1b7fd-fcf1-494e-b03a-321bb70d4151-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-71bc05-fsl95\" (UID: \"49c1b7fd-fcf1-494e-b03a-321bb70d4151\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-71bc05-fsl95"
Oct 31 12:59:12.026874 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:12.026848    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49c1b7fd-fcf1-494e-b03a-321bb70d4151-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-71bc05-fsl95\" (UID: \"49c1b7fd-fcf1-494e-b03a-321bb70d4151\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-71bc05-fsl95"
Oct 31 12:59:12.028581 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:12.028415    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49c1b7fd-fcf1-494e-b03a-321bb70d4151-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-71bc05-fsl95\" (UID: \"49c1b7fd-fcf1-494e-b03a-321bb70d4151\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-71bc05-fsl95"
Oct 31 12:59:12.038181 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:12.038154    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lx4xt\" (UniqueName: \"kubernetes.io/projected/49c1b7fd-fcf1-494e-b03a-321bb70d4151-kube-api-access-lx4xt\") pod \"maas-api-revoke-keys-e2e-inf-b-71bc05-fsl95\" (UID: \"49c1b7fd-fcf1-494e-b03a-321bb70d4151\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-71bc05-fsl95"
Oct 31 12:59:12.076764 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:12.076726    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-71bc05-fsl95"
Oct 31 12:59:12.264315 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:12.264287    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2c393f-gq4fj"
Oct 31 12:59:12.264728 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:12.264316    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2c393f-gq4fj" event={"ID":"545ba362-f42e-48b4-aa04-afe3e25401cb","Type":"ContainerDied","Data":"feb725149bf7dd3a2ba04b4c5eb6967f7e4b6745882f7c089996df7a16bf9058"}
Oct 31 12:59:12.264728 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:12.264378    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="feb725149bf7dd3a2ba04b4c5eb6967f7e4b6745882f7c089996df7a16bf9058"
Oct 31 12:59:12.370717 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:12.369354    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-71bc05-fsl95"]
Oct 31 12:59:12.443616 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:12.443572    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9b62a0-openshift-default-6f648468b7-g7tch"
Oct 31 12:59:12.540268 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:12.540171    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ac598e5-c4d3-4aa5-ac25-3bcbd0b3c968-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ac598e5-c4d3-4aa5-ac25-3bcbd0b3c968-istio-envoy\") pod \"6ac598e5-c4d3-4aa5-ac25-3bcbd0b3c968\" (UID: \"6ac598e5-c4d3-4aa5-ac25-3bcbd0b3c968\") "
Oct 31 12:59:12.540426 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:12.540282    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ac598e5-c4d3-4aa5-ac25-3bcbd0b3c968-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ac598e5-c4d3-4aa5-ac25-3bcbd0b3c968-workload-certs\") pod \"6ac598e5-c4d3-4aa5-ac25-3bcbd0b3c968\" (UID: \"6ac598e5-c4d3-4aa5-ac25-3bcbd0b3c968\") "
Oct 31 12:59:12.540426 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:12.540394    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ac598e5-c4d3-4aa5-ac25-3bcbd0b3c968-kube-api-access-8765f\" (UniqueName: \"kubernetes.io/projected/6ac598e5-c4d3-4aa5-ac25-3bcbd0b3c968-kube-api-access-8765f\") pod \"6ac598e5-c4d3-4aa5-ac25-3bcbd0b3c968\" (UID: \"6ac598e5-c4d3-4aa5-ac25-3bcbd0b3c968\") "
Oct 31 12:59:12.540647 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:12.540437    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ac598e5-c4d3-4aa5-ac25-3bcbd0b3c968-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ac598e5-c4d3-4aa5-ac25-3bcbd0b3c968-workload-socket\") pod \"6ac598e5-c4d3-4aa5-ac25-3bcbd0b3c968\" (UID: \"6ac598e5-c4d3-4aa5-ac25-3bcbd0b3c968\") "
Oct 31 12:59:12.540647 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:12.540524    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ac598e5-c4d3-4aa5-ac25-3bcbd0b3c968-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ac598e5-c4d3-4aa5-ac25-3bcbd0b3c968-credential-socket\") pod \"6ac598e5-c4d3-4aa5-ac25-3bcbd0b3c968\" (UID: \"6ac598e5-c4d3-4aa5-ac25-3bcbd0b3c968\") "
Oct 31 12:59:12.540772 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:12.540713    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ac598e5-c4d3-4aa5-ac25-3bcbd0b3c968-istio-token\" (UniqueName: \"kubernetes.io/projected/6ac598e5-c4d3-4aa5-ac25-3bcbd0b3c968-istio-token\") pod \"6ac598e5-c4d3-4aa5-ac25-3bcbd0b3c968\" (UID: \"6ac598e5-c4d3-4aa5-ac25-3bcbd0b3c968\") "
Oct 31 12:59:12.540924 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:12.540887    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ac598e5-c4d3-4aa5-ac25-3bcbd0b3c968-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ac598e5-c4d3-4aa5-ac25-3bcbd0b3c968-istio-data\") pod \"6ac598e5-c4d3-4aa5-ac25-3bcbd0b3c968\" (UID: \"6ac598e5-c4d3-4aa5-ac25-3bcbd0b3c968\") "
Oct 31 12:59:12.542602 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:12.541131    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ac598e5-c4d3-4aa5-ac25-3bcbd0b3c968-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ac598e5-c4d3-4aa5-ac25-3bcbd0b3c968-istiod-ca-cert\") pod \"6ac598e5-c4d3-4aa5-ac25-3bcbd0b3c968\" (UID: \"6ac598e5-c4d3-4aa5-ac25-3bcbd0b3c968\") "
Oct 31 12:59:12.543406 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:12.541467    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ac598e5-c4d3-4aa5-ac25-3bcbd0b3c968-credential-socket" pod "6ac598e5-c4d3-4aa5-ac25-3bcbd0b3c968" (UID: "6ac598e5-c4d3-4aa5-ac25-3bcbd0b3c968"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:59:12.543406 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:12.541673    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ac598e5-c4d3-4aa5-ac25-3bcbd0b3c968-workload-socket" pod "6ac598e5-c4d3-4aa5-ac25-3bcbd0b3c968" (UID: "6ac598e5-c4d3-4aa5-ac25-3bcbd0b3c968"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:59:12.546822 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:12.546788    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ac598e5-c4d3-4aa5-ac25-3bcbd0b3c968-workload-certs" pod "6ac598e5-c4d3-4aa5-ac25-3bcbd0b3c968" (UID: "6ac598e5-c4d3-4aa5-ac25-3bcbd0b3c968"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:59:12.547165 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:12.547141    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ac598e5-c4d3-4aa5-ac25-3bcbd0b3c968-istio-data" pod "6ac598e5-c4d3-4aa5-ac25-3bcbd0b3c968" (UID: "6ac598e5-c4d3-4aa5-ac25-3bcbd0b3c968"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:59:12.547445 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:12.547426    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ac598e5-c4d3-4aa5-ac25-3bcbd0b3c968-istiod-ca-cert" pod "6ac598e5-c4d3-4aa5-ac25-3bcbd0b3c968" (UID: "6ac598e5-c4d3-4aa5-ac25-3bcbd0b3c968"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:59:12.550414 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:12.550387    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ac598e5-c4d3-4aa5-ac25-3bcbd0b3c968-istio-envoy" pod "6ac598e5-c4d3-4aa5-ac25-3bcbd0b3c968" (UID: "6ac598e5-c4d3-4aa5-ac25-3bcbd0b3c968"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:59:12.551878 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:12.551848    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ac598e5-c4d3-4aa5-ac25-3bcbd0b3c968-istio-token" pod "6ac598e5-c4d3-4aa5-ac25-3bcbd0b3c968" (UID: "6ac598e5-c4d3-4aa5-ac25-3bcbd0b3c968"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:59:12.552093 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:12.552069    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ac598e5-c4d3-4aa5-ac25-3bcbd0b3c968-kube-api-access-8765f" pod "6ac598e5-c4d3-4aa5-ac25-3bcbd0b3c968" (UID: "6ac598e5-c4d3-4aa5-ac25-3bcbd0b3c968"). InnerVolumeSpecName "kube-api-access-8765f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:59:12.644481 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:12.644441    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6ac598e5-c4d3-4aa5-ac25-3bcbd0b3c968-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ac598e5-c4d3-4aa5-ac25-3bcbd0b3c968-istio-podinfo\") pod \"6ac598e5-c4d3-4aa5-ac25-3bcbd0b3c968\" (UID: \"6ac598e5-c4d3-4aa5-ac25-3bcbd0b3c968\") "
Oct 31 12:59:12.644900 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:12.644876    5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ac598e5-c4d3-4aa5-ac25-3bcbd0b3c968-istio-envoy\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:12.644900 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:12.644897    5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ac598e5-c4d3-4aa5-ac25-3bcbd0b3c968-workload-certs\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:12.645029 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:12.644907    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8765f\" (UniqueName: \"kubernetes.io/projected/6ac598e5-c4d3-4aa5-ac25-3bcbd0b3c968-kube-api-access-8765f\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:12.645029 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:12.644917    5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ac598e5-c4d3-4aa5-ac25-3bcbd0b3c968-workload-socket\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:12.645029 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:12.644927    5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ac598e5-c4d3-4aa5-ac25-3bcbd0b3c968-credential-socket\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:12.645029 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:12.644936    5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ac598e5-c4d3-4aa5-ac25-3bcbd0b3c968-istio-token\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:12.645029 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:12.644949    5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ac598e5-c4d3-4aa5-ac25-3bcbd0b3c968-istio-data\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:12.645029 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:12.644962    5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ac598e5-c4d3-4aa5-ac25-3bcbd0b3c968-istiod-ca-cert\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:12.646575 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:12.646548    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6ac598e5-c4d3-4aa5-ac25-3bcbd0b3c968-istio-podinfo" pod "6ac598e5-c4d3-4aa5-ac25-3bcbd0b3c968" (UID: "6ac598e5-c4d3-4aa5-ac25-3bcbd0b3c968"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 12:59:12.746414 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:12.746378    5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ac598e5-c4d3-4aa5-ac25-3bcbd0b3c968-istio-podinfo\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:13.285310 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:13.285277    5060 generic.go:356] "Generic (PLEG): container finished" podID="6ac598e5-c4d3-4aa5-ac25-3bcbd0b3c968" containerID="fd1d1f5f6aaec977ef4430afe8bb838bcc102f2b3ecd2213540edfbe34e1a1f8" exitCode=0
Oct 31 12:59:13.285757 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:13.285353    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9b62a0-openshift-default-6f648468b7-g7tch"
Oct 31 12:59:13.285757 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:13.285363    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9b62a0-openshift-default-6f648468b7-g7tch" event={"ID":"6ac598e5-c4d3-4aa5-ac25-3bcbd0b3c968","Type":"ContainerDied","Data":"fd1d1f5f6aaec977ef4430afe8bb838bcc102f2b3ecd2213540edfbe34e1a1f8"}
Oct 31 12:59:13.285757 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:13.285479    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9b62a0-openshift-default-6f648468b7-g7tch" event={"ID":"6ac598e5-c4d3-4aa5-ac25-3bcbd0b3c968","Type":"ContainerDied","Data":"deb7ea752fa8096606ad6acf96d5af9d1d6c2537f0a2e84b1a4919889acb4306"}
Oct 31 12:59:13.285757 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:13.285527    5060 scope.go:122] "RemoveContainer" containerID="fd1d1f5f6aaec977ef4430afe8bb838bcc102f2b3ecd2213540edfbe34e1a1f8"
Oct 31 12:59:13.291580 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:13.291548    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-71bc05-fsl95" event={"ID":"49c1b7fd-fcf1-494e-b03a-321bb70d4151","Type":"ContainerStarted","Data":"bcc9dff60dd5843791c36e7563fc86cdd0ded9d00b9a5d96269771a71bcca9e3"}
Oct 31 12:59:13.291745 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:13.291727    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-71bc05-fsl95" event={"ID":"49c1b7fd-fcf1-494e-b03a-321bb70d4151","Type":"ContainerStarted","Data":"2fe49ba7e1fd5c26166da4b40695a0b5c1f3036a4eaf621c24b37022782a2294"}
Oct 31 12:59:13.299812 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:13.299620    5060 scope.go:122] "RemoveContainer" containerID="fd1d1f5f6aaec977ef4430afe8bb838bcc102f2b3ecd2213540edfbe34e1a1f8"
Oct 31 12:59:13.300095 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:59:13.300069    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd1d1f5f6aaec977ef4430afe8bb838bcc102f2b3ecd2213540edfbe34e1a1f8\": container with ID starting with fd1d1f5f6aaec977ef4430afe8bb838bcc102f2b3ecd2213540edfbe34e1a1f8 not found: ID does not exist" containerID="fd1d1f5f6aaec977ef4430afe8bb838bcc102f2b3ecd2213540edfbe34e1a1f8"
Oct 31 12:59:13.300182 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:13.300114    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd1d1f5f6aaec977ef4430afe8bb838bcc102f2b3ecd2213540edfbe34e1a1f8"} err="failed to get container status \"fd1d1f5f6aaec977ef4430afe8bb838bcc102f2b3ecd2213540edfbe34e1a1f8\": rpc error: code = NotFound desc = could not find container \"fd1d1f5f6aaec977ef4430afe8bb838bcc102f2b3ecd2213540edfbe34e1a1f8\": container with ID starting with fd1d1f5f6aaec977ef4430afe8bb838bcc102f2b3ecd2213540edfbe34e1a1f8 not found: ID does not exist"
Oct 31 12:59:13.315948 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:13.315923    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-9b62a0-openshift-default-6f648468b7-g7tch"]
Oct 31 12:59:13.319679 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:13.319659    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-9b62a0-openshift-default-6f648468b7-g7tch"]
Oct 31 12:59:13.599718 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:13.599669    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d176a5-57f6cd6df5-tzc5c" podUID="40133e45-b954-420a-aa2c-8e5b9c873ef4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 12:59:14.306576 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:14.306542    5060 generic.go:356] "Generic (PLEG): container finished" podID="49c1b7fd-fcf1-494e-b03a-321bb70d4151" containerID="bcc9dff60dd5843791c36e7563fc86cdd0ded9d00b9a5d96269771a71bcca9e3" exitCode=0
Oct 31 12:59:14.307044 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:14.306623    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-71bc05-fsl95" event={"ID":"49c1b7fd-fcf1-494e-b03a-321bb70d4151","Type":"ContainerDied","Data":"bcc9dff60dd5843791c36e7563fc86cdd0ded9d00b9a5d96269771a71bcca9e3"}
Oct 31 12:59:14.478113 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:14.478080    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-71bc05-fsl95"
Oct 31 12:59:14.569501 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:14.569404    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/49c1b7fd-fcf1-494e-b03a-321bb70d4151-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49c1b7fd-fcf1-494e-b03a-321bb70d4151-maas-api-xks-ca\") pod \"49c1b7fd-fcf1-494e-b03a-321bb70d4151\" (UID: \"49c1b7fd-fcf1-494e-b03a-321bb70d4151\") "
Oct 31 12:59:14.569671 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:14.569480    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49c1b7fd-fcf1-494e-b03a-321bb70d4151-kube-api-access-lx4xt\" (UniqueName: \"kubernetes.io/projected/49c1b7fd-fcf1-494e-b03a-321bb70d4151-kube-api-access-lx4xt\") pod \"49c1b7fd-fcf1-494e-b03a-321bb70d4151\" (UID: \"49c1b7fd-fcf1-494e-b03a-321bb70d4151\") "
Oct 31 12:59:14.569671 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:14.569574    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49c1b7fd-fcf1-494e-b03a-321bb70d4151-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49c1b7fd-fcf1-494e-b03a-321bb70d4151-maas-api-openshift-ca\") pod \"49c1b7fd-fcf1-494e-b03a-321bb70d4151\" (UID: \"49c1b7fd-fcf1-494e-b03a-321bb70d4151\") "
Oct 31 12:59:14.569993 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:14.569960    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49c1b7fd-fcf1-494e-b03a-321bb70d4151-maas-api-openshift-ca" pod "49c1b7fd-fcf1-494e-b03a-321bb70d4151" (UID: "49c1b7fd-fcf1-494e-b03a-321bb70d4151"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:59:14.570140 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:14.570116    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49c1b7fd-fcf1-494e-b03a-321bb70d4151-maas-api-openshift-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:14.571601 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:14.571573    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49c1b7fd-fcf1-494e-b03a-321bb70d4151-maas-api-xks-ca" pod "49c1b7fd-fcf1-494e-b03a-321bb70d4151" (UID: "49c1b7fd-fcf1-494e-b03a-321bb70d4151"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:59:14.572034 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:14.572014    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49c1b7fd-fcf1-494e-b03a-321bb70d4151-kube-api-access-lx4xt" pod "49c1b7fd-fcf1-494e-b03a-321bb70d4151" (UID: "49c1b7fd-fcf1-494e-b03a-321bb70d4151"). InnerVolumeSpecName "kube-api-access-lx4xt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:59:14.671660 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:14.671624    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49c1b7fd-fcf1-494e-b03a-321bb70d4151-maas-api-xks-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:14.671660 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:14.671652    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lx4xt\" (UniqueName: \"kubernetes.io/projected/49c1b7fd-fcf1-494e-b03a-321bb70d4151-kube-api-access-lx4xt\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:15.049906 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:15.049864    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ac598e5-c4d3-4aa5-ac25-3bcbd0b3c968" path="/var/lib/kubelet/pods/6ac598e5-c4d3-4aa5-ac25-3bcbd0b3c968/volumes"
Oct 31 12:59:15.317580 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:15.317512    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-71bc05-fsl95"
Oct 31 12:59:15.317959 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:15.317515    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-71bc05-fsl95" event={"ID":"49c1b7fd-fcf1-494e-b03a-321bb70d4151","Type":"ContainerDied","Data":"2fe49ba7e1fd5c26166da4b40695a0b5c1f3036a4eaf621c24b37022782a2294"}
Oct 31 12:59:15.317959 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:15.317642    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2fe49ba7e1fd5c26166da4b40695a0b5c1f3036a4eaf621c24b37022782a2294"
Oct 31 12:59:15.509558 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:15.509529    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-pz8cf"
Oct 31 12:59:15.583891 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:15.583808    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/326d88b8-e041-4b4f-ab99-f1d2622d76b4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/326d88b8-e041-4b4f-ab99-f1d2622d76b4-workload-socket\") pod \"326d88b8-e041-4b4f-ab99-f1d2622d76b4\" (UID: \"326d88b8-e041-4b4f-ab99-f1d2622d76b4\") "
Oct 31 12:59:15.583891 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:15.583854    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/326d88b8-e041-4b4f-ab99-f1d2622d76b4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/326d88b8-e041-4b4f-ab99-f1d2622d76b4-istio-envoy\") pod \"326d88b8-e041-4b4f-ab99-f1d2622d76b4\" (UID: \"326d88b8-e041-4b4f-ab99-f1d2622d76b4\") "
Oct 31 12:59:15.584405 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:15.583895    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/326d88b8-e041-4b4f-ab99-f1d2622d76b4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/326d88b8-e041-4b4f-ab99-f1d2622d76b4-istiod-ca-cert\") pod \"326d88b8-e041-4b4f-ab99-f1d2622d76b4\" (UID: \"326d88b8-e041-4b4f-ab99-f1d2622d76b4\") "
Oct 31 12:59:15.584405 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:15.584002    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/326d88b8-e041-4b4f-ab99-f1d2622d76b4-istio-token\" (UniqueName: \"kubernetes.io/projected/326d88b8-e041-4b4f-ab99-f1d2622d76b4-istio-token\") pod \"326d88b8-e041-4b4f-ab99-f1d2622d76b4\" (UID: \"326d88b8-e041-4b4f-ab99-f1d2622d76b4\") "
Oct 31 12:59:15.584405 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:15.584125    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/326d88b8-e041-4b4f-ab99-f1d2622d76b4-workload-socket" pod "326d88b8-e041-4b4f-ab99-f1d2622d76b4" (UID: "326d88b8-e041-4b4f-ab99-f1d2622d76b4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:59:15.584405 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:15.584275    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/326d88b8-e041-4b4f-ab99-f1d2622d76b4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/326d88b8-e041-4b4f-ab99-f1d2622d76b4-istio-data\") pod \"326d88b8-e041-4b4f-ab99-f1d2622d76b4\" (UID: \"326d88b8-e041-4b4f-ab99-f1d2622d76b4\") "
Oct 31 12:59:15.584405 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:15.584307    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/326d88b8-e041-4b4f-ab99-f1d2622d76b4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/326d88b8-e041-4b4f-ab99-f1d2622d76b4-istio-podinfo\") pod \"326d88b8-e041-4b4f-ab99-f1d2622d76b4\" (UID: \"326d88b8-e041-4b4f-ab99-f1d2622d76b4\") "
Oct 31 12:59:15.584405 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:15.584341    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/326d88b8-e041-4b4f-ab99-f1d2622d76b4-kube-api-access-rx486\" (UniqueName: \"kubernetes.io/projected/326d88b8-e041-4b4f-ab99-f1d2622d76b4-kube-api-access-rx486\") pod \"326d88b8-e041-4b4f-ab99-f1d2622d76b4\" (UID: \"326d88b8-e041-4b4f-ab99-f1d2622d76b4\") "
Oct 31 12:59:15.584405 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:15.584373    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/326d88b8-e041-4b4f-ab99-f1d2622d76b4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/326d88b8-e041-4b4f-ab99-f1d2622d76b4-credential-socket\") pod \"326d88b8-e041-4b4f-ab99-f1d2622d76b4\" (UID: \"326d88b8-e041-4b4f-ab99-f1d2622d76b4\") "
Oct 31 12:59:15.585076 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:15.584418    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/326d88b8-e041-4b4f-ab99-f1d2622d76b4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/326d88b8-e041-4b4f-ab99-f1d2622d76b4-workload-certs\") pod \"326d88b8-e041-4b4f-ab99-f1d2622d76b4\" (UID: \"326d88b8-e041-4b4f-ab99-f1d2622d76b4\") "
Oct 31 12:59:15.585076 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:15.584695    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/326d88b8-e041-4b4f-ab99-f1d2622d76b4-credential-socket" pod "326d88b8-e041-4b4f-ab99-f1d2622d76b4" (UID: "326d88b8-e041-4b4f-ab99-f1d2622d76b4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:59:15.585076 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:15.584889    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/326d88b8-e041-4b4f-ab99-f1d2622d76b4-workload-certs" pod "326d88b8-e041-4b4f-ab99-f1d2622d76b4" (UID: "326d88b8-e041-4b4f-ab99-f1d2622d76b4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:59:15.585076 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:15.584985    5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/326d88b8-e041-4b4f-ab99-f1d2622d76b4-workload-socket\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:15.585076 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:15.585005    5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/326d88b8-e041-4b4f-ab99-f1d2622d76b4-credential-socket\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:15.585076 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:15.585017    5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/326d88b8-e041-4b4f-ab99-f1d2622d76b4-workload-certs\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:15.585076 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:15.585060    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/326d88b8-e041-4b4f-ab99-f1d2622d76b4-istio-data" pod "326d88b8-e041-4b4f-ab99-f1d2622d76b4" (UID: "326d88b8-e041-4b4f-ab99-f1d2622d76b4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:59:15.587248 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:15.587216    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/326d88b8-e041-4b4f-ab99-f1d2622d76b4-istio-podinfo" pod "326d88b8-e041-4b4f-ab99-f1d2622d76b4" (UID: "326d88b8-e041-4b4f-ab99-f1d2622d76b4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 12:59:15.587989 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:15.587964    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/326d88b8-e041-4b4f-ab99-f1d2622d76b4-kube-api-access-rx486" pod "326d88b8-e041-4b4f-ab99-f1d2622d76b4" (UID: "326d88b8-e041-4b4f-ab99-f1d2622d76b4"). InnerVolumeSpecName "kube-api-access-rx486". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:59:15.590616 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:15.590591    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/326d88b8-e041-4b4f-ab99-f1d2622d76b4-istio-envoy" pod "326d88b8-e041-4b4f-ab99-f1d2622d76b4" (UID: "326d88b8-e041-4b4f-ab99-f1d2622d76b4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:59:15.595916 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:15.595875    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/326d88b8-e041-4b4f-ab99-f1d2622d76b4-istiod-ca-cert" pod "326d88b8-e041-4b4f-ab99-f1d2622d76b4" (UID: "326d88b8-e041-4b4f-ab99-f1d2622d76b4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:59:15.597353 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:15.597328    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/326d88b8-e041-4b4f-ab99-f1d2622d76b4-istio-token" pod "326d88b8-e041-4b4f-ab99-f1d2622d76b4" (UID: "326d88b8-e041-4b4f-ab99-f1d2622d76b4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:59:15.686066 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:15.686035    5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/326d88b8-e041-4b4f-ab99-f1d2622d76b4-istiod-ca-cert\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:15.686066 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:15.686065    5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/326d88b8-e041-4b4f-ab99-f1d2622d76b4-istio-token\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:15.686066 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:15.686074    5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/326d88b8-e041-4b4f-ab99-f1d2622d76b4-istio-data\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:15.686330 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:15.686084    5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/326d88b8-e041-4b4f-ab99-f1d2622d76b4-istio-podinfo\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:15.686330 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:15.686094    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rx486\" (UniqueName: \"kubernetes.io/projected/326d88b8-e041-4b4f-ab99-f1d2622d76b4-kube-api-access-rx486\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:15.686330 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:15.686105    5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/326d88b8-e041-4b4f-ab99-f1d2622d76b4-istio-envoy\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:15.694064 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:15.694033    5060 patch_prober.go:29] interesting pod/e2e-inf-a-d086f3-openshift-default-5c7884ccc-nzrl9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 12:59:15.694203 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:15.694085    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-nzrl9" podUID="b1ac43d5-eb68-45aa-9e8e-268d7c6355dd" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 12:59:16.329733 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:16.329704    5060 generic.go:356] "Generic (PLEG): container finished" podID="326d88b8-e041-4b4f-ab99-f1d2622d76b4" containerID="adf65d533598e86aae889d3f711cf879259eb58f2663d14215861777e4ff8ad3" exitCode=0
Oct 31 12:59:16.330103 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:16.329752    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-pz8cf"
Oct 31 12:59:16.330103 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:16.329784    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-pz8cf" event={"ID":"326d88b8-e041-4b4f-ab99-f1d2622d76b4","Type":"ContainerDied","Data":"adf65d533598e86aae889d3f711cf879259eb58f2663d14215861777e4ff8ad3"}
Oct 31 12:59:16.330103 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:16.329842    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-pz8cf" event={"ID":"326d88b8-e041-4b4f-ab99-f1d2622d76b4","Type":"ContainerDied","Data":"f13c80dabdb5f38aa5f229bae3faf663bfb114ed7cac8ec872711cbc3ec11537"}
Oct 31 12:59:16.330103 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:16.329865    5060 scope.go:122] "RemoveContainer" containerID="adf65d533598e86aae889d3f711cf879259eb58f2663d14215861777e4ff8ad3"
Oct 31 12:59:16.425947 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:16.425631    5060 patch_prober.go:29] interesting pod/e2e-inf-a-d086f3-openshift-default-5c7884ccc-7stx7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body=
Oct 31 12:59:16.425947 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:16.425691    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-7stx7" podUID="25899808-1323-425c-9359-3b56f99ad5cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused"
Oct 31 12:59:16.438267 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:16.438244    5060 scope.go:122] "RemoveContainer" containerID="adf65d533598e86aae889d3f711cf879259eb58f2663d14215861777e4ff8ad3"
Oct 31 12:59:16.438733 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:59:16.438712    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"adf65d533598e86aae889d3f711cf879259eb58f2663d14215861777e4ff8ad3\": container with ID starting with adf65d533598e86aae889d3f711cf879259eb58f2663d14215861777e4ff8ad3 not found: ID does not exist" containerID="adf65d533598e86aae889d3f711cf879259eb58f2663d14215861777e4ff8ad3"
Oct 31 12:59:16.438894 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:16.438875    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"adf65d533598e86aae889d3f711cf879259eb58f2663d14215861777e4ff8ad3"} err="failed to get container status \"adf65d533598e86aae889d3f711cf879259eb58f2663d14215861777e4ff8ad3\": rpc error: code = NotFound desc = could not find container \"adf65d533598e86aae889d3f711cf879259eb58f2663d14215861777e4ff8ad3\": container with ID starting with adf65d533598e86aae889d3f711cf879259eb58f2663d14215861777e4ff8ad3 not found: ID does not exist"
Oct 31 12:59:16.452470 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:16.452408    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-pz8cf"]
Oct 31 12:59:16.496647 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:16.496417    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-pz8cf"]
Oct 31 12:59:16.594527 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:16.594500    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-nzrl9"
Oct 31 12:59:16.703963 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:16.703922    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1ac43d5-eb68-45aa-9e8e-268d7c6355dd-istio-token\" (UniqueName: \"kubernetes.io/projected/b1ac43d5-eb68-45aa-9e8e-268d7c6355dd-istio-token\") pod \"b1ac43d5-eb68-45aa-9e8e-268d7c6355dd\" (UID: \"b1ac43d5-eb68-45aa-9e8e-268d7c6355dd\") "
Oct 31 12:59:16.704167 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:16.703994    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1ac43d5-eb68-45aa-9e8e-268d7c6355dd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1ac43d5-eb68-45aa-9e8e-268d7c6355dd-istio-envoy\") pod \"b1ac43d5-eb68-45aa-9e8e-268d7c6355dd\" (UID: \"b1ac43d5-eb68-45aa-9e8e-268d7c6355dd\") "
Oct 31 12:59:16.704167 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:16.704097    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1ac43d5-eb68-45aa-9e8e-268d7c6355dd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1ac43d5-eb68-45aa-9e8e-268d7c6355dd-credential-socket\") pod \"b1ac43d5-eb68-45aa-9e8e-268d7c6355dd\" (UID: \"b1ac43d5-eb68-45aa-9e8e-268d7c6355dd\") "
Oct 31 12:59:16.704167 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:16.704135    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1ac43d5-eb68-45aa-9e8e-268d7c6355dd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1ac43d5-eb68-45aa-9e8e-268d7c6355dd-workload-certs\") pod \"b1ac43d5-eb68-45aa-9e8e-268d7c6355dd\" (UID: \"b1ac43d5-eb68-45aa-9e8e-268d7c6355dd\") "
Oct 31 12:59:16.704341 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:16.704187    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1ac43d5-eb68-45aa-9e8e-268d7c6355dd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1ac43d5-eb68-45aa-9e8e-268d7c6355dd-istio-data\") pod \"b1ac43d5-eb68-45aa-9e8e-268d7c6355dd\" (UID: \"b1ac43d5-eb68-45aa-9e8e-268d7c6355dd\") "
Oct 31 12:59:16.704341 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:16.704268    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1ac43d5-eb68-45aa-9e8e-268d7c6355dd-kube-api-access-nlf96\" (UniqueName: \"kubernetes.io/projected/b1ac43d5-eb68-45aa-9e8e-268d7c6355dd-kube-api-access-nlf96\") pod \"b1ac43d5-eb68-45aa-9e8e-268d7c6355dd\" (UID: \"b1ac43d5-eb68-45aa-9e8e-268d7c6355dd\") "
Oct 31 12:59:16.704460 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:16.704344    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1ac43d5-eb68-45aa-9e8e-268d7c6355dd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1ac43d5-eb68-45aa-9e8e-268d7c6355dd-istiod-ca-cert\") pod \"b1ac43d5-eb68-45aa-9e8e-268d7c6355dd\" (UID: \"b1ac43d5-eb68-45aa-9e8e-268d7c6355dd\") "
Oct 31 12:59:16.704460 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:16.704403    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b1ac43d5-eb68-45aa-9e8e-268d7c6355dd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1ac43d5-eb68-45aa-9e8e-268d7c6355dd-istio-podinfo\") pod \"b1ac43d5-eb68-45aa-9e8e-268d7c6355dd\" (UID: \"b1ac43d5-eb68-45aa-9e8e-268d7c6355dd\") "
Oct 31 12:59:16.704460 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:16.704432    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1ac43d5-eb68-45aa-9e8e-268d7c6355dd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1ac43d5-eb68-45aa-9e8e-268d7c6355dd-workload-socket\") pod \"b1ac43d5-eb68-45aa-9e8e-268d7c6355dd\" (UID: \"b1ac43d5-eb68-45aa-9e8e-268d7c6355dd\") "
Oct 31 12:59:16.706548 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:16.706319    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1ac43d5-eb68-45aa-9e8e-268d7c6355dd-istio-data" pod "b1ac43d5-eb68-45aa-9e8e-268d7c6355dd" (UID: "b1ac43d5-eb68-45aa-9e8e-268d7c6355dd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:59:16.707008 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:16.706730    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1ac43d5-eb68-45aa-9e8e-268d7c6355dd-credential-socket" pod "b1ac43d5-eb68-45aa-9e8e-268d7c6355dd" (UID: "b1ac43d5-eb68-45aa-9e8e-268d7c6355dd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:59:16.707462 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:16.707438    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1ac43d5-eb68-45aa-9e8e-268d7c6355dd-istiod-ca-cert" pod "b1ac43d5-eb68-45aa-9e8e-268d7c6355dd" (UID: "b1ac43d5-eb68-45aa-9e8e-268d7c6355dd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:59:16.707582 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:16.707503    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1ac43d5-eb68-45aa-9e8e-268d7c6355dd-workload-certs" pod "b1ac43d5-eb68-45aa-9e8e-268d7c6355dd" (UID: "b1ac43d5-eb68-45aa-9e8e-268d7c6355dd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:59:16.707743 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:16.707722    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1ac43d5-eb68-45aa-9e8e-268d7c6355dd-workload-socket" pod "b1ac43d5-eb68-45aa-9e8e-268d7c6355dd" (UID: "b1ac43d5-eb68-45aa-9e8e-268d7c6355dd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:59:16.717952 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:16.717922    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1ac43d5-eb68-45aa-9e8e-268d7c6355dd-kube-api-access-nlf96" pod "b1ac43d5-eb68-45aa-9e8e-268d7c6355dd" (UID: "b1ac43d5-eb68-45aa-9e8e-268d7c6355dd"). InnerVolumeSpecName "kube-api-access-nlf96". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:59:16.720139 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:16.719984    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1ac43d5-eb68-45aa-9e8e-268d7c6355dd-istio-envoy" pod "b1ac43d5-eb68-45aa-9e8e-268d7c6355dd" (UID: "b1ac43d5-eb68-45aa-9e8e-268d7c6355dd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:59:16.723114 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:16.723074    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b1ac43d5-eb68-45aa-9e8e-268d7c6355dd-istio-podinfo" pod "b1ac43d5-eb68-45aa-9e8e-268d7c6355dd" (UID: "b1ac43d5-eb68-45aa-9e8e-268d7c6355dd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 12:59:16.724329 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:16.724291    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1ac43d5-eb68-45aa-9e8e-268d7c6355dd-istio-token" pod "b1ac43d5-eb68-45aa-9e8e-268d7c6355dd" (UID: "b1ac43d5-eb68-45aa-9e8e-268d7c6355dd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:59:16.753207 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:16.753146    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-7stx7"
Oct 31 12:59:16.755878 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:16.755855    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-s9nqq"
Oct 31 12:59:16.806532 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:16.806460    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25899808-1323-425c-9359-3b56f99ad5cb-istio-token\" (UniqueName: \"kubernetes.io/projected/25899808-1323-425c-9359-3b56f99ad5cb-istio-token\") pod \"25899808-1323-425c-9359-3b56f99ad5cb\" (UID: \"25899808-1323-425c-9359-3b56f99ad5cb\") "
Oct 31 12:59:16.806696 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:16.806552    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebcf940c-4d23-4944-9948-d3dea5ba99f6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebcf940c-4d23-4944-9948-d3dea5ba99f6-credential-socket\") pod \"ebcf940c-4d23-4944-9948-d3dea5ba99f6\" (UID: \"ebcf940c-4d23-4944-9948-d3dea5ba99f6\") "
Oct 31 12:59:16.806864 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:16.806834    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebcf940c-4d23-4944-9948-d3dea5ba99f6-credential-socket" pod "ebcf940c-4d23-4944-9948-d3dea5ba99f6" (UID: "ebcf940c-4d23-4944-9948-d3dea5ba99f6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:59:16.807006 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:16.806985    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ebcf940c-4d23-4944-9948-d3dea5ba99f6-kube-api-access-zlg6q\" (UniqueName: \"kubernetes.io/projected/ebcf940c-4d23-4944-9948-d3dea5ba99f6-kube-api-access-zlg6q\") pod \"ebcf940c-4d23-4944-9948-d3dea5ba99f6\" (UID: \"ebcf940c-4d23-4944-9948-d3dea5ba99f6\") "
Oct 31 12:59:16.807085 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:16.807046    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ebcf940c-4d23-4944-9948-d3dea5ba99f6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebcf940c-4d23-4944-9948-d3dea5ba99f6-istiod-ca-cert\") pod \"ebcf940c-4d23-4944-9948-d3dea5ba99f6\" (UID: \"ebcf940c-4d23-4944-9948-d3dea5ba99f6\") "
Oct 31 12:59:16.807085 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:16.807077    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25899808-1323-425c-9359-3b56f99ad5cb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25899808-1323-425c-9359-3b56f99ad5cb-istio-data\") pod \"25899808-1323-425c-9359-3b56f99ad5cb\" (UID: \"25899808-1323-425c-9359-3b56f99ad5cb\") "
Oct 31 12:59:16.807208 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:16.807134    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebcf940c-4d23-4944-9948-d3dea5ba99f6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebcf940c-4d23-4944-9948-d3dea5ba99f6-workload-socket\") pod \"ebcf940c-4d23-4944-9948-d3dea5ba99f6\" (UID: \"ebcf940c-4d23-4944-9948-d3dea5ba99f6\") "
Oct 31 12:59:16.807271 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:16.807205    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/25899808-1323-425c-9359-3b56f99ad5cb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25899808-1323-425c-9359-3b56f99ad5cb-istio-podinfo\") pod \"25899808-1323-425c-9359-3b56f99ad5cb\" (UID: \"25899808-1323-425c-9359-3b56f99ad5cb\") "
Oct 31 12:59:16.807271 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:16.807233    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebcf940c-4d23-4944-9948-d3dea5ba99f6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebcf940c-4d23-4944-9948-d3dea5ba99f6-istio-envoy\") pod \"ebcf940c-4d23-4944-9948-d3dea5ba99f6\" (UID: \"ebcf940c-4d23-4944-9948-d3dea5ba99f6\") "
Oct 31 12:59:16.807387 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:16.807296    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25899808-1323-425c-9359-3b56f99ad5cb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25899808-1323-425c-9359-3b56f99ad5cb-credential-socket\") pod \"25899808-1323-425c-9359-3b56f99ad5cb\" (UID: \"25899808-1323-425c-9359-3b56f99ad5cb\") "
Oct 31 12:59:16.807387 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:16.807327    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebcf940c-4d23-4944-9948-d3dea5ba99f6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebcf940c-4d23-4944-9948-d3dea5ba99f6-istio-data\") pod \"ebcf940c-4d23-4944-9948-d3dea5ba99f6\" (UID: \"ebcf940c-4d23-4944-9948-d3dea5ba99f6\") "
Oct 31 12:59:16.807528 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:16.807385    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ebcf940c-4d23-4944-9948-d3dea5ba99f6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebcf940c-4d23-4944-9948-d3dea5ba99f6-istio-podinfo\") pod \"ebcf940c-4d23-4944-9948-d3dea5ba99f6\" (UID: \"ebcf940c-4d23-4944-9948-d3dea5ba99f6\") "
Oct 31 12:59:16.807528 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:16.807422    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25899808-1323-425c-9359-3b56f99ad5cb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25899808-1323-425c-9359-3b56f99ad5cb-istio-envoy\") pod \"25899808-1323-425c-9359-3b56f99ad5cb\" (UID: \"25899808-1323-425c-9359-3b56f99ad5cb\") "
Oct 31 12:59:16.807657 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:16.807520    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebcf940c-4d23-4944-9948-d3dea5ba99f6-workload-socket" pod "ebcf940c-4d23-4944-9948-d3dea5ba99f6" (UID: "ebcf940c-4d23-4944-9948-d3dea5ba99f6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:59:16.807657 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:16.807575    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ebcf940c-4d23-4944-9948-d3dea5ba99f6-istio-token\" (UniqueName: \"kubernetes.io/projected/ebcf940c-4d23-4944-9948-d3dea5ba99f6-istio-token\") pod \"ebcf940c-4d23-4944-9948-d3dea5ba99f6\" (UID: \"ebcf940c-4d23-4944-9948-d3dea5ba99f6\") "
Oct 31 12:59:16.807657 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:16.807629    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25899808-1323-425c-9359-3b56f99ad5cb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25899808-1323-425c-9359-3b56f99ad5cb-workload-socket\") pod \"25899808-1323-425c-9359-3b56f99ad5cb\" (UID: \"25899808-1323-425c-9359-3b56f99ad5cb\") "
Oct 31 12:59:16.807864 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:16.807662    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25899808-1323-425c-9359-3b56f99ad5cb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25899808-1323-425c-9359-3b56f99ad5cb-istiod-ca-cert\") pod \"25899808-1323-425c-9359-3b56f99ad5cb\" (UID: \"25899808-1323-425c-9359-3b56f99ad5cb\") "
Oct 31 12:59:16.807864 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:16.807721    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25899808-1323-425c-9359-3b56f99ad5cb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25899808-1323-425c-9359-3b56f99ad5cb-workload-certs\") pod \"25899808-1323-425c-9359-3b56f99ad5cb\" (UID: \"25899808-1323-425c-9359-3b56f99ad5cb\") "
Oct 31 12:59:16.807864 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:16.807764    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25899808-1323-425c-9359-3b56f99ad5cb-kube-api-access-665vv\" (UniqueName: \"kubernetes.io/projected/25899808-1323-425c-9359-3b56f99ad5cb-kube-api-access-665vv\") pod \"25899808-1323-425c-9359-3b56f99ad5cb\" (UID: \"25899808-1323-425c-9359-3b56f99ad5cb\") "
Oct 31 12:59:16.807864 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:16.807832    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebcf940c-4d23-4944-9948-d3dea5ba99f6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebcf940c-4d23-4944-9948-d3dea5ba99f6-workload-certs\") pod \"ebcf940c-4d23-4944-9948-d3dea5ba99f6\" (UID: \"ebcf940c-4d23-4944-9948-d3dea5ba99f6\") "
Oct 31 12:59:16.808510 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:16.808464    5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1ac43d5-eb68-45aa-9e8e-268d7c6355dd-istio-token\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:16.808724 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:16.808705    5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1ac43d5-eb68-45aa-9e8e-268d7c6355dd-istio-envoy\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:16.808854 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:16.808838    5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebcf940c-4d23-4944-9948-d3dea5ba99f6-credential-socket\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:16.809006 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:16.808991    5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1ac43d5-eb68-45aa-9e8e-268d7c6355dd-credential-socket\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:16.809606 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:16.809079    5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1ac43d5-eb68-45aa-9e8e-268d7c6355dd-workload-certs\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:16.809606 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:16.809096    5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebcf940c-4d23-4944-9948-d3dea5ba99f6-workload-socket\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:16.809606 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:16.809110    5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1ac43d5-eb68-45aa-9e8e-268d7c6355dd-istio-data\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:16.809606 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:16.809123    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nlf96\" (UniqueName: \"kubernetes.io/projected/b1ac43d5-eb68-45aa-9e8e-268d7c6355dd-kube-api-access-nlf96\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:16.809606 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:16.809139    5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1ac43d5-eb68-45aa-9e8e-268d7c6355dd-istiod-ca-cert\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:16.809606 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:16.809152    5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1ac43d5-eb68-45aa-9e8e-268d7c6355dd-istio-podinfo\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:16.809606 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:16.809166    5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1ac43d5-eb68-45aa-9e8e-268d7c6355dd-workload-socket\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:16.809606 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:16.807831    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebcf940c-4d23-4944-9948-d3dea5ba99f6-istiod-ca-cert" pod "ebcf940c-4d23-4944-9948-d3dea5ba99f6" (UID: "ebcf940c-4d23-4944-9948-d3dea5ba99f6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:59:16.809606 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:16.808066    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25899808-1323-425c-9359-3b56f99ad5cb-istio-data" pod "25899808-1323-425c-9359-3b56f99ad5cb" (UID: "25899808-1323-425c-9359-3b56f99ad5cb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:59:16.809606 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:16.809435    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebcf940c-4d23-4944-9948-d3dea5ba99f6-workload-certs" pod "ebcf940c-4d23-4944-9948-d3dea5ba99f6" (UID: "ebcf940c-4d23-4944-9948-d3dea5ba99f6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:59:16.809606 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:16.809452    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25899808-1323-425c-9359-3b56f99ad5cb-istiod-ca-cert" pod "25899808-1323-425c-9359-3b56f99ad5cb" (UID: "25899808-1323-425c-9359-3b56f99ad5cb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:59:16.810410 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:16.809933    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25899808-1323-425c-9359-3b56f99ad5cb-workload-socket" pod "25899808-1323-425c-9359-3b56f99ad5cb" (UID: "25899808-1323-425c-9359-3b56f99ad5cb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:59:16.810410 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:16.810101    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebcf940c-4d23-4944-9948-d3dea5ba99f6-istio-data" pod "ebcf940c-4d23-4944-9948-d3dea5ba99f6" (UID: "ebcf940c-4d23-4944-9948-d3dea5ba99f6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:59:16.810785 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:16.810758    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25899808-1323-425c-9359-3b56f99ad5cb-credential-socket" pod "25899808-1323-425c-9359-3b56f99ad5cb" (UID: "25899808-1323-425c-9359-3b56f99ad5cb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:59:16.810991 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:16.810970    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25899808-1323-425c-9359-3b56f99ad5cb-workload-certs" pod "25899808-1323-425c-9359-3b56f99ad5cb" (UID: "25899808-1323-425c-9359-3b56f99ad5cb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:59:16.811230 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:16.811204    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25899808-1323-425c-9359-3b56f99ad5cb-istio-envoy" pod "25899808-1323-425c-9359-3b56f99ad5cb" (UID: "25899808-1323-425c-9359-3b56f99ad5cb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:59:16.811230 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:16.819691    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebcf940c-4d23-4944-9948-d3dea5ba99f6-istio-token" pod "ebcf940c-4d23-4944-9948-d3dea5ba99f6" (UID: "ebcf940c-4d23-4944-9948-d3dea5ba99f6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:59:16.811230 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:16.821955    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25899808-1323-425c-9359-3b56f99ad5cb-kube-api-access-665vv" pod "25899808-1323-425c-9359-3b56f99ad5cb" (UID: "25899808-1323-425c-9359-3b56f99ad5cb"). InnerVolumeSpecName "kube-api-access-665vv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:59:16.823725 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:16.823653    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ebcf940c-4d23-4944-9948-d3dea5ba99f6-istio-podinfo" pod "ebcf940c-4d23-4944-9948-d3dea5ba99f6" (UID: "ebcf940c-4d23-4944-9948-d3dea5ba99f6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 12:59:16.823835 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:16.823727    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/25899808-1323-425c-9359-3b56f99ad5cb-istio-podinfo" pod "25899808-1323-425c-9359-3b56f99ad5cb" (UID: "25899808-1323-425c-9359-3b56f99ad5cb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 12:59:16.823905 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:16.823851    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebcf940c-4d23-4944-9948-d3dea5ba99f6-kube-api-access-zlg6q" pod "ebcf940c-4d23-4944-9948-d3dea5ba99f6" (UID: "ebcf940c-4d23-4944-9948-d3dea5ba99f6"). InnerVolumeSpecName "kube-api-access-zlg6q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:59:16.823966 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:16.823954    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25899808-1323-425c-9359-3b56f99ad5cb-istio-token" pod "25899808-1323-425c-9359-3b56f99ad5cb" (UID: "25899808-1323-425c-9359-3b56f99ad5cb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:59:16.825123 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:16.825099    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebcf940c-4d23-4944-9948-d3dea5ba99f6-istio-envoy" pod "ebcf940c-4d23-4944-9948-d3dea5ba99f6" (UID: "ebcf940c-4d23-4944-9948-d3dea5ba99f6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:59:16.910142 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:16.910104    5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25899808-1323-425c-9359-3b56f99ad5cb-istio-token\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:16.910142 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:16.910137    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zlg6q\" (UniqueName: \"kubernetes.io/projected/ebcf940c-4d23-4944-9948-d3dea5ba99f6-kube-api-access-zlg6q\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:16.910142 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:16.910150    5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebcf940c-4d23-4944-9948-d3dea5ba99f6-istiod-ca-cert\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:16.910398 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:16.910161    5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25899808-1323-425c-9359-3b56f99ad5cb-istio-data\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:16.910398 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:16.910170    5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25899808-1323-425c-9359-3b56f99ad5cb-istio-podinfo\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:16.910398 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:16.910178    5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebcf940c-4d23-4944-9948-d3dea5ba99f6-istio-envoy\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:16.910398 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:16.910187    5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25899808-1323-425c-9359-3b56f99ad5cb-credential-socket\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:16.910398 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:16.910195    5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebcf940c-4d23-4944-9948-d3dea5ba99f6-istio-data\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:16.910398 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:16.910203    5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebcf940c-4d23-4944-9948-d3dea5ba99f6-istio-podinfo\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:16.910398 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:16.910212    5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25899808-1323-425c-9359-3b56f99ad5cb-istio-envoy\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:16.910398 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:16.910220    5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebcf940c-4d23-4944-9948-d3dea5ba99f6-istio-token\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:16.910398 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:16.910228    5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25899808-1323-425c-9359-3b56f99ad5cb-workload-socket\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:16.910398 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:16.910236    5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25899808-1323-425c-9359-3b56f99ad5cb-istiod-ca-cert\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:16.910398 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:16.910245    5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25899808-1323-425c-9359-3b56f99ad5cb-workload-certs\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:16.910398 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:16.910254    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-665vv\" (UniqueName: \"kubernetes.io/projected/25899808-1323-425c-9359-3b56f99ad5cb-kube-api-access-665vv\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:16.910398 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:16.910262    5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebcf940c-4d23-4944-9948-d3dea5ba99f6-workload-certs\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:17.050619 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:17.050586    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="326d88b8-e041-4b4f-ab99-f1d2622d76b4" path="/var/lib/kubelet/pods/326d88b8-e041-4b4f-ab99-f1d2622d76b4/volumes"
Oct 31 12:59:17.340731 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:17.340695    5060 generic.go:356] "Generic (PLEG): container finished" podID="ebcf940c-4d23-4944-9948-d3dea5ba99f6" containerID="36995cc9e7366968dd91f78614b082c5381b377037cf4d2be4495625365cb598" exitCode=0
Oct 31 12:59:17.341196 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:17.340753    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-s9nqq"
Oct 31 12:59:17.341196 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:17.340782    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-s9nqq" event={"ID":"ebcf940c-4d23-4944-9948-d3dea5ba99f6","Type":"ContainerDied","Data":"36995cc9e7366968dd91f78614b082c5381b377037cf4d2be4495625365cb598"}
Oct 31 12:59:17.341196 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:17.340848    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-s9nqq" event={"ID":"ebcf940c-4d23-4944-9948-d3dea5ba99f6","Type":"ContainerDied","Data":"1e2f6d4fe31786df00accd6aa7be1bea62c2a700341d8fd7fb362d468269bd9a"}
Oct 31 12:59:17.341196 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:17.340879    5060 scope.go:122] "RemoveContainer" containerID="36995cc9e7366968dd91f78614b082c5381b377037cf4d2be4495625365cb598"
Oct 31 12:59:17.342900 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:17.342865    5060 generic.go:356] "Generic (PLEG): container finished" podID="b1ac43d5-eb68-45aa-9e8e-268d7c6355dd" containerID="8f91d4f726d41d37d7125a8084d62df37bf466c88fd28b02196b5aeff9951a2e" exitCode=0
Oct 31 12:59:17.343135 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:17.342929    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-nzrl9"
Oct 31 12:59:17.343317 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:17.343004    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-nzrl9" event={"ID":"b1ac43d5-eb68-45aa-9e8e-268d7c6355dd","Type":"ContainerDied","Data":"8f91d4f726d41d37d7125a8084d62df37bf466c88fd28b02196b5aeff9951a2e"}
Oct 31 12:59:17.343520 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:17.343459    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-nzrl9" event={"ID":"b1ac43d5-eb68-45aa-9e8e-268d7c6355dd","Type":"ContainerDied","Data":"c3b9f8720655ae094f462bc627c2d623fb6b7393e2146cf3d9c8b5a50c206bdb"}
Oct 31 12:59:17.345619 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:17.345594    5060 generic.go:356] "Generic (PLEG): container finished" podID="25899808-1323-425c-9359-3b56f99ad5cb" containerID="0e99ef78dd089b9c1a0dba68109675b3b4434a15d41edf16b2cd47366ecb78b9" exitCode=0
Oct 31 12:59:17.345769 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:17.345666    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-7stx7"
Oct 31 12:59:17.345769 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:17.345704    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-7stx7" event={"ID":"25899808-1323-425c-9359-3b56f99ad5cb","Type":"ContainerDied","Data":"0e99ef78dd089b9c1a0dba68109675b3b4434a15d41edf16b2cd47366ecb78b9"}
Oct 31 12:59:17.345769 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:17.345745    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-7stx7" event={"ID":"25899808-1323-425c-9359-3b56f99ad5cb","Type":"ContainerDied","Data":"d75286916ab35ebe5672ab15aea942360fc21a1b36049d2f87476892c002b60a"}
Oct 31 12:59:17.355547 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:17.355523    5060 scope.go:122] "RemoveContainer" containerID="36995cc9e7366968dd91f78614b082c5381b377037cf4d2be4495625365cb598"
Oct 31 12:59:17.355904 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:59:17.355876    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36995cc9e7366968dd91f78614b082c5381b377037cf4d2be4495625365cb598\": container with ID starting with 36995cc9e7366968dd91f78614b082c5381b377037cf4d2be4495625365cb598 not found: ID does not exist" containerID="36995cc9e7366968dd91f78614b082c5381b377037cf4d2be4495625365cb598"
Oct 31 12:59:17.356675 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:17.355914    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36995cc9e7366968dd91f78614b082c5381b377037cf4d2be4495625365cb598"} err="failed to get container status \"36995cc9e7366968dd91f78614b082c5381b377037cf4d2be4495625365cb598\": rpc error: code = NotFound desc = could not find container \"36995cc9e7366968dd91f78614b082c5381b377037cf4d2be4495625365cb598\": container with ID starting with 36995cc9e7366968dd91f78614b082c5381b377037cf4d2be4495625365cb598 not found: ID does not exist"
Oct 31 12:59:17.356675 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:17.355940    5060 scope.go:122] "RemoveContainer" containerID="8f91d4f726d41d37d7125a8084d62df37bf466c88fd28b02196b5aeff9951a2e"
Oct 31 12:59:17.372763 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:17.372740    5060 scope.go:122] "RemoveContainer" containerID="8f91d4f726d41d37d7125a8084d62df37bf466c88fd28b02196b5aeff9951a2e"
Oct 31 12:59:17.373154 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:59:17.373121    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f91d4f726d41d37d7125a8084d62df37bf466c88fd28b02196b5aeff9951a2e\": container with ID starting with 8f91d4f726d41d37d7125a8084d62df37bf466c88fd28b02196b5aeff9951a2e not found: ID does not exist" containerID="8f91d4f726d41d37d7125a8084d62df37bf466c88fd28b02196b5aeff9951a2e"
Oct 31 12:59:17.373296 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:17.373166    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f91d4f726d41d37d7125a8084d62df37bf466c88fd28b02196b5aeff9951a2e"} err="failed to get container status \"8f91d4f726d41d37d7125a8084d62df37bf466c88fd28b02196b5aeff9951a2e\": rpc error: code = NotFound desc = could not find container \"8f91d4f726d41d37d7125a8084d62df37bf466c88fd28b02196b5aeff9951a2e\": container with ID starting with 8f91d4f726d41d37d7125a8084d62df37bf466c88fd28b02196b5aeff9951a2e not found: ID does not exist"
Oct 31 12:59:17.373296 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:17.373204    5060 scope.go:122] "RemoveContainer" containerID="0e99ef78dd089b9c1a0dba68109675b3b4434a15d41edf16b2cd47366ecb78b9"
Oct 31 12:59:17.379705 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:17.379673    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-s9nqq"]
Oct 31 12:59:17.400856 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:17.400821    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-s9nqq"]
Oct 31 12:59:17.401239 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:17.401219    5060 scope.go:122] "RemoveContainer" containerID="26e44b9d2ea1cc00868da3adafd4ba6e1bbce4e6f6194a2b35f7de329d87b522"
Oct 31 12:59:17.415817 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:17.415734    5060 scope.go:122] "RemoveContainer" containerID="0e99ef78dd089b9c1a0dba68109675b3b4434a15d41edf16b2cd47366ecb78b9"
Oct 31 12:59:17.416335 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:59:17.416309    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e99ef78dd089b9c1a0dba68109675b3b4434a15d41edf16b2cd47366ecb78b9\": container with ID starting with 0e99ef78dd089b9c1a0dba68109675b3b4434a15d41edf16b2cd47366ecb78b9 not found: ID does not exist" containerID="0e99ef78dd089b9c1a0dba68109675b3b4434a15d41edf16b2cd47366ecb78b9"
Oct 31 12:59:17.416579 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:17.416542    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e99ef78dd089b9c1a0dba68109675b3b4434a15d41edf16b2cd47366ecb78b9"} err="failed to get container status \"0e99ef78dd089b9c1a0dba68109675b3b4434a15d41edf16b2cd47366ecb78b9\": rpc error: code = NotFound desc = could not find container \"0e99ef78dd089b9c1a0dba68109675b3b4434a15d41edf16b2cd47366ecb78b9\": container with ID starting with 0e99ef78dd089b9c1a0dba68109675b3b4434a15d41edf16b2cd47366ecb78b9 not found: ID does not exist"
Oct 31 12:59:17.416689 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:17.416601    5060 scope.go:122] "RemoveContainer" containerID="26e44b9d2ea1cc00868da3adafd4ba6e1bbce4e6f6194a2b35f7de329d87b522"
Oct 31 12:59:17.417151 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:59:17.417089    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26e44b9d2ea1cc00868da3adafd4ba6e1bbce4e6f6194a2b35f7de329d87b522\": container with ID starting with 26e44b9d2ea1cc00868da3adafd4ba6e1bbce4e6f6194a2b35f7de329d87b522 not found: ID does not exist" containerID="26e44b9d2ea1cc00868da3adafd4ba6e1bbce4e6f6194a2b35f7de329d87b522"
Oct 31 12:59:17.417151 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:17.417126    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26e44b9d2ea1cc00868da3adafd4ba6e1bbce4e6f6194a2b35f7de329d87b522"} err="failed to get container status \"26e44b9d2ea1cc00868da3adafd4ba6e1bbce4e6f6194a2b35f7de329d87b522\": rpc error: code = NotFound desc = could not find container \"26e44b9d2ea1cc00868da3adafd4ba6e1bbce4e6f6194a2b35f7de329d87b522\": container with ID starting with 26e44b9d2ea1cc00868da3adafd4ba6e1bbce4e6f6194a2b35f7de329d87b522 not found: ID does not exist"
Oct 31 12:59:17.425769 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:17.425746    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-nzrl9"]
Oct 31 12:59:17.432540 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:17.432514    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-nzrl9"]
Oct 31 12:59:17.445688 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:17.445664    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-7stx7"]
Oct 31 12:59:17.454469 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:17.454447    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-7stx7"]
Oct 31 12:59:17.973316 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:17.973284    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fd988c-5cf7g"]
Oct 31 12:59:17.973897 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:17.973882    5060 cpu_manager.go:414] "containerMap: removing container" podUID="b1ac43d5-eb68-45aa-9e8e-268d7c6355dd" containerName="istio-proxy"
Oct 31 12:59:17.973942 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:17.973899    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1ac43d5-eb68-45aa-9e8e-268d7c6355dd" containerName="istio-proxy"
Oct 31 12:59:17.973942 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:17.973908    5060 cpu_manager.go:414] "containerMap: removing container" podUID="49c1b7fd-fcf1-494e-b03a-321bb70d4151" containerName="revoke-keys"
Oct 31 12:59:17.973942 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:17.973914    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49c1b7fd-fcf1-494e-b03a-321bb70d4151" containerName="revoke-keys"
Oct 31 12:59:17.973942 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:17.973930    5060 cpu_manager.go:414] "containerMap: removing container" podUID="25899808-1323-425c-9359-3b56f99ad5cb" containerName="istio-proxy"
Oct 31 12:59:17.973942 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:17.973936    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25899808-1323-425c-9359-3b56f99ad5cb" containerName="istio-proxy"
Oct 31 12:59:17.974113 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:17.973945    5060 cpu_manager.go:414] "containerMap: removing container" podUID="25899808-1323-425c-9359-3b56f99ad5cb" containerName="istio-proxy"
Oct 31 12:59:17.974113 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:17.973950    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25899808-1323-425c-9359-3b56f99ad5cb" containerName="istio-proxy"
Oct 31 12:59:17.974113 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:17.973956    5060 cpu_manager.go:414] "containerMap: removing container" podUID="ebcf940c-4d23-4944-9948-d3dea5ba99f6" containerName="istio-proxy"
Oct 31 12:59:17.974113 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:17.973961    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebcf940c-4d23-4944-9948-d3dea5ba99f6" containerName="istio-proxy"
Oct 31 12:59:17.974113 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:17.973969    5060 cpu_manager.go:414] "containerMap: removing container" podUID="6ac598e5-c4d3-4aa5-ac25-3bcbd0b3c968" containerName="istio-proxy"
Oct 31 12:59:17.974113 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:17.973973    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ac598e5-c4d3-4aa5-ac25-3bcbd0b3c968" containerName="istio-proxy"
Oct 31 12:59:17.974113 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:17.973990    5060 cpu_manager.go:414] "containerMap: removing container" podUID="326d88b8-e041-4b4f-ab99-f1d2622d76b4" containerName="istio-proxy"
Oct 31 12:59:17.974113 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:17.973995    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="326d88b8-e041-4b4f-ab99-f1d2622d76b4" containerName="istio-proxy"
Oct 31 12:59:17.974113 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:17.974086    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1ac43d5-eb68-45aa-9e8e-268d7c6355dd" containerName="istio-proxy"
Oct 31 12:59:17.974113 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:17.974093    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="49c1b7fd-fcf1-494e-b03a-321bb70d4151" containerName="revoke-keys"
Oct 31 12:59:17.974113 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:17.974101    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="25899808-1323-425c-9359-3b56f99ad5cb" containerName="istio-proxy"
Oct 31 12:59:17.974113 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:17.974109    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="25899808-1323-425c-9359-3b56f99ad5cb" containerName="istio-proxy"
Oct 31 12:59:17.974113 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:17.974114    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebcf940c-4d23-4944-9948-d3dea5ba99f6" containerName="istio-proxy"
Oct 31 12:59:17.974572 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:17.974120    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ac598e5-c4d3-4aa5-ac25-3bcbd0b3c968" containerName="istio-proxy"
Oct 31 12:59:17.974572 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:17.974128    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="326d88b8-e041-4b4f-ab99-f1d2622d76b4" containerName="istio-proxy"
Oct 31 12:59:17.979663 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:17.979642    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fd988c-5cf7g"
Oct 31 12:59:17.989344 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:17.989319    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 12:59:17.989552 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:17.989534    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7wsmb\""
Oct 31 12:59:17.994194 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:17.994170    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fd988c-5cf7g"]
Oct 31 12:59:18.027281 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:18.027243    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4b6c141-cbee-48d8-91e7-bdec2b65c7e5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-fd988c-5cf7g\" (UID: \"e4b6c141-cbee-48d8-91e7-bdec2b65c7e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fd988c-5cf7g"
Oct 31 12:59:18.027566 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:18.027342    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4b6c141-cbee-48d8-91e7-bdec2b65c7e5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-fd988c-5cf7g\" (UID: \"e4b6c141-cbee-48d8-91e7-bdec2b65c7e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fd988c-5cf7g"
Oct 31 12:59:18.027566 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:18.027444    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hr2kp\" (UniqueName: \"kubernetes.io/projected/e4b6c141-cbee-48d8-91e7-bdec2b65c7e5-kube-api-access-hr2kp\") pod \"maas-api-revoke-keys-e2e-inf-b-fd988c-5cf7g\" (UID: \"e4b6c141-cbee-48d8-91e7-bdec2b65c7e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fd988c-5cf7g"
Oct 31 12:59:18.128236 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:18.128200    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4b6c141-cbee-48d8-91e7-bdec2b65c7e5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-fd988c-5cf7g\" (UID: \"e4b6c141-cbee-48d8-91e7-bdec2b65c7e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fd988c-5cf7g"
Oct 31 12:59:18.128412 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:18.128303    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4b6c141-cbee-48d8-91e7-bdec2b65c7e5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-fd988c-5cf7g\" (UID: \"e4b6c141-cbee-48d8-91e7-bdec2b65c7e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fd988c-5cf7g"
Oct 31 12:59:18.128412 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:18.128403    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hr2kp\" (UniqueName: \"kubernetes.io/projected/e4b6c141-cbee-48d8-91e7-bdec2b65c7e5-kube-api-access-hr2kp\") pod \"maas-api-revoke-keys-e2e-inf-b-fd988c-5cf7g\" (UID: \"e4b6c141-cbee-48d8-91e7-bdec2b65c7e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fd988c-5cf7g"
Oct 31 12:59:18.129023 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:18.128991    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4b6c141-cbee-48d8-91e7-bdec2b65c7e5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-fd988c-5cf7g\" (UID: \"e4b6c141-cbee-48d8-91e7-bdec2b65c7e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fd988c-5cf7g"
Oct 31 12:59:18.130650 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:18.130629    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4b6c141-cbee-48d8-91e7-bdec2b65c7e5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-fd988c-5cf7g\" (UID: \"e4b6c141-cbee-48d8-91e7-bdec2b65c7e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fd988c-5cf7g"
Oct 31 12:59:18.141974 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:18.141946    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hr2kp\" (UniqueName: \"kubernetes.io/projected/e4b6c141-cbee-48d8-91e7-bdec2b65c7e5-kube-api-access-hr2kp\") pod \"maas-api-revoke-keys-e2e-inf-b-fd988c-5cf7g\" (UID: \"e4b6c141-cbee-48d8-91e7-bdec2b65c7e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fd988c-5cf7g"
Oct 31 12:59:18.295306 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:18.295258    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fd988c-5cf7g"
Oct 31 12:59:18.590183 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:18.590150    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fd988c-5cf7g"]
Oct 31 12:59:18.590606 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:59:18.590541    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4b6c141_cbee_48d8_91e7_bdec2b65c7e5.slice/crio-3389a0376530e8479444352ea3382a74334455d858480230f1767c952fe5e5f6 WatchSource:0}: Error finding container 3389a0376530e8479444352ea3382a74334455d858480230f1767c952fe5e5f6: Status 404 returned error can't find the container with id 3389a0376530e8479444352ea3382a74334455d858480230f1767c952fe5e5f6
Oct 31 12:59:19.048337 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:19.048304    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25899808-1323-425c-9359-3b56f99ad5cb" path="/var/lib/kubelet/pods/25899808-1323-425c-9359-3b56f99ad5cb/volumes"
Oct 31 12:59:19.048790 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:19.048773    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1ac43d5-eb68-45aa-9e8e-268d7c6355dd" path="/var/lib/kubelet/pods/b1ac43d5-eb68-45aa-9e8e-268d7c6355dd/volumes"
Oct 31 12:59:19.049212 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:19.049195    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ebcf940c-4d23-4944-9948-d3dea5ba99f6" path="/var/lib/kubelet/pods/ebcf940c-4d23-4944-9948-d3dea5ba99f6/volumes"
Oct 31 12:59:19.373789 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:19.373755    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fd988c-5cf7g" event={"ID":"e4b6c141-cbee-48d8-91e7-bdec2b65c7e5","Type":"ContainerStarted","Data":"f3f56582bd0372765e78061fb6b025d1bf3ecbc8d30890360526cae3b072edb4"}
Oct 31 12:59:19.373973 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:19.373814    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fd988c-5cf7g" event={"ID":"e4b6c141-cbee-48d8-91e7-bdec2b65c7e5","Type":"ContainerStarted","Data":"3389a0376530e8479444352ea3382a74334455d858480230f1767c952fe5e5f6"}
Oct 31 12:59:19.397560 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:19.397465    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fd988c-5cf7g" podStartSLOduration=2.397447568 podStartE2EDuration="2.397447568s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:59:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:59:19.393813478 +0000 UTC m=+1922.496881190" watchObservedRunningTime="2026-10-31 12:59:19.397447568 +0000 UTC m=+1922.500515281"
Oct 31 12:59:20.011016 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:20.010975    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2c393f-77f8c6697-l7t2b"]
Oct 31 12:59:20.011420 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:20.011196    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2c393f-77f8c6697-l7t2b" podUID="6a1124a2-55e6-424f-a91e-9f89f37db017" containerName="maas-api" containerID="cri-o://1888c2e313cec61c6abe3682b20c02f48554be8b77acf81a7b0a7138a3f2da43" gracePeriod=30
Oct 31 12:59:20.412516 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:20.403746    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2c393f-gq4fj"]
Oct 31 12:59:20.412516 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:20.409006    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2c393f-gq4fj"]
Oct 31 12:59:20.412516 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:20.409945    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2c393f-77f8c6697-l7t2b"
Oct 31 12:59:20.419296 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:20.415085    5060 generic.go:356] "Generic (PLEG): container finished" podID="e4b6c141-cbee-48d8-91e7-bdec2b65c7e5" containerID="f3f56582bd0372765e78061fb6b025d1bf3ecbc8d30890360526cae3b072edb4" exitCode=0
Oct 31 12:59:20.419296 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:20.415176    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fd988c-5cf7g" event={"ID":"e4b6c141-cbee-48d8-91e7-bdec2b65c7e5","Type":"ContainerDied","Data":"f3f56582bd0372765e78061fb6b025d1bf3ecbc8d30890360526cae3b072edb4"}
Oct 31 12:59:20.419296 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:20.417015    5060 generic.go:356] "Generic (PLEG): container finished" podID="6a1124a2-55e6-424f-a91e-9f89f37db017" containerID="1888c2e313cec61c6abe3682b20c02f48554be8b77acf81a7b0a7138a3f2da43" exitCode=0
Oct 31 12:59:20.419296 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:20.417210    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2c393f-77f8c6697-l7t2b" event={"ID":"6a1124a2-55e6-424f-a91e-9f89f37db017","Type":"ContainerDied","Data":"1888c2e313cec61c6abe3682b20c02f48554be8b77acf81a7b0a7138a3f2da43"}
Oct 31 12:59:20.419296 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:20.417242    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2c393f-77f8c6697-l7t2b" event={"ID":"6a1124a2-55e6-424f-a91e-9f89f37db017","Type":"ContainerDied","Data":"ad6494eeea14f1efe5544eb7116cb6812f6feb98923df7aaebfc647a561e707c"}
Oct 31 12:59:20.419296 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:20.417263    5060 scope.go:122] "RemoveContainer" containerID="1888c2e313cec61c6abe3682b20c02f48554be8b77acf81a7b0a7138a3f2da43"
Oct 31 12:59:20.419296 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:20.417365    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2c393f-77f8c6697-l7t2b"
Oct 31 12:59:20.444435 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:20.441596    5060 scope.go:122] "RemoveContainer" containerID="1888c2e313cec61c6abe3682b20c02f48554be8b77acf81a7b0a7138a3f2da43"
Oct 31 12:59:20.444435 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:59:20.442131    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1888c2e313cec61c6abe3682b20c02f48554be8b77acf81a7b0a7138a3f2da43\": container with ID starting with 1888c2e313cec61c6abe3682b20c02f48554be8b77acf81a7b0a7138a3f2da43 not found: ID does not exist" containerID="1888c2e313cec61c6abe3682b20c02f48554be8b77acf81a7b0a7138a3f2da43"
Oct 31 12:59:20.444435 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:20.442173    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1888c2e313cec61c6abe3682b20c02f48554be8b77acf81a7b0a7138a3f2da43"} err="failed to get container status \"1888c2e313cec61c6abe3682b20c02f48554be8b77acf81a7b0a7138a3f2da43\": rpc error: code = NotFound desc = could not find container \"1888c2e313cec61c6abe3682b20c02f48554be8b77acf81a7b0a7138a3f2da43\": container with ID starting with 1888c2e313cec61c6abe3682b20c02f48554be8b77acf81a7b0a7138a3f2da43 not found: ID does not exist"
Oct 31 12:59:20.475951 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:20.475919    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a1124a2-55e6-424f-a91e-9f89f37db017-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a1124a2-55e6-424f-a91e-9f89f37db017-maas-api-tls\") pod \"6a1124a2-55e6-424f-a91e-9f89f37db017\" (UID: \"6a1124a2-55e6-424f-a91e-9f89f37db017\") "
Oct 31 12:59:20.476154 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:20.476047    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a1124a2-55e6-424f-a91e-9f89f37db017-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a1124a2-55e6-424f-a91e-9f89f37db017-openshift-service-ca\") pod \"6a1124a2-55e6-424f-a91e-9f89f37db017\" (UID: \"6a1124a2-55e6-424f-a91e-9f89f37db017\") "
Oct 31 12:59:20.476241 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:20.476151    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a1124a2-55e6-424f-a91e-9f89f37db017-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a1124a2-55e6-424f-a91e-9f89f37db017-metrics-certs\") pod \"6a1124a2-55e6-424f-a91e-9f89f37db017\" (UID: \"6a1124a2-55e6-424f-a91e-9f89f37db017\") "
Oct 31 12:59:20.476304 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:20.476261    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a1124a2-55e6-424f-a91e-9f89f37db017-kube-api-access-xl9jd\" (UniqueName: \"kubernetes.io/projected/6a1124a2-55e6-424f-a91e-9f89f37db017-kube-api-access-xl9jd\") pod \"6a1124a2-55e6-424f-a91e-9f89f37db017\" (UID: \"6a1124a2-55e6-424f-a91e-9f89f37db017\") "
Oct 31 12:59:20.476420 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:20.476393    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a1124a2-55e6-424f-a91e-9f89f37db017-openshift-service-ca" pod "6a1124a2-55e6-424f-a91e-9f89f37db017" (UID: "6a1124a2-55e6-424f-a91e-9f89f37db017"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:59:20.476990 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:20.476731    5060 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a1124a2-55e6-424f-a91e-9f89f37db017-openshift-service-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:20.479167 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:20.479141    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a1124a2-55e6-424f-a91e-9f89f37db017-maas-api-tls" pod "6a1124a2-55e6-424f-a91e-9f89f37db017" (UID: "6a1124a2-55e6-424f-a91e-9f89f37db017"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:59:20.479646 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:20.479608    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a1124a2-55e6-424f-a91e-9f89f37db017-kube-api-access-xl9jd" pod "6a1124a2-55e6-424f-a91e-9f89f37db017" (UID: "6a1124a2-55e6-424f-a91e-9f89f37db017"). InnerVolumeSpecName "kube-api-access-xl9jd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:59:20.479799 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:20.479774    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a1124a2-55e6-424f-a91e-9f89f37db017-metrics-certs" pod "6a1124a2-55e6-424f-a91e-9f89f37db017" (UID: "6a1124a2-55e6-424f-a91e-9f89f37db017"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:59:20.578398 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:20.578353    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xl9jd\" (UniqueName: \"kubernetes.io/projected/6a1124a2-55e6-424f-a91e-9f89f37db017-kube-api-access-xl9jd\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:20.578398 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:20.578392    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a1124a2-55e6-424f-a91e-9f89f37db017-maas-api-tls\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:20.578398 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:20.578408    5060 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a1124a2-55e6-424f-a91e-9f89f37db017-metrics-certs\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:20.743631 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:20.743600    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2c393f-77f8c6697-l7t2b"]
Oct 31 12:59:20.751269 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:20.751240    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2c393f-77f8c6697-l7t2b"]
Oct 31 12:59:21.049588 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:21.049557    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="545ba362-f42e-48b4-aa04-afe3e25401cb" path="/var/lib/kubelet/pods/545ba362-f42e-48b4-aa04-afe3e25401cb/volumes"
Oct 31 12:59:21.050076 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:21.050060    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a1124a2-55e6-424f-a91e-9f89f37db017" path="/var/lib/kubelet/pods/6a1124a2-55e6-424f-a91e-9f89f37db017/volumes"
Oct 31 12:59:21.624212 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:21.624182    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fd988c-5cf7g"
Oct 31 12:59:21.691643 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:21.691549    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e4b6c141-cbee-48d8-91e7-bdec2b65c7e5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4b6c141-cbee-48d8-91e7-bdec2b65c7e5-maas-api-xks-ca\") pod \"e4b6c141-cbee-48d8-91e7-bdec2b65c7e5\" (UID: \"e4b6c141-cbee-48d8-91e7-bdec2b65c7e5\") "
Oct 31 12:59:21.691643 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:21.691617    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4b6c141-cbee-48d8-91e7-bdec2b65c7e5-kube-api-access-hr2kp\" (UniqueName: \"kubernetes.io/projected/e4b6c141-cbee-48d8-91e7-bdec2b65c7e5-kube-api-access-hr2kp\") pod \"e4b6c141-cbee-48d8-91e7-bdec2b65c7e5\" (UID: \"e4b6c141-cbee-48d8-91e7-bdec2b65c7e5\") "
Oct 31 12:59:21.691887 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:21.691657    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4b6c141-cbee-48d8-91e7-bdec2b65c7e5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4b6c141-cbee-48d8-91e7-bdec2b65c7e5-maas-api-openshift-ca\") pod \"e4b6c141-cbee-48d8-91e7-bdec2b65c7e5\" (UID: \"e4b6c141-cbee-48d8-91e7-bdec2b65c7e5\") "
Oct 31 12:59:21.692150 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:21.692123    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4b6c141-cbee-48d8-91e7-bdec2b65c7e5-maas-api-openshift-ca" pod "e4b6c141-cbee-48d8-91e7-bdec2b65c7e5" (UID: "e4b6c141-cbee-48d8-91e7-bdec2b65c7e5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:59:21.694476 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:21.694453    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4b6c141-cbee-48d8-91e7-bdec2b65c7e5-kube-api-access-hr2kp" pod "e4b6c141-cbee-48d8-91e7-bdec2b65c7e5" (UID: "e4b6c141-cbee-48d8-91e7-bdec2b65c7e5"). InnerVolumeSpecName "kube-api-access-hr2kp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:59:21.694776 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:21.694752    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4b6c141-cbee-48d8-91e7-bdec2b65c7e5-maas-api-xks-ca" pod "e4b6c141-cbee-48d8-91e7-bdec2b65c7e5" (UID: "e4b6c141-cbee-48d8-91e7-bdec2b65c7e5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:59:21.793696 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:21.793656    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4b6c141-cbee-48d8-91e7-bdec2b65c7e5-maas-api-xks-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:21.793696 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:21.793685    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hr2kp\" (UniqueName: \"kubernetes.io/projected/e4b6c141-cbee-48d8-91e7-bdec2b65c7e5-kube-api-access-hr2kp\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:21.793696 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:21.793695    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4b6c141-cbee-48d8-91e7-bdec2b65c7e5-maas-api-openshift-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:22.436756 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:22.436726    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fd988c-5cf7g"
Oct 31 12:59:22.437222 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:22.436754    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fd988c-5cf7g" event={"ID":"e4b6c141-cbee-48d8-91e7-bdec2b65c7e5","Type":"ContainerDied","Data":"3389a0376530e8479444352ea3382a74334455d858480230f1767c952fe5e5f6"}
Oct 31 12:59:22.437222 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:22.436804    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3389a0376530e8479444352ea3382a74334455d858480230f1767c952fe5e5f6"
Oct 31 12:59:23.505397 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:23.505353    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-71bc05-fsl95"]
Oct 31 12:59:23.510816 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:23.510785    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-71bc05-fsl95"]
Oct 31 12:59:23.598850 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:23.598806    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d176a5-57f6cd6df5-tzc5c" podUID="40133e45-b954-420a-aa2c-8e5b9c873ef4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 12:59:25.048388 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:25.048352    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49c1b7fd-fcf1-494e-b03a-321bb70d4151" path="/var/lib/kubelet/pods/49c1b7fd-fcf1-494e-b03a-321bb70d4151/volumes"
Oct 31 12:59:25.497429 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:25.497226    5060 generic.go:356] "Generic (PLEG): container finished" podID="40133e45-b954-420a-aa2c-8e5b9c873ef4" containerID="1e156eee88d8e158f3ae297081e63eee06fef2459d71a4f8ade2c3ec2e726d39" exitCode=0
Oct 31 12:59:25.497429 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:25.497353    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d176a5-57f6cd6df5-tzc5c" event={"ID":"40133e45-b954-420a-aa2c-8e5b9c873ef4","Type":"ContainerDied","Data":"1e156eee88d8e158f3ae297081e63eee06fef2459d71a4f8ade2c3ec2e726d39"}
Oct 31 12:59:25.564469 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:25.564433    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d176a5-57f6cd6df5-tzc5c"
Oct 31 12:59:25.645714 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:25.645685    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40133e45-b954-420a-aa2c-8e5b9c873ef4-kube-api-access-7ltmb\" (UniqueName: \"kubernetes.io/projected/40133e45-b954-420a-aa2c-8e5b9c873ef4-kube-api-access-7ltmb\") pod \"40133e45-b954-420a-aa2c-8e5b9c873ef4\" (UID: \"40133e45-b954-420a-aa2c-8e5b9c873ef4\") "
Oct 31 12:59:25.645879 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:25.645820    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40133e45-b954-420a-aa2c-8e5b9c873ef4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40133e45-b954-420a-aa2c-8e5b9c873ef4-plugins-config-volume\") pod \"40133e45-b954-420a-aa2c-8e5b9c873ef4\" (UID: \"40133e45-b954-420a-aa2c-8e5b9c873ef4\") "
Oct 31 12:59:25.646251 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:25.646193    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40133e45-b954-420a-aa2c-8e5b9c873ef4-plugins-config-volume" pod "40133e45-b954-420a-aa2c-8e5b9c873ef4" (UID: "40133e45-b954-420a-aa2c-8e5b9c873ef4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:59:25.648321 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:25.648293    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40133e45-b954-420a-aa2c-8e5b9c873ef4-kube-api-access-7ltmb" pod "40133e45-b954-420a-aa2c-8e5b9c873ef4" (UID: "40133e45-b954-420a-aa2c-8e5b9c873ef4"). InnerVolumeSpecName "kube-api-access-7ltmb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:59:25.748044 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:25.747710    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ltmb\" (UniqueName: \"kubernetes.io/projected/40133e45-b954-420a-aa2c-8e5b9c873ef4-kube-api-access-7ltmb\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:25.748044 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:25.747753    5060 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40133e45-b954-420a-aa2c-8e5b9c873ef4-plugins-config-volume\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:26.131473 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:26.131435    5060 scope.go:122] "RemoveContainer" containerID="5639e6818bba3be7fa72253bfb4ca3ed52dc79c5726b90a8bbf52c4520c10e44"
Oct 31 12:59:26.142467 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:26.142423    5060 scope.go:122] "RemoveContainer" containerID="e3f476e9efd100c5f7f824ca329b93c37eb01200542f98d9f9907991b4aa8045"
Oct 31 12:59:26.152976 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:26.152954    5060 scope.go:122] "RemoveContainer" containerID="1e156eee88d8e158f3ae297081e63eee06fef2459d71a4f8ade2c3ec2e726d39"
Oct 31 12:59:26.163339 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:26.163312    5060 scope.go:122] "RemoveContainer" containerID="5b3e1587d6c3e6ff23958a705b523269aaf7f555ce70a9ce9ad5343ce3d0c431"
Oct 31 12:59:26.173336 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:26.173311    5060 scope.go:122] "RemoveContainer" containerID="028ec2c00286bccbe1187c8fdba35c324f29c29447cf5bc88797c227060c454f"
Oct 31 12:59:26.504942 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:26.504838    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d176a5-57f6cd6df5-tzc5c" event={"ID":"40133e45-b954-420a-aa2c-8e5b9c873ef4","Type":"ContainerDied","Data":"10e04fda01a3c95d435e192e8ce58ac94f335dd629be14e01f4e09b4ab3c90c7"}
Oct 31 12:59:26.504942 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:26.504854    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d176a5-57f6cd6df5-tzc5c"
Oct 31 12:59:26.539787 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:26.539753    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d176a5-57f6cd6df5-tzc5c"]
Oct 31 12:59:26.544067 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:26.544036    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d176a5-57f6cd6df5-tzc5c"]
Oct 31 12:59:27.047690 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:27.047660    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40133e45-b954-420a-aa2c-8e5b9c873ef4" path="/var/lib/kubelet/pods/40133e45-b954-420a-aa2c-8e5b9c873ef4/volumes"
Oct 31 12:59:28.909326 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:28.909139    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-2c393f-openshift-default-6cb59888bf-xrph2"]
Oct 31 12:59:28.909775 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:28.909430    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-2c393f-openshift-default-6cb59888bf-xrph2" podUID="32a2e0ec-5eaa-4cce-a7c0-87512fe277ed" containerName="istio-proxy" containerID="cri-o://a91d050ca52a5d0fa0283a6c2e6d723c813e093ed0c7f6f6d077a58c1334ab52" gracePeriod=30
Oct 31 12:59:29.393017 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:29.392984    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-62734c-pzgfn"]
Oct 31 12:59:29.393666 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:29.393647    5060 cpu_manager.go:414] "containerMap: removing container" podUID="6a1124a2-55e6-424f-a91e-9f89f37db017" containerName="maas-api"
Oct 31 12:59:29.393666 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:29.393665    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a1124a2-55e6-424f-a91e-9f89f37db017" containerName="maas-api"
Oct 31 12:59:29.393813 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:29.393681    5060 cpu_manager.go:414] "containerMap: removing container" podUID="e4b6c141-cbee-48d8-91e7-bdec2b65c7e5" containerName="revoke-keys"
Oct 31 12:59:29.393813 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:29.393689    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4b6c141-cbee-48d8-91e7-bdec2b65c7e5" containerName="revoke-keys"
Oct 31 12:59:29.393813 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:29.393739    5060 cpu_manager.go:414] "containerMap: removing container" podUID="40133e45-b954-420a-aa2c-8e5b9c873ef4" containerName="payload-pre-processing"
Oct 31 12:59:29.393813 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:29.393749    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40133e45-b954-420a-aa2c-8e5b9c873ef4" containerName="payload-pre-processing"
Oct 31 12:59:29.393942 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:29.393870    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a1124a2-55e6-424f-a91e-9f89f37db017" containerName="maas-api"
Oct 31 12:59:29.393942 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:29.393878    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4b6c141-cbee-48d8-91e7-bdec2b65c7e5" containerName="revoke-keys"
Oct 31 12:59:29.393942 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:29.393888    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="40133e45-b954-420a-aa2c-8e5b9c873ef4" containerName="payload-pre-processing"
Oct 31 12:59:29.399519 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:29.399480    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-62734c-pzgfn"
Oct 31 12:59:29.405582 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:29.405559    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7wsmb\""
Oct 31 12:59:29.405770 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:29.405663    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 12:59:29.407861 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:29.407825    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-62734c-pzgfn"]
Oct 31 12:59:29.502952 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:29.502916    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79tv6\" (UniqueName: \"kubernetes.io/projected/16672439-1413-4fc8-81a2-9f6b91770cce-kube-api-access-79tv6\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-62734c-pzgfn\" (UID: \"16672439-1413-4fc8-81a2-9f6b91770cce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-62734c-pzgfn"
Oct 31 12:59:29.503102 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:29.503072    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16672439-1413-4fc8-81a2-9f6b91770cce-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-62734c-pzgfn\" (UID: \"16672439-1413-4fc8-81a2-9f6b91770cce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-62734c-pzgfn"
Oct 31 12:59:29.503102 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:29.503092    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16672439-1413-4fc8-81a2-9f6b91770cce-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-62734c-pzgfn\" (UID: \"16672439-1413-4fc8-81a2-9f6b91770cce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-62734c-pzgfn"
Oct 31 12:59:29.604425 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:29.604388    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79tv6\" (UniqueName: \"kubernetes.io/projected/16672439-1413-4fc8-81a2-9f6b91770cce-kube-api-access-79tv6\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-62734c-pzgfn\" (UID: \"16672439-1413-4fc8-81a2-9f6b91770cce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-62734c-pzgfn"
Oct 31 12:59:29.604663 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:29.604584    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16672439-1413-4fc8-81a2-9f6b91770cce-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-62734c-pzgfn\" (UID: \"16672439-1413-4fc8-81a2-9f6b91770cce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-62734c-pzgfn"
Oct 31 12:59:29.604663 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:29.604620    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16672439-1413-4fc8-81a2-9f6b91770cce-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-62734c-pzgfn\" (UID: \"16672439-1413-4fc8-81a2-9f6b91770cce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-62734c-pzgfn"
Oct 31 12:59:29.605376 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:29.605346    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16672439-1413-4fc8-81a2-9f6b91770cce-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-62734c-pzgfn\" (UID: \"16672439-1413-4fc8-81a2-9f6b91770cce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-62734c-pzgfn"
Oct 31 12:59:29.607021 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:29.606995    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16672439-1413-4fc8-81a2-9f6b91770cce-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-62734c-pzgfn\" (UID: \"16672439-1413-4fc8-81a2-9f6b91770cce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-62734c-pzgfn"
Oct 31 12:59:29.616013 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:29.615983    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79tv6\" (UniqueName: \"kubernetes.io/projected/16672439-1413-4fc8-81a2-9f6b91770cce-kube-api-access-79tv6\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-62734c-pzgfn\" (UID: \"16672439-1413-4fc8-81a2-9f6b91770cce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-62734c-pzgfn"
Oct 31 12:59:29.722760 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:29.722686    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-62734c-pzgfn"
Oct 31 12:59:29.923803 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:59:29.923760    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7phb" containerName="extractor"
Oct 31 12:59:30.025200 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:30.025128    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-62734c-pzgfn"]
Oct 31 12:59:30.025336 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:59:30.025241    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16672439_1413_4fc8_81a2_9f6b91770cce.slice/crio-426546748fe558f75c19c96f86d2436cd27413f7a2f966fa6bc5903cc325d32c WatchSource:0}: Error finding container 426546748fe558f75c19c96f86d2436cd27413f7a2f966fa6bc5903cc325d32c: Status 404 returned error can't find the container with id 426546748fe558f75c19c96f86d2436cd27413f7a2f966fa6bc5903cc325d32c
Oct 31 12:59:30.034575 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:30.034539    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b8de4c2b-4cvrx"]
Oct 31 12:59:30.039755 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:30.039645    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27a1e456-tpdkw"]
Oct 31 12:59:30.043251 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:30.043227    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b8de4c2b-4cvrx"]
Oct 31 12:59:30.046773 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:30.046749    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27a1e456-tpdkw"]
Oct 31 12:59:30.567605 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:30.567520    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-62734c-pzgfn" event={"ID":"16672439-1413-4fc8-81a2-9f6b91770cce","Type":"ContainerStarted","Data":"3fc7124fac130459a49f92a1b45743e6b3427fecbf7479aba369b3902b3efa7f"}
Oct 31 12:59:30.567605 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:30.567578    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-62734c-pzgfn" event={"ID":"16672439-1413-4fc8-81a2-9f6b91770cce","Type":"ContainerStarted","Data":"426546748fe558f75c19c96f86d2436cd27413f7a2f966fa6bc5903cc325d32c"}
Oct 31 12:59:30.609290 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:30.609239    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-62734c-pzgfn" podStartSLOduration=1.609224866 podStartE2EDuration="1.609224866s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:59:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:59:30.607056546 +0000 UTC m=+1933.710124260" watchObservedRunningTime="2026-10-31 12:59:30.609224866 +0000 UTC m=+1933.712292577"
Oct 31 12:59:31.049038 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:31.048994    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1721b308-47a8-44a0-a941-7c09b71f9d69" path="/var/lib/kubelet/pods/1721b308-47a8-44a0-a941-7c09b71f9d69/volumes"
Oct 31 12:59:31.049669 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:31.049649    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f31bbc5b-c0a2-4380-b1c7-e2571b544260" path="/var/lib/kubelet/pods/f31bbc5b-c0a2-4380-b1c7-e2571b544260/volumes"
Oct 31 12:59:31.455057 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:31.455012    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-71bc05-openshift-default-dd6bc8b77-b65mw"]
Oct 31 12:59:31.455905 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:31.455868    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-71bc05-openshift-default-dd6bc8b77-b65mw" podUID="b3992303-31dd-484d-b3ad-3fececdf57fa" containerName="istio-proxy" containerID="cri-o://f3312ffbe5d00b5a088382db79f0b7a35cab76989dea919ce7a3555a8878a7a8" gracePeriod=30
Oct 31 12:59:31.660677 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:31.660634    5060 generic.go:356] "Generic (PLEG): container finished" podID="16672439-1413-4fc8-81a2-9f6b91770cce" containerID="3fc7124fac130459a49f92a1b45743e6b3427fecbf7479aba369b3902b3efa7f" exitCode=0
Oct 31 12:59:31.660857 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:31.660808    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-62734c-pzgfn" event={"ID":"16672439-1413-4fc8-81a2-9f6b91770cce","Type":"ContainerDied","Data":"3fc7124fac130459a49f92a1b45743e6b3427fecbf7479aba369b3902b3efa7f"}
Oct 31 12:59:32.850207 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:32.850173    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-62734c-pzgfn"
Oct 31 12:59:32.965931 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:32.965838    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16672439-1413-4fc8-81a2-9f6b91770cce-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16672439-1413-4fc8-81a2-9f6b91770cce-maas-api-openshift-ca\") pod \"16672439-1413-4fc8-81a2-9f6b91770cce\" (UID: \"16672439-1413-4fc8-81a2-9f6b91770cce\") "
Oct 31 12:59:32.966093 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:32.966038    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16672439-1413-4fc8-81a2-9f6b91770cce-kube-api-access-79tv6\" (UniqueName: \"kubernetes.io/projected/16672439-1413-4fc8-81a2-9f6b91770cce-kube-api-access-79tv6\") pod \"16672439-1413-4fc8-81a2-9f6b91770cce\" (UID: \"16672439-1413-4fc8-81a2-9f6b91770cce\") "
Oct 31 12:59:32.966093 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:32.966075    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/16672439-1413-4fc8-81a2-9f6b91770cce-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16672439-1413-4fc8-81a2-9f6b91770cce-maas-api-xks-ca\") pod \"16672439-1413-4fc8-81a2-9f6b91770cce\" (UID: \"16672439-1413-4fc8-81a2-9f6b91770cce\") "
Oct 31 12:59:32.966271 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:32.966239    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16672439-1413-4fc8-81a2-9f6b91770cce-maas-api-openshift-ca" pod "16672439-1413-4fc8-81a2-9f6b91770cce" (UID: "16672439-1413-4fc8-81a2-9f6b91770cce"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:59:32.966720 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:32.966696    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16672439-1413-4fc8-81a2-9f6b91770cce-maas-api-openshift-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:32.968250 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:32.968215    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16672439-1413-4fc8-81a2-9f6b91770cce-maas-api-xks-ca" pod "16672439-1413-4fc8-81a2-9f6b91770cce" (UID: "16672439-1413-4fc8-81a2-9f6b91770cce"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:59:32.968380 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:32.968288    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16672439-1413-4fc8-81a2-9f6b91770cce-kube-api-access-79tv6" pod "16672439-1413-4fc8-81a2-9f6b91770cce" (UID: "16672439-1413-4fc8-81a2-9f6b91770cce"). InnerVolumeSpecName "kube-api-access-79tv6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:59:33.068522 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:33.068471    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79tv6\" (UniqueName: \"kubernetes.io/projected/16672439-1413-4fc8-81a2-9f6b91770cce-kube-api-access-79tv6\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:33.068522 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:33.068520    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16672439-1413-4fc8-81a2-9f6b91770cce-maas-api-xks-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:33.680212 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:33.680183    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-62734c-pzgfn"
Oct 31 12:59:33.680397 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:33.680220    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-62734c-pzgfn" event={"ID":"16672439-1413-4fc8-81a2-9f6b91770cce","Type":"ContainerDied","Data":"426546748fe558f75c19c96f86d2436cd27413f7a2f966fa6bc5903cc325d32c"}
Oct 31 12:59:33.680397 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:33.680271    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="426546748fe558f75c19c96f86d2436cd27413f7a2f966fa6bc5903cc325d32c"
Oct 31 12:59:34.009356 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:34.009315    5060 patch_prober.go:29] interesting pod/e2e-inf-b-71bc05-openshift-default-dd6bc8b77-b65mw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 12:59:34.009791 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:34.009390    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-71bc05-openshift-default-dd6bc8b77-b65mw" podUID="b3992303-31dd-484d-b3ad-3fececdf57fa" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 12:59:34.196685 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:34.196623    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-2c393f-openshift-default-6cb59888bf-xrph2"
Oct 31 12:59:34.280625 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:34.280587    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/32a2e0ec-5eaa-4cce-a7c0-87512fe277ed-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32a2e0ec-5eaa-4cce-a7c0-87512fe277ed-istio-podinfo\") pod \"32a2e0ec-5eaa-4cce-a7c0-87512fe277ed\" (UID: \"32a2e0ec-5eaa-4cce-a7c0-87512fe277ed\") "
Oct 31 12:59:34.280818 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:34.280688    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32a2e0ec-5eaa-4cce-a7c0-87512fe277ed-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32a2e0ec-5eaa-4cce-a7c0-87512fe277ed-workload-certs\") pod \"32a2e0ec-5eaa-4cce-a7c0-87512fe277ed\" (UID: \"32a2e0ec-5eaa-4cce-a7c0-87512fe277ed\") "
Oct 31 12:59:34.280818 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:34.280787    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32a2e0ec-5eaa-4cce-a7c0-87512fe277ed-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32a2e0ec-5eaa-4cce-a7c0-87512fe277ed-credential-socket\") pod \"32a2e0ec-5eaa-4cce-a7c0-87512fe277ed\" (UID: \"32a2e0ec-5eaa-4cce-a7c0-87512fe277ed\") "
Oct 31 12:59:34.280966 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:34.280873    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32a2e0ec-5eaa-4cce-a7c0-87512fe277ed-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32a2e0ec-5eaa-4cce-a7c0-87512fe277ed-istio-data\") pod \"32a2e0ec-5eaa-4cce-a7c0-87512fe277ed\" (UID: \"32a2e0ec-5eaa-4cce-a7c0-87512fe277ed\") "
Oct 31 12:59:34.281087 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:34.281039    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32a2e0ec-5eaa-4cce-a7c0-87512fe277ed-workload-certs" pod "32a2e0ec-5eaa-4cce-a7c0-87512fe277ed" (UID: "32a2e0ec-5eaa-4cce-a7c0-87512fe277ed"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:59:34.281172 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:34.281119    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32a2e0ec-5eaa-4cce-a7c0-87512fe277ed-credential-socket" pod "32a2e0ec-5eaa-4cce-a7c0-87512fe277ed" (UID: "32a2e0ec-5eaa-4cce-a7c0-87512fe277ed"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:59:34.281172 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:34.281132    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32a2e0ec-5eaa-4cce-a7c0-87512fe277ed-istio-data" pod "32a2e0ec-5eaa-4cce-a7c0-87512fe277ed" (UID: "32a2e0ec-5eaa-4cce-a7c0-87512fe277ed"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:59:34.281298 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:34.281278    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32a2e0ec-5eaa-4cce-a7c0-87512fe277ed-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32a2e0ec-5eaa-4cce-a7c0-87512fe277ed-istiod-ca-cert\") pod \"32a2e0ec-5eaa-4cce-a7c0-87512fe277ed\" (UID: \"32a2e0ec-5eaa-4cce-a7c0-87512fe277ed\") "
Oct 31 12:59:34.281355 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:34.281341    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32a2e0ec-5eaa-4cce-a7c0-87512fe277ed-istio-token\" (UniqueName: \"kubernetes.io/projected/32a2e0ec-5eaa-4cce-a7c0-87512fe277ed-istio-token\") pod \"32a2e0ec-5eaa-4cce-a7c0-87512fe277ed\" (UID: \"32a2e0ec-5eaa-4cce-a7c0-87512fe277ed\") "
Oct 31 12:59:34.281411 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:34.281374    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32a2e0ec-5eaa-4cce-a7c0-87512fe277ed-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32a2e0ec-5eaa-4cce-a7c0-87512fe277ed-istio-envoy\") pod \"32a2e0ec-5eaa-4cce-a7c0-87512fe277ed\" (UID: \"32a2e0ec-5eaa-4cce-a7c0-87512fe277ed\") "
Oct 31 12:59:34.281680 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:34.281655    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32a2e0ec-5eaa-4cce-a7c0-87512fe277ed-istiod-ca-cert" pod "32a2e0ec-5eaa-4cce-a7c0-87512fe277ed" (UID: "32a2e0ec-5eaa-4cce-a7c0-87512fe277ed"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:59:34.281772 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:34.281749    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32a2e0ec-5eaa-4cce-a7c0-87512fe277ed-kube-api-access-27bll\" (UniqueName: \"kubernetes.io/projected/32a2e0ec-5eaa-4cce-a7c0-87512fe277ed-kube-api-access-27bll\") pod \"32a2e0ec-5eaa-4cce-a7c0-87512fe277ed\" (UID: \"32a2e0ec-5eaa-4cce-a7c0-87512fe277ed\") "
Oct 31 12:59:34.281847 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:34.281805    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32a2e0ec-5eaa-4cce-a7c0-87512fe277ed-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32a2e0ec-5eaa-4cce-a7c0-87512fe277ed-workload-socket\") pod \"32a2e0ec-5eaa-4cce-a7c0-87512fe277ed\" (UID: \"32a2e0ec-5eaa-4cce-a7c0-87512fe277ed\") "
Oct 31 12:59:34.282099 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:34.282062    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32a2e0ec-5eaa-4cce-a7c0-87512fe277ed-workload-socket" pod "32a2e0ec-5eaa-4cce-a7c0-87512fe277ed" (UID: "32a2e0ec-5eaa-4cce-a7c0-87512fe277ed"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:59:34.282361 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:34.282338    5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32a2e0ec-5eaa-4cce-a7c0-87512fe277ed-credential-socket\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:34.282361 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:34.282365    5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32a2e0ec-5eaa-4cce-a7c0-87512fe277ed-istio-data\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:34.282694 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:34.282379    5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32a2e0ec-5eaa-4cce-a7c0-87512fe277ed-istiod-ca-cert\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:34.282694 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:34.282393    5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32a2e0ec-5eaa-4cce-a7c0-87512fe277ed-workload-socket\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:34.282694 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:34.282407    5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32a2e0ec-5eaa-4cce-a7c0-87512fe277ed-workload-certs\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:34.284065 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:34.284027    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/32a2e0ec-5eaa-4cce-a7c0-87512fe277ed-istio-podinfo" pod "32a2e0ec-5eaa-4cce-a7c0-87512fe277ed" (UID: "32a2e0ec-5eaa-4cce-a7c0-87512fe277ed"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 12:59:34.284178 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:34.284096    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32a2e0ec-5eaa-4cce-a7c0-87512fe277ed-istio-envoy" pod "32a2e0ec-5eaa-4cce-a7c0-87512fe277ed" (UID: "32a2e0ec-5eaa-4cce-a7c0-87512fe277ed"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:59:34.284178 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:34.284124    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32a2e0ec-5eaa-4cce-a7c0-87512fe277ed-istio-token" pod "32a2e0ec-5eaa-4cce-a7c0-87512fe277ed" (UID: "32a2e0ec-5eaa-4cce-a7c0-87512fe277ed"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:59:34.284303 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:34.284286    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32a2e0ec-5eaa-4cce-a7c0-87512fe277ed-kube-api-access-27bll" pod "32a2e0ec-5eaa-4cce-a7c0-87512fe277ed" (UID: "32a2e0ec-5eaa-4cce-a7c0-87512fe277ed"). InnerVolumeSpecName "kube-api-access-27bll". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:59:34.383937 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:34.383893    5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32a2e0ec-5eaa-4cce-a7c0-87512fe277ed-istio-token\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:34.383937 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:34.383936    5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32a2e0ec-5eaa-4cce-a7c0-87512fe277ed-istio-envoy\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:34.383937 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:34.383947    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-27bll\" (UniqueName: \"kubernetes.io/projected/32a2e0ec-5eaa-4cce-a7c0-87512fe277ed-kube-api-access-27bll\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:34.384186 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:34.383957    5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32a2e0ec-5eaa-4cce-a7c0-87512fe277ed-istio-podinfo\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:34.430044 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:34.430008    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-fd988c-798cc7df69-jdx5x"]
Oct 31 12:59:34.430268 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:34.430243    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fd988c-798cc7df69-jdx5x" podUID="9d442c3e-cd82-423f-92ba-d10ca01bc97b" containerName="maas-api" containerID="cri-o://f403d67a50f834db69ffd1a2b6d2fd9edd1dac184afc268aad5b9ec2676c10a2" gracePeriod=30
Oct 31 12:59:34.696615 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:34.693253    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fd988c-5cf7g"]
Oct 31 12:59:34.711517 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:34.700767    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fd988c-5cf7g"]
Oct 31 12:59:34.717510 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:34.716727    5060 generic.go:356] "Generic (PLEG): container finished" podID="32a2e0ec-5eaa-4cce-a7c0-87512fe277ed" containerID="a91d050ca52a5d0fa0283a6c2e6d723c813e093ed0c7f6f6d077a58c1334ab52" exitCode=0
Oct 31 12:59:34.717510 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:34.716769    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2c393f-openshift-default-6cb59888bf-xrph2" event={"ID":"32a2e0ec-5eaa-4cce-a7c0-87512fe277ed","Type":"ContainerDied","Data":"a91d050ca52a5d0fa0283a6c2e6d723c813e093ed0c7f6f6d077a58c1334ab52"}
Oct 31 12:59:34.717510 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:34.716844    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2c393f-openshift-default-6cb59888bf-xrph2" event={"ID":"32a2e0ec-5eaa-4cce-a7c0-87512fe277ed","Type":"ContainerDied","Data":"5ee2db7fc82a147691cf7870d1dceaecbac3f1b6e5bd9e70fcd6b19cecabac7a"}
Oct 31 12:59:34.717510 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:34.716877    5060 scope.go:122] "RemoveContainer" containerID="a91d050ca52a5d0fa0283a6c2e6d723c813e093ed0c7f6f6d077a58c1334ab52"
Oct 31 12:59:34.733602 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:34.718814    5060 generic.go:356] "Generic (PLEG): container finished" podID="9d442c3e-cd82-423f-92ba-d10ca01bc97b" containerID="f403d67a50f834db69ffd1a2b6d2fd9edd1dac184afc268aad5b9ec2676c10a2" exitCode=0
Oct 31 12:59:34.733602 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:34.718836    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fd988c-798cc7df69-jdx5x" event={"ID":"9d442c3e-cd82-423f-92ba-d10ca01bc97b","Type":"ContainerDied","Data":"f403d67a50f834db69ffd1a2b6d2fd9edd1dac184afc268aad5b9ec2676c10a2"}
Oct 31 12:59:34.733602 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:34.718822    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-2c393f-openshift-default-6cb59888bf-xrph2"
Oct 31 12:59:34.751711 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:34.751678    5060 scope.go:122] "RemoveContainer" containerID="7d49e71e687504e7a0bd13babd59f3c3abded0d6427501c69d03fa9f1cdad5e2"
Oct 31 12:59:34.760688 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:34.760296    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-2c393f-openshift-default-6cb59888bf-xrph2"]
Oct 31 12:59:34.817762 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:34.817664    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-2c393f-openshift-default-6cb59888bf-xrph2"]
Oct 31 12:59:34.823775 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:34.823660    5060 scope.go:122] "RemoveContainer" containerID="a91d050ca52a5d0fa0283a6c2e6d723c813e093ed0c7f6f6d077a58c1334ab52"
Oct 31 12:59:34.824070 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:59:34.824034    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a91d050ca52a5d0fa0283a6c2e6d723c813e093ed0c7f6f6d077a58c1334ab52\": container with ID starting with a91d050ca52a5d0fa0283a6c2e6d723c813e093ed0c7f6f6d077a58c1334ab52 not found: ID does not exist" containerID="a91d050ca52a5d0fa0283a6c2e6d723c813e093ed0c7f6f6d077a58c1334ab52"
Oct 31 12:59:34.824146 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:34.824105    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a91d050ca52a5d0fa0283a6c2e6d723c813e093ed0c7f6f6d077a58c1334ab52"} err="failed to get container status \"a91d050ca52a5d0fa0283a6c2e6d723c813e093ed0c7f6f6d077a58c1334ab52\": rpc error: code = NotFound desc = could not find container \"a91d050ca52a5d0fa0283a6c2e6d723c813e093ed0c7f6f6d077a58c1334ab52\": container with ID starting with a91d050ca52a5d0fa0283a6c2e6d723c813e093ed0c7f6f6d077a58c1334ab52 not found: ID does not exist"
Oct 31 12:59:34.824146 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:34.824130    5060 scope.go:122] "RemoveContainer" containerID="7d49e71e687504e7a0bd13babd59f3c3abded0d6427501c69d03fa9f1cdad5e2"
Oct 31 12:59:34.824539 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:59:34.824511    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d49e71e687504e7a0bd13babd59f3c3abded0d6427501c69d03fa9f1cdad5e2\": container with ID starting with 7d49e71e687504e7a0bd13babd59f3c3abded0d6427501c69d03fa9f1cdad5e2 not found: ID does not exist" containerID="7d49e71e687504e7a0bd13babd59f3c3abded0d6427501c69d03fa9f1cdad5e2"
Oct 31 12:59:34.824643 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:34.824548    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d49e71e687504e7a0bd13babd59f3c3abded0d6427501c69d03fa9f1cdad5e2"} err="failed to get container status \"7d49e71e687504e7a0bd13babd59f3c3abded0d6427501c69d03fa9f1cdad5e2\": rpc error: code = NotFound desc = could not find container \"7d49e71e687504e7a0bd13babd59f3c3abded0d6427501c69d03fa9f1cdad5e2\": container with ID starting with 7d49e71e687504e7a0bd13babd59f3c3abded0d6427501c69d03fa9f1cdad5e2 not found: ID does not exist"
Oct 31 12:59:34.835979 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:34.835958    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fd988c-798cc7df69-jdx5x"
Oct 31 12:59:34.996045 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:34.996009    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d442c3e-cd82-423f-92ba-d10ca01bc97b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d442c3e-cd82-423f-92ba-d10ca01bc97b-maas-api-tls\") pod \"9d442c3e-cd82-423f-92ba-d10ca01bc97b\" (UID: \"9d442c3e-cd82-423f-92ba-d10ca01bc97b\") "
Oct 31 12:59:34.996222 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:34.996094    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d442c3e-cd82-423f-92ba-d10ca01bc97b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d442c3e-cd82-423f-92ba-d10ca01bc97b-openshift-service-ca\") pod \"9d442c3e-cd82-423f-92ba-d10ca01bc97b\" (UID: \"9d442c3e-cd82-423f-92ba-d10ca01bc97b\") "
Oct 31 12:59:34.996222 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:34.996154    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d442c3e-cd82-423f-92ba-d10ca01bc97b-kube-api-access-pxj2x\" (UniqueName: \"kubernetes.io/projected/9d442c3e-cd82-423f-92ba-d10ca01bc97b-kube-api-access-pxj2x\") pod \"9d442c3e-cd82-423f-92ba-d10ca01bc97b\" (UID: \"9d442c3e-cd82-423f-92ba-d10ca01bc97b\") "
Oct 31 12:59:34.996369 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:34.996355    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d442c3e-cd82-423f-92ba-d10ca01bc97b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d442c3e-cd82-423f-92ba-d10ca01bc97b-metrics-certs\") pod \"9d442c3e-cd82-423f-92ba-d10ca01bc97b\" (UID: \"9d442c3e-cd82-423f-92ba-d10ca01bc97b\") "
Oct 31 12:59:34.996587 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:34.996561    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d442c3e-cd82-423f-92ba-d10ca01bc97b-openshift-service-ca" pod "9d442c3e-cd82-423f-92ba-d10ca01bc97b" (UID: "9d442c3e-cd82-423f-92ba-d10ca01bc97b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:59:34.996873 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:34.996854    5060 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d442c3e-cd82-423f-92ba-d10ca01bc97b-openshift-service-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:34.998806 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:34.998770    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d442c3e-cd82-423f-92ba-d10ca01bc97b-maas-api-tls" pod "9d442c3e-cd82-423f-92ba-d10ca01bc97b" (UID: "9d442c3e-cd82-423f-92ba-d10ca01bc97b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:59:34.998925 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:34.998801    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d442c3e-cd82-423f-92ba-d10ca01bc97b-metrics-certs" pod "9d442c3e-cd82-423f-92ba-d10ca01bc97b" (UID: "9d442c3e-cd82-423f-92ba-d10ca01bc97b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:59:34.998925 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:34.998845    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d442c3e-cd82-423f-92ba-d10ca01bc97b-kube-api-access-pxj2x" pod "9d442c3e-cd82-423f-92ba-d10ca01bc97b" (UID: "9d442c3e-cd82-423f-92ba-d10ca01bc97b"). InnerVolumeSpecName "kube-api-access-pxj2x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:59:35.048702 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:35.048666    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32a2e0ec-5eaa-4cce-a7c0-87512fe277ed" path="/var/lib/kubelet/pods/32a2e0ec-5eaa-4cce-a7c0-87512fe277ed/volumes"
Oct 31 12:59:35.049159 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:35.049144    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4b6c141-cbee-48d8-91e7-bdec2b65c7e5" path="/var/lib/kubelet/pods/e4b6c141-cbee-48d8-91e7-bdec2b65c7e5/volumes"
Oct 31 12:59:35.097793 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:35.097718    5060 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d442c3e-cd82-423f-92ba-d10ca01bc97b-metrics-certs\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:35.097793 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:35.097748    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d442c3e-cd82-423f-92ba-d10ca01bc97b-maas-api-tls\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:35.097793 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:35.097758    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxj2x\" (UniqueName: \"kubernetes.io/projected/9d442c3e-cd82-423f-92ba-d10ca01bc97b-kube-api-access-pxj2x\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:35.729015 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:35.728980    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fd988c-798cc7df69-jdx5x"
Oct 31 12:59:35.729217 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:35.729024    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fd988c-798cc7df69-jdx5x" event={"ID":"9d442c3e-cd82-423f-92ba-d10ca01bc97b","Type":"ContainerDied","Data":"1c56b7a3074ab562ee0bc3aa7051f11a5c54eda8e3afb03eb3e14caf7180f123"}
Oct 31 12:59:35.729217 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:35.729092    5060 scope.go:122] "RemoveContainer" containerID="f403d67a50f834db69ffd1a2b6d2fd9edd1dac184afc268aad5b9ec2676c10a2"
Oct 31 12:59:35.767046 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:35.767009    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-fd988c-798cc7df69-jdx5x"]
Oct 31 12:59:35.772664 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:35.772634    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-fd988c-798cc7df69-jdx5x"]
Oct 31 12:59:36.750409 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:36.750375    5060 generic.go:356] "Generic (PLEG): container finished" podID="b3992303-31dd-484d-b3ad-3fececdf57fa" containerID="f3312ffbe5d00b5a088382db79f0b7a35cab76989dea919ce7a3555a8878a7a8" exitCode=0
Oct 31 12:59:36.750819 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:36.750451    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-71bc05-openshift-default-dd6bc8b77-b65mw" event={"ID":"b3992303-31dd-484d-b3ad-3fececdf57fa","Type":"ContainerDied","Data":"f3312ffbe5d00b5a088382db79f0b7a35cab76989dea919ce7a3555a8878a7a8"}
Oct 31 12:59:36.750819 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:36.750563    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-71bc05-openshift-default-dd6bc8b77-b65mw" event={"ID":"b3992303-31dd-484d-b3ad-3fececdf57fa","Type":"ContainerDied","Data":"83309eb552ff9b9ac2d79b78e2d5606c33e8c50b44c711e80bb3ecfa6e1b5a70"}
Oct 31 12:59:36.750819 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:36.750582    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="83309eb552ff9b9ac2d79b78e2d5606c33e8c50b44c711e80bb3ecfa6e1b5a70"
Oct 31 12:59:36.750819 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:36.750598    5060 scope.go:122] "RemoveContainer" containerID="e33caf28b00f04205fa360aaf1fadccce6cb70b25e057f52a58a026e4d2c2ea6"
Oct 31 12:59:36.753761 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:36.753743    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-71bc05-openshift-default-dd6bc8b77-b65mw"
Oct 31 12:59:36.918127 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:36.918091    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b3992303-31dd-484d-b3ad-3fececdf57fa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3992303-31dd-484d-b3ad-3fececdf57fa-istio-podinfo\") pod \"b3992303-31dd-484d-b3ad-3fececdf57fa\" (UID: \"b3992303-31dd-484d-b3ad-3fececdf57fa\") "
Oct 31 12:59:36.918289 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:36.918209    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3992303-31dd-484d-b3ad-3fececdf57fa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3992303-31dd-484d-b3ad-3fececdf57fa-istiod-ca-cert\") pod \"b3992303-31dd-484d-b3ad-3fececdf57fa\" (UID: \"b3992303-31dd-484d-b3ad-3fececdf57fa\") "
Oct 31 12:59:36.918289 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:36.918240    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3992303-31dd-484d-b3ad-3fececdf57fa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3992303-31dd-484d-b3ad-3fececdf57fa-credential-socket\") pod \"b3992303-31dd-484d-b3ad-3fececdf57fa\" (UID: \"b3992303-31dd-484d-b3ad-3fececdf57fa\") "
Oct 31 12:59:36.918289 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:36.918263    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3992303-31dd-484d-b3ad-3fececdf57fa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3992303-31dd-484d-b3ad-3fececdf57fa-istio-envoy\") pod \"b3992303-31dd-484d-b3ad-3fececdf57fa\" (UID: \"b3992303-31dd-484d-b3ad-3fececdf57fa\") "
Oct 31 12:59:36.918289 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:36.918289    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3992303-31dd-484d-b3ad-3fececdf57fa-istio-token\" (UniqueName: \"kubernetes.io/projected/b3992303-31dd-484d-b3ad-3fececdf57fa-istio-token\") pod \"b3992303-31dd-484d-b3ad-3fececdf57fa\" (UID: \"b3992303-31dd-484d-b3ad-3fececdf57fa\") "
Oct 31 12:59:36.918555 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:36.918313    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3992303-31dd-484d-b3ad-3fececdf57fa-kube-api-access-btcs6\" (UniqueName: \"kubernetes.io/projected/b3992303-31dd-484d-b3ad-3fececdf57fa-kube-api-access-btcs6\") pod \"b3992303-31dd-484d-b3ad-3fececdf57fa\" (UID: \"b3992303-31dd-484d-b3ad-3fececdf57fa\") "
Oct 31 12:59:36.918555 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:36.918376    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3992303-31dd-484d-b3ad-3fececdf57fa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3992303-31dd-484d-b3ad-3fececdf57fa-workload-certs\") pod \"b3992303-31dd-484d-b3ad-3fececdf57fa\" (UID: \"b3992303-31dd-484d-b3ad-3fececdf57fa\") "
Oct 31 12:59:36.918698 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:36.918560    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3992303-31dd-484d-b3ad-3fececdf57fa-credential-socket" pod "b3992303-31dd-484d-b3ad-3fececdf57fa" (UID: "b3992303-31dd-484d-b3ad-3fececdf57fa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:59:36.918767 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:36.918690    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3992303-31dd-484d-b3ad-3fececdf57fa-workload-certs" pod "b3992303-31dd-484d-b3ad-3fececdf57fa" (UID: "b3992303-31dd-484d-b3ad-3fececdf57fa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:59:36.918767 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:36.918706    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3992303-31dd-484d-b3ad-3fececdf57fa-istiod-ca-cert" pod "b3992303-31dd-484d-b3ad-3fececdf57fa" (UID: "b3992303-31dd-484d-b3ad-3fececdf57fa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:59:36.918767 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:36.918730    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3992303-31dd-484d-b3ad-3fececdf57fa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3992303-31dd-484d-b3ad-3fececdf57fa-workload-socket\") pod \"b3992303-31dd-484d-b3ad-3fececdf57fa\" (UID: \"b3992303-31dd-484d-b3ad-3fececdf57fa\") "
Oct 31 12:59:36.918932 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:36.918853    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3992303-31dd-484d-b3ad-3fececdf57fa-workload-socket" pod "b3992303-31dd-484d-b3ad-3fececdf57fa" (UID: "b3992303-31dd-484d-b3ad-3fececdf57fa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:59:36.918932 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:36.918909    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3992303-31dd-484d-b3ad-3fececdf57fa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3992303-31dd-484d-b3ad-3fececdf57fa-istio-data\") pod \"b3992303-31dd-484d-b3ad-3fececdf57fa\" (UID: \"b3992303-31dd-484d-b3ad-3fececdf57fa\") "
Oct 31 12:59:36.919087 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:36.919064    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3992303-31dd-484d-b3ad-3fececdf57fa-istio-data" pod "b3992303-31dd-484d-b3ad-3fececdf57fa" (UID: "b3992303-31dd-484d-b3ad-3fececdf57fa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:59:36.919469 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:36.919442    5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3992303-31dd-484d-b3ad-3fececdf57fa-workload-certs\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:36.919595 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:36.919475    5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3992303-31dd-484d-b3ad-3fececdf57fa-workload-socket\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:36.919595 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:36.919528    5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3992303-31dd-484d-b3ad-3fececdf57fa-istio-data\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:36.919595 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:36.919544    5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3992303-31dd-484d-b3ad-3fececdf57fa-istiod-ca-cert\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:36.919595 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:36.919558    5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3992303-31dd-484d-b3ad-3fececdf57fa-credential-socket\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:36.920664 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:36.920637    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3992303-31dd-484d-b3ad-3fececdf57fa-istio-token" pod "b3992303-31dd-484d-b3ad-3fececdf57fa" (UID: "b3992303-31dd-484d-b3ad-3fececdf57fa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:59:36.920771 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:36.920685    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3992303-31dd-484d-b3ad-3fececdf57fa-istio-envoy" pod "b3992303-31dd-484d-b3ad-3fececdf57fa" (UID: "b3992303-31dd-484d-b3ad-3fececdf57fa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:59:36.920880 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:36.920836    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b3992303-31dd-484d-b3ad-3fececdf57fa-istio-podinfo" pod "b3992303-31dd-484d-b3ad-3fececdf57fa" (UID: "b3992303-31dd-484d-b3ad-3fececdf57fa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 12:59:36.920939 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:36.920891    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3992303-31dd-484d-b3ad-3fececdf57fa-kube-api-access-btcs6" pod "b3992303-31dd-484d-b3ad-3fececdf57fa" (UID: "b3992303-31dd-484d-b3ad-3fececdf57fa"). InnerVolumeSpecName "kube-api-access-btcs6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:59:36.976915 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:36.976838    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-b25d3b/distinct-2-w5-b25d3b-kserve-76f798d59f-vvzhg"]
Oct 31 12:59:37.021233 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:37.021198    5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3992303-31dd-484d-b3ad-3fececdf57fa-istio-podinfo\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:37.021233 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:37.021225    5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3992303-31dd-484d-b3ad-3fececdf57fa-istio-envoy\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:37.021233 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:37.021235    5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3992303-31dd-484d-b3ad-3fececdf57fa-istio-token\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:37.021476 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:37.021247    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-btcs6\" (UniqueName: \"kubernetes.io/projected/b3992303-31dd-484d-b3ad-3fececdf57fa-kube-api-access-btcs6\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:37.050382 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:37.050344    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d442c3e-cd82-423f-92ba-d10ca01bc97b" path="/var/lib/kubelet/pods/9d442c3e-cd82-423f-92ba-d10ca01bc97b/volumes"
Oct 31 12:59:37.752104 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:37.752057    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fd988c-798cc7df69-jdx5x" podUID="9d442c3e-cd82-423f-92ba-d10ca01bc97b" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.235:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:59:37.760885 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:37.760861    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-71bc05-openshift-default-dd6bc8b77-b65mw"
Oct 31 12:59:37.803346 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:37.803309    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-71bc05-openshift-default-dd6bc8b77-b65mw"]
Oct 31 12:59:37.808734 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:37.808703    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-71bc05-openshift-default-dd6bc8b77-b65mw"]
Oct 31 12:59:39.047863 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:39.047824    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3992303-31dd-484d-b3ad-3fececdf57fa" path="/var/lib/kubelet/pods/b3992303-31dd-484d-b3ad-3fececdf57fa/volumes"
Oct 31 12:59:42.263119 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:42.263078    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-b25d3b-66f84f5d8f-6w4n2"]
Oct 31 12:59:42.393081 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:42.393042    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-fd988c-openshift-default-96bd4cd5c-g4dgc"]
Oct 31 12:59:42.393345 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:42.393300    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-fd988c-openshift-default-96bd4cd5c-g4dgc" podUID="0c93ff68-7e63-40b2-882b-8b2aed13e982" containerName="istio-proxy" containerID="cri-o://e162b85c3c8d5b2f01ce21d6b80bffca81e78ad33c6852cd87ec34686222ad6d" gracePeriod=30
Oct 31 12:59:42.931328 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:42.931292    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-9b17b9-t2n9k"]
Oct 31 12:59:42.931852 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:42.931838    5060 cpu_manager.go:414] "containerMap: removing container" podUID="32a2e0ec-5eaa-4cce-a7c0-87512fe277ed" containerName="istio-proxy"
Oct 31 12:59:42.931916 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:42.931854    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32a2e0ec-5eaa-4cce-a7c0-87512fe277ed" containerName="istio-proxy"
Oct 31 12:59:42.931916 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:42.931873    5060 cpu_manager.go:414] "containerMap: removing container" podUID="b3992303-31dd-484d-b3ad-3fececdf57fa" containerName="istio-proxy"
Oct 31 12:59:42.931916 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:42.931878    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3992303-31dd-484d-b3ad-3fececdf57fa" containerName="istio-proxy"
Oct 31 12:59:42.931916 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:42.931888    5060 cpu_manager.go:414] "containerMap: removing container" podUID="9d442c3e-cd82-423f-92ba-d10ca01bc97b" containerName="maas-api"
Oct 31 12:59:42.931916 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:42.931893    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d442c3e-cd82-423f-92ba-d10ca01bc97b" containerName="maas-api"
Oct 31 12:59:42.931916 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:42.931910    5060 cpu_manager.go:414] "containerMap: removing container" podUID="16672439-1413-4fc8-81a2-9f6b91770cce" containerName="revoke-keys"
Oct 31 12:59:42.931916 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:42.931915    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16672439-1413-4fc8-81a2-9f6b91770cce" containerName="revoke-keys"
Oct 31 12:59:42.932153 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:42.932004    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3992303-31dd-484d-b3ad-3fececdf57fa" containerName="istio-proxy"
Oct 31 12:59:42.932153 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:42.932010    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d442c3e-cd82-423f-92ba-d10ca01bc97b" containerName="maas-api"
Oct 31 12:59:42.932153 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:42.932020    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="16672439-1413-4fc8-81a2-9f6b91770cce" containerName="revoke-keys"
Oct 31 12:59:42.932153 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:42.932026    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="32a2e0ec-5eaa-4cce-a7c0-87512fe277ed" containerName="istio-proxy"
Oct 31 12:59:42.935340 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:42.935318    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-9b17b9-t2n9k"
Oct 31 12:59:42.941788 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:42.941760    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 12:59:42.941946 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:42.941853    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 12:59:42.941946 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:42.941773    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 12:59:42.942059 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:42.941976    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7wsmb\""
Oct 31 12:59:42.946866 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:42.946842    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-9b17b9-t2n9k"]
Oct 31 12:59:43.001482 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:43.001446    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xj25\" (UniqueName: \"kubernetes.io/projected/b5d32cf4-f969-45b2-a94c-273809cbf9ed-kube-api-access-7xj25\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-9b17b9-t2n9k\" (UID: \"b5d32cf4-f969-45b2-a94c-273809cbf9ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-9b17b9-t2n9k"
Oct 31 12:59:43.001681 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:43.001572    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5d32cf4-f969-45b2-a94c-273809cbf9ed-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-9b17b9-t2n9k\" (UID: \"b5d32cf4-f969-45b2-a94c-273809cbf9ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-9b17b9-t2n9k"
Oct 31 12:59:43.001681 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:43.001648    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5d32cf4-f969-45b2-a94c-273809cbf9ed-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-9b17b9-t2n9k\" (UID: \"b5d32cf4-f969-45b2-a94c-273809cbf9ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-9b17b9-t2n9k"
Oct 31 12:59:43.102706 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:43.102662    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5d32cf4-f969-45b2-a94c-273809cbf9ed-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-9b17b9-t2n9k\" (UID: \"b5d32cf4-f969-45b2-a94c-273809cbf9ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-9b17b9-t2n9k"
Oct 31 12:59:43.102915 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:43.102880    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xj25\" (UniqueName: \"kubernetes.io/projected/b5d32cf4-f969-45b2-a94c-273809cbf9ed-kube-api-access-7xj25\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-9b17b9-t2n9k\" (UID: \"b5d32cf4-f969-45b2-a94c-273809cbf9ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-9b17b9-t2n9k"
Oct 31 12:59:43.102998 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:43.102946    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5d32cf4-f969-45b2-a94c-273809cbf9ed-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-9b17b9-t2n9k\" (UID: \"b5d32cf4-f969-45b2-a94c-273809cbf9ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-9b17b9-t2n9k"
Oct 31 12:59:43.103764 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:43.103737    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5d32cf4-f969-45b2-a94c-273809cbf9ed-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-9b17b9-t2n9k\" (UID: \"b5d32cf4-f969-45b2-a94c-273809cbf9ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-9b17b9-t2n9k"
Oct 31 12:59:43.105298 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:43.105273    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5d32cf4-f969-45b2-a94c-273809cbf9ed-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-9b17b9-t2n9k\" (UID: \"b5d32cf4-f969-45b2-a94c-273809cbf9ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-9b17b9-t2n9k"
Oct 31 12:59:43.115057 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:43.115030    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xj25\" (UniqueName: \"kubernetes.io/projected/b5d32cf4-f969-45b2-a94c-273809cbf9ed-kube-api-access-7xj25\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-9b17b9-t2n9k\" (UID: \"b5d32cf4-f969-45b2-a94c-273809cbf9ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-9b17b9-t2n9k"
Oct 31 12:59:43.252427 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:43.252349    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-9b17b9-t2n9k"
Oct 31 12:59:43.547108 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:43.547067    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-9b17b9-t2n9k"]
Oct 31 12:59:43.557178 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:59:43.557095    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5d32cf4_f969_45b2_a94c_273809cbf9ed.slice/crio-285c2a97d412715cd12fd981e0959d2c70de573cc90fcc5a5b84e92faa426cc7 WatchSource:0}: Error finding container 285c2a97d412715cd12fd981e0959d2c70de573cc90fcc5a5b84e92faa426cc7: Status 404 returned error can't find the container with id 285c2a97d412715cd12fd981e0959d2c70de573cc90fcc5a5b84e92faa426cc7
Oct 31 12:59:43.815897 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:43.815809    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-9b17b9-t2n9k" event={"ID":"b5d32cf4-f969-45b2-a94c-273809cbf9ed","Type":"ContainerStarted","Data":"285c2a97d412715cd12fd981e0959d2c70de573cc90fcc5a5b84e92faa426cc7"}
Oct 31 12:59:44.278600 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:44.278560    5060 patch_prober.go:29] interesting pod/e2e-inf-b-fd988c-openshift-default-96bd4cd5c-g4dgc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 12:59:44.278768 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:44.278628    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-fd988c-openshift-default-96bd4cd5c-g4dgc" podUID="0c93ff68-7e63-40b2-882b-8b2aed13e982" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 12:59:44.827991 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:44.827953    5060 generic.go:356] "Generic (PLEG): container finished" podID="b5d32cf4-f969-45b2-a94c-273809cbf9ed" containerID="ebbd7fa4f5bf47ae102fa6c68be365e19defd1b0d65517ab20f9d1d0f19721ae" exitCode=0
Oct 31 12:59:44.828765 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:44.828723    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-9b17b9-t2n9k" event={"ID":"b5d32cf4-f969-45b2-a94c-273809cbf9ed","Type":"ContainerDied","Data":"ebbd7fa4f5bf47ae102fa6c68be365e19defd1b0d65517ab20f9d1d0f19721ae"}
Oct 31 12:59:44.828986 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:44.828958    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b95e1a3-82c69"]
Oct 31 12:59:44.829726 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:44.829704    5060 cpu_manager.go:414] "containerMap: removing container" podUID="32a2e0ec-5eaa-4cce-a7c0-87512fe277ed" containerName="istio-proxy"
Oct 31 12:59:44.829726 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:44.829724    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32a2e0ec-5eaa-4cce-a7c0-87512fe277ed" containerName="istio-proxy"
Oct 31 12:59:44.829864 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:44.829743    5060 cpu_manager.go:414] "containerMap: removing container" podUID="b3992303-31dd-484d-b3ad-3fececdf57fa" containerName="istio-proxy"
Oct 31 12:59:44.829864 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:44.829748    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3992303-31dd-484d-b3ad-3fececdf57fa" containerName="istio-proxy"
Oct 31 12:59:44.829864 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:44.829820    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3992303-31dd-484d-b3ad-3fececdf57fa" containerName="istio-proxy"
Oct 31 12:59:44.829864 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:44.829832    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="32a2e0ec-5eaa-4cce-a7c0-87512fe277ed" containerName="istio-proxy"
Oct 31 12:59:44.833648 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:44.833630    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b95e1a3-82c69"
Oct 31 12:59:44.856786 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:44.845874    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b95e1a3-82c69"]
Oct 31 12:59:44.927371 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:44.927329    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkmq6\" (UniqueName: \"kubernetes.io/projected/61ba0787-c638-4376-86d9-4c1c90dd8b6d-kube-api-access-xkmq6\") pod \"maas-api-revoke-sub-4b95e1a3-82c69\" (UID: \"61ba0787-c638-4376-86d9-4c1c90dd8b6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b95e1a3-82c69"
Oct 31 12:59:44.927568 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:44.927382    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61ba0787-c638-4376-86d9-4c1c90dd8b6d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4b95e1a3-82c69\" (UID: \"61ba0787-c638-4376-86d9-4c1c90dd8b6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b95e1a3-82c69"
Oct 31 12:59:44.927687 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:44.927664    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61ba0787-c638-4376-86d9-4c1c90dd8b6d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4b95e1a3-82c69\" (UID: \"61ba0787-c638-4376-86d9-4c1c90dd8b6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b95e1a3-82c69"
Oct 31 12:59:45.029447 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:45.029409    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkmq6\" (UniqueName: \"kubernetes.io/projected/61ba0787-c638-4376-86d9-4c1c90dd8b6d-kube-api-access-xkmq6\") pod \"maas-api-revoke-sub-4b95e1a3-82c69\" (UID: \"61ba0787-c638-4376-86d9-4c1c90dd8b6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b95e1a3-82c69"
Oct 31 12:59:45.029665 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:45.029457    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61ba0787-c638-4376-86d9-4c1c90dd8b6d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4b95e1a3-82c69\" (UID: \"61ba0787-c638-4376-86d9-4c1c90dd8b6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b95e1a3-82c69"
Oct 31 12:59:45.029665 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:45.029624    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61ba0787-c638-4376-86d9-4c1c90dd8b6d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4b95e1a3-82c69\" (UID: \"61ba0787-c638-4376-86d9-4c1c90dd8b6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b95e1a3-82c69"
Oct 31 12:59:45.030300 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:45.030266    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61ba0787-c638-4376-86d9-4c1c90dd8b6d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4b95e1a3-82c69\" (UID: \"61ba0787-c638-4376-86d9-4c1c90dd8b6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b95e1a3-82c69"
Oct 31 12:59:45.032083 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:45.032053    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61ba0787-c638-4376-86d9-4c1c90dd8b6d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4b95e1a3-82c69\" (UID: \"61ba0787-c638-4376-86d9-4c1c90dd8b6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b95e1a3-82c69"
Oct 31 12:59:45.042774 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:45.042378    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkmq6\" (UniqueName: \"kubernetes.io/projected/61ba0787-c638-4376-86d9-4c1c90dd8b6d-kube-api-access-xkmq6\") pod \"maas-api-revoke-sub-4b95e1a3-82c69\" (UID: \"61ba0787-c638-4376-86d9-4c1c90dd8b6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b95e1a3-82c69"
Oct 31 12:59:45.161675 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:45.161588    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b95e1a3-82c69"
Oct 31 12:59:45.263553 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:45.248817    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-62734c-pzgfn"]
Oct 31 12:59:45.263553 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:45.261854    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-62734c-pzgfn"]
Oct 31 12:59:45.452053 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:45.452008    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b95e1a3-82c69"]
Oct 31 12:59:45.455009 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:59:45.454973    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61ba0787_c638_4376_86d9_4c1c90dd8b6d.slice/crio-fed1ff44d2f513a98cc472eed73026bd46589d7dd615c1d36b19412d22bba7df WatchSource:0}: Error finding container fed1ff44d2f513a98cc472eed73026bd46589d7dd615c1d36b19412d22bba7df: Status 404 returned error can't find the container with id fed1ff44d2f513a98cc472eed73026bd46589d7dd615c1d36b19412d22bba7df
Oct 31 12:59:45.838814 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:45.838757    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b95e1a3-82c69" event={"ID":"61ba0787-c638-4376-86d9-4c1c90dd8b6d","Type":"ContainerStarted","Data":"dfeda8a20ae6abe67354c38176d3ce94af9c708d58adf1d82fb9b2a0e395487e"}
Oct 31 12:59:45.839268 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:45.838824    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b95e1a3-82c69" event={"ID":"61ba0787-c638-4376-86d9-4c1c90dd8b6d","Type":"ContainerStarted","Data":"fed1ff44d2f513a98cc472eed73026bd46589d7dd615c1d36b19412d22bba7df"}
Oct 31 12:59:45.859472 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:45.859404    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b95e1a3-82c69" podStartSLOduration=1.859384662 podStartE2EDuration="1.859384662s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:59:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:59:45.856893882 +0000 UTC m=+1948.959961595" watchObservedRunningTime="2026-10-31 12:59:45.859384662 +0000 UTC m=+1948.962452375"
Oct 31 12:59:46.019471 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:46.019410    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-9b17b9-t2n9k"
Oct 31 12:59:46.045383 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:46.045348    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5d32cf4-f969-45b2-a94c-273809cbf9ed-kube-api-access-7xj25\" (UniqueName: \"kubernetes.io/projected/b5d32cf4-f969-45b2-a94c-273809cbf9ed-kube-api-access-7xj25\") pod \"b5d32cf4-f969-45b2-a94c-273809cbf9ed\" (UID: \"b5d32cf4-f969-45b2-a94c-273809cbf9ed\") "
Oct 31 12:59:46.045587 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:46.045435    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5d32cf4-f969-45b2-a94c-273809cbf9ed-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5d32cf4-f969-45b2-a94c-273809cbf9ed-maas-api-openshift-ca\") pod \"b5d32cf4-f969-45b2-a94c-273809cbf9ed\" (UID: \"b5d32cf4-f969-45b2-a94c-273809cbf9ed\") "
Oct 31 12:59:46.045587 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:46.045538    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5d32cf4-f969-45b2-a94c-273809cbf9ed-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5d32cf4-f969-45b2-a94c-273809cbf9ed-maas-api-xks-ca\") pod \"b5d32cf4-f969-45b2-a94c-273809cbf9ed\" (UID: \"b5d32cf4-f969-45b2-a94c-273809cbf9ed\") "
Oct 31 12:59:46.046165 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:46.046121    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5d32cf4-f969-45b2-a94c-273809cbf9ed-maas-api-openshift-ca" pod "b5d32cf4-f969-45b2-a94c-273809cbf9ed" (UID: "b5d32cf4-f969-45b2-a94c-273809cbf9ed"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:59:46.048246 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:46.048218    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5d32cf4-f969-45b2-a94c-273809cbf9ed-kube-api-access-7xj25" pod "b5d32cf4-f969-45b2-a94c-273809cbf9ed" (UID: "b5d32cf4-f969-45b2-a94c-273809cbf9ed"). InnerVolumeSpecName "kube-api-access-7xj25". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:59:46.048355 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:46.048224    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5d32cf4-f969-45b2-a94c-273809cbf9ed-maas-api-xks-ca" pod "b5d32cf4-f969-45b2-a94c-273809cbf9ed" (UID: "b5d32cf4-f969-45b2-a94c-273809cbf9ed"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:59:46.147675 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:46.147476    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5d32cf4-f969-45b2-a94c-273809cbf9ed-maas-api-xks-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:46.147675 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:46.147543    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7xj25\" (UniqueName: \"kubernetes.io/projected/b5d32cf4-f969-45b2-a94c-273809cbf9ed-kube-api-access-7xj25\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:46.147675 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:46.147560    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5d32cf4-f969-45b2-a94c-273809cbf9ed-maas-api-openshift-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:46.851292 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:46.851254    5060 generic.go:356] "Generic (PLEG): container finished" podID="61ba0787-c638-4376-86d9-4c1c90dd8b6d" containerID="dfeda8a20ae6abe67354c38176d3ce94af9c708d58adf1d82fb9b2a0e395487e" exitCode=0
Oct 31 12:59:46.851810 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:46.851339    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b95e1a3-82c69" event={"ID":"61ba0787-c638-4376-86d9-4c1c90dd8b6d","Type":"ContainerDied","Data":"dfeda8a20ae6abe67354c38176d3ce94af9c708d58adf1d82fb9b2a0e395487e"}
Oct 31 12:59:46.853367 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:46.853339    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-9b17b9-t2n9k" event={"ID":"b5d32cf4-f969-45b2-a94c-273809cbf9ed","Type":"ContainerDied","Data":"285c2a97d412715cd12fd981e0959d2c70de573cc90fcc5a5b84e92faa426cc7"}
Oct 31 12:59:46.853477 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:46.853368    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-9b17b9-t2n9k"
Oct 31 12:59:46.853477 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:46.853387    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="285c2a97d412715cd12fd981e0959d2c70de573cc90fcc5a5b84e92faa426cc7"
Oct 31 12:59:46.929915 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:46.929872    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-v2sv5"]
Oct 31 12:59:46.930117 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:46.930090    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-v2sv5" podUID="1e33b790-15e5-4754-929f-c0a7330b9361" containerName="istio-proxy" containerID="cri-o://61d1f66705dbeb17fd9cbd694f56a41316654ebc7e876f422e77fca134c16d3f" gracePeriod=30
Oct 31 12:59:47.049840 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:47.049800    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16672439-1413-4fc8-81a2-9f6b91770cce" path="/var/lib/kubelet/pods/16672439-1413-4fc8-81a2-9f6b91770cce/volumes"
Oct 31 12:59:47.264080 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:47.263977    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w5-b25d3b-66f84f5d8f-6w4n2" podUID="85cd1407-54f9-493a-b8fd-9e638a6615a5" containerName="payload-processing" containerID="cri-o://751646ab827e81ab96b1b3b08200b865727aa93d32279c669f803a16eb4d57ba" gracePeriod=25
Oct 31 12:59:47.883002 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:47.882962    5060 generic.go:356] "Generic (PLEG): container finished" podID="0c93ff68-7e63-40b2-882b-8b2aed13e982" containerID="e162b85c3c8d5b2f01ce21d6b80bffca81e78ad33c6852cd87ec34686222ad6d" exitCode=0
Oct 31 12:59:47.883462 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:47.883033    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-fd988c-openshift-default-96bd4cd5c-g4dgc" event={"ID":"0c93ff68-7e63-40b2-882b-8b2aed13e982","Type":"ContainerDied","Data":"e162b85c3c8d5b2f01ce21d6b80bffca81e78ad33c6852cd87ec34686222ad6d"}
Oct 31 12:59:47.883462 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:47.883109    5060 scope.go:122] "RemoveContainer" containerID="55e1b15a513434f3d2d8350ad52e7342327fa7ab784312e346b4241a14ba93ee"
Oct 31 12:59:48.066253 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:48.066223    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b95e1a3-82c69"
Oct 31 12:59:48.116992 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:48.116961    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-fd988c-openshift-default-96bd4cd5c-g4dgc"
Oct 31 12:59:48.181362 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:48.181258    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61ba0787-c638-4376-86d9-4c1c90dd8b6d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61ba0787-c638-4376-86d9-4c1c90dd8b6d-maas-api-openshift-ca\") pod \"61ba0787-c638-4376-86d9-4c1c90dd8b6d\" (UID: \"61ba0787-c638-4376-86d9-4c1c90dd8b6d\") "
Oct 31 12:59:48.181362 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:48.181305    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c93ff68-7e63-40b2-882b-8b2aed13e982-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c93ff68-7e63-40b2-882b-8b2aed13e982-credential-socket\") pod \"0c93ff68-7e63-40b2-882b-8b2aed13e982\" (UID: \"0c93ff68-7e63-40b2-882b-8b2aed13e982\") "
Oct 31 12:59:48.181362 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:48.181358    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c93ff68-7e63-40b2-882b-8b2aed13e982-istio-token\" (UniqueName: \"kubernetes.io/projected/0c93ff68-7e63-40b2-882b-8b2aed13e982-istio-token\") pod \"0c93ff68-7e63-40b2-882b-8b2aed13e982\" (UID: \"0c93ff68-7e63-40b2-882b-8b2aed13e982\") "
Oct 31 12:59:48.181804 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:48.181394    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61ba0787-c638-4376-86d9-4c1c90dd8b6d-kube-api-access-xkmq6\" (UniqueName: \"kubernetes.io/projected/61ba0787-c638-4376-86d9-4c1c90dd8b6d-kube-api-access-xkmq6\") pod \"61ba0787-c638-4376-86d9-4c1c90dd8b6d\" (UID: \"61ba0787-c638-4376-86d9-4c1c90dd8b6d\") "
Oct 31 12:59:48.181804 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:48.181459    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c93ff68-7e63-40b2-882b-8b2aed13e982-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c93ff68-7e63-40b2-882b-8b2aed13e982-workload-socket\") pod \"0c93ff68-7e63-40b2-882b-8b2aed13e982\" (UID: \"0c93ff68-7e63-40b2-882b-8b2aed13e982\") "
Oct 31 12:59:48.181804 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:48.181483    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c93ff68-7e63-40b2-882b-8b2aed13e982-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c93ff68-7e63-40b2-882b-8b2aed13e982-istio-data\") pod \"0c93ff68-7e63-40b2-882b-8b2aed13e982\" (UID: \"0c93ff68-7e63-40b2-882b-8b2aed13e982\") "
Oct 31 12:59:48.181804 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:48.181524    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c93ff68-7e63-40b2-882b-8b2aed13e982-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c93ff68-7e63-40b2-882b-8b2aed13e982-istiod-ca-cert\") pod \"0c93ff68-7e63-40b2-882b-8b2aed13e982\" (UID: \"0c93ff68-7e63-40b2-882b-8b2aed13e982\") "
Oct 31 12:59:48.181804 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:48.181552    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c93ff68-7e63-40b2-882b-8b2aed13e982-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c93ff68-7e63-40b2-882b-8b2aed13e982-istio-envoy\") pod \"0c93ff68-7e63-40b2-882b-8b2aed13e982\" (UID: \"0c93ff68-7e63-40b2-882b-8b2aed13e982\") "
Oct 31 12:59:48.181804 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:48.181578    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c93ff68-7e63-40b2-882b-8b2aed13e982-kube-api-access-82qzw\" (UniqueName: \"kubernetes.io/projected/0c93ff68-7e63-40b2-882b-8b2aed13e982-kube-api-access-82qzw\") pod \"0c93ff68-7e63-40b2-882b-8b2aed13e982\" (UID: \"0c93ff68-7e63-40b2-882b-8b2aed13e982\") "
Oct 31 12:59:48.181804 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:48.181604    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c93ff68-7e63-40b2-882b-8b2aed13e982-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c93ff68-7e63-40b2-882b-8b2aed13e982-workload-certs\") pod \"0c93ff68-7e63-40b2-882b-8b2aed13e982\" (UID: \"0c93ff68-7e63-40b2-882b-8b2aed13e982\") "
Oct 31 12:59:48.181804 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:48.181640    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/61ba0787-c638-4376-86d9-4c1c90dd8b6d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61ba0787-c638-4376-86d9-4c1c90dd8b6d-maas-api-xks-ca\") pod \"61ba0787-c638-4376-86d9-4c1c90dd8b6d\" (UID: \"61ba0787-c638-4376-86d9-4c1c90dd8b6d\") "
Oct 31 12:59:48.182610 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:48.181970    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0c93ff68-7e63-40b2-882b-8b2aed13e982-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c93ff68-7e63-40b2-882b-8b2aed13e982-istio-podinfo\") pod \"0c93ff68-7e63-40b2-882b-8b2aed13e982\" (UID: \"0c93ff68-7e63-40b2-882b-8b2aed13e982\") "
Oct 31 12:59:48.183336 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:48.183268    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c93ff68-7e63-40b2-882b-8b2aed13e982-workload-certs" pod "0c93ff68-7e63-40b2-882b-8b2aed13e982" (UID: "0c93ff68-7e63-40b2-882b-8b2aed13e982"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:59:48.183336 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:48.183292    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c93ff68-7e63-40b2-882b-8b2aed13e982-istio-data" pod "0c93ff68-7e63-40b2-882b-8b2aed13e982" (UID: "0c93ff68-7e63-40b2-882b-8b2aed13e982"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:59:48.183772 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:48.183582    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c93ff68-7e63-40b2-882b-8b2aed13e982-workload-socket" pod "0c93ff68-7e63-40b2-882b-8b2aed13e982" (UID: "0c93ff68-7e63-40b2-882b-8b2aed13e982"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:59:48.183772 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:48.183618    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c93ff68-7e63-40b2-882b-8b2aed13e982-istiod-ca-cert" pod "0c93ff68-7e63-40b2-882b-8b2aed13e982" (UID: "0c93ff68-7e63-40b2-882b-8b2aed13e982"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:59:48.183772 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:48.183690    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61ba0787-c638-4376-86d9-4c1c90dd8b6d-maas-api-openshift-ca" pod "61ba0787-c638-4376-86d9-4c1c90dd8b6d" (UID: "61ba0787-c638-4376-86d9-4c1c90dd8b6d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:59:48.184076 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:48.183860    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c93ff68-7e63-40b2-882b-8b2aed13e982-credential-socket" pod "0c93ff68-7e63-40b2-882b-8b2aed13e982" (UID: "0c93ff68-7e63-40b2-882b-8b2aed13e982"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:59:48.184370 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:48.184342    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0c93ff68-7e63-40b2-882b-8b2aed13e982-istio-podinfo" pod "0c93ff68-7e63-40b2-882b-8b2aed13e982" (UID: "0c93ff68-7e63-40b2-882b-8b2aed13e982"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 12:59:48.186111 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:48.186086    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61ba0787-c638-4376-86d9-4c1c90dd8b6d-maas-api-xks-ca" pod "61ba0787-c638-4376-86d9-4c1c90dd8b6d" (UID: "61ba0787-c638-4376-86d9-4c1c90dd8b6d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:59:48.186707 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:48.186663    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c93ff68-7e63-40b2-882b-8b2aed13e982-istio-token" pod "0c93ff68-7e63-40b2-882b-8b2aed13e982" (UID: "0c93ff68-7e63-40b2-882b-8b2aed13e982"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:59:48.186807 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:48.186775    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c93ff68-7e63-40b2-882b-8b2aed13e982-kube-api-access-82qzw" pod "0c93ff68-7e63-40b2-882b-8b2aed13e982" (UID: "0c93ff68-7e63-40b2-882b-8b2aed13e982"). InnerVolumeSpecName "kube-api-access-82qzw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:59:48.187128 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:48.187045    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61ba0787-c638-4376-86d9-4c1c90dd8b6d-kube-api-access-xkmq6" pod "61ba0787-c638-4376-86d9-4c1c90dd8b6d" (UID: "61ba0787-c638-4376-86d9-4c1c90dd8b6d"). InnerVolumeSpecName "kube-api-access-xkmq6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:59:48.188017 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:48.187986    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c93ff68-7e63-40b2-882b-8b2aed13e982-istio-envoy" pod "0c93ff68-7e63-40b2-882b-8b2aed13e982" (UID: "0c93ff68-7e63-40b2-882b-8b2aed13e982"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:59:48.283984 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:48.283946    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61ba0787-c638-4376-86d9-4c1c90dd8b6d-maas-api-openshift-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:48.283984 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:48.283978    5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c93ff68-7e63-40b2-882b-8b2aed13e982-credential-socket\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:48.284187 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:48.283994    5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c93ff68-7e63-40b2-882b-8b2aed13e982-istio-token\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:48.284187 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:48.284004    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkmq6\" (UniqueName: \"kubernetes.io/projected/61ba0787-c638-4376-86d9-4c1c90dd8b6d-kube-api-access-xkmq6\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:48.284187 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:48.284014    5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c93ff68-7e63-40b2-882b-8b2aed13e982-workload-socket\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:48.284187 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:48.284023    5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c93ff68-7e63-40b2-882b-8b2aed13e982-istio-data\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:48.284187 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:48.284032    5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c93ff68-7e63-40b2-882b-8b2aed13e982-istiod-ca-cert\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:48.284187 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:48.284040    5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c93ff68-7e63-40b2-882b-8b2aed13e982-istio-envoy\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:48.284187 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:48.284049    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-82qzw\" (UniqueName: \"kubernetes.io/projected/0c93ff68-7e63-40b2-882b-8b2aed13e982-kube-api-access-82qzw\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:48.284187 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:48.284060    5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c93ff68-7e63-40b2-882b-8b2aed13e982-workload-certs\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:48.284187 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:48.284072    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61ba0787-c638-4376-86d9-4c1c90dd8b6d-maas-api-xks-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:48.284187 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:48.284082    5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c93ff68-7e63-40b2-882b-8b2aed13e982-istio-podinfo\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:48.894323 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:48.894285    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b95e1a3-82c69"
Oct 31 12:59:48.894824 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:48.894321    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b95e1a3-82c69" event={"ID":"61ba0787-c638-4376-86d9-4c1c90dd8b6d","Type":"ContainerDied","Data":"fed1ff44d2f513a98cc472eed73026bd46589d7dd615c1d36b19412d22bba7df"}
Oct 31 12:59:48.894824 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:48.894378    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fed1ff44d2f513a98cc472eed73026bd46589d7dd615c1d36b19412d22bba7df"
Oct 31 12:59:48.895933 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:48.895908    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-fd988c-openshift-default-96bd4cd5c-g4dgc" event={"ID":"0c93ff68-7e63-40b2-882b-8b2aed13e982","Type":"ContainerDied","Data":"a59d78aff00f2d7375c284ad9b63491a5e74ef8cdfb2b1834d9b3292f91a5dbb"}
Oct 31 12:59:48.896041 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:48.895961    5060 scope.go:122] "RemoveContainer" containerID="e162b85c3c8d5b2f01ce21d6b80bffca81e78ad33c6852cd87ec34686222ad6d"
Oct 31 12:59:48.896097 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:48.896042    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-fd988c-openshift-default-96bd4cd5c-g4dgc"
Oct 31 12:59:48.928864 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:48.928831    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-fd988c-openshift-default-96bd4cd5c-g4dgc"]
Oct 31 12:59:48.937436 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:48.937410    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-fd988c-openshift-default-96bd4cd5c-g4dgc"]
Oct 31 12:59:49.049986 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:49.049476    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c93ff68-7e63-40b2-882b-8b2aed13e982" path="/var/lib/kubelet/pods/0c93ff68-7e63-40b2-882b-8b2aed13e982/volumes"
Oct 31 12:59:51.999145 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:51.999077    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-2-w5-b25d3b-kserve-76f798d59f-vvzhg" podUID="394ed76c-84ac-4612-9acb-a7e36cb68d89" containerName="main" containerID="cri-o://1270a228fe9e90c1c467bed01ee44027f4ab4b23919044785878fe418213c5bd" gracePeriod=45
Oct 31 12:59:52.325154 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:52.325119    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-v2sv5"
Oct 31 12:59:52.362053 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:52.361972    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-2-w5-b25d3b-kserve-76f798d59f-vvzhg"
Oct 31 12:59:52.443908 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:52.443740    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/394ed76c-84ac-4612-9acb-a7e36cb68d89-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/394ed76c-84ac-4612-9acb-a7e36cb68d89-tmp-dir\") pod \"394ed76c-84ac-4612-9acb-a7e36cb68d89\" (UID: \"394ed76c-84ac-4612-9acb-a7e36cb68d89\") "
Oct 31 12:59:52.443908 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:52.443807    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/394ed76c-84ac-4612-9acb-a7e36cb68d89-tls-certs\" (UniqueName: \"kubernetes.io/secret/394ed76c-84ac-4612-9acb-a7e36cb68d89-tls-certs\") pod \"394ed76c-84ac-4612-9acb-a7e36cb68d89\" (UID: \"394ed76c-84ac-4612-9acb-a7e36cb68d89\") "
Oct 31 12:59:52.443908 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:52.443858    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1e33b790-15e5-4754-929f-c0a7330b9361-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e33b790-15e5-4754-929f-c0a7330b9361-istio-podinfo\") pod \"1e33b790-15e5-4754-929f-c0a7330b9361\" (UID: \"1e33b790-15e5-4754-929f-c0a7330b9361\") "
Oct 31 12:59:52.443908 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:52.443897    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/394ed76c-84ac-4612-9acb-a7e36cb68d89-kube-api-access-6qhrx\" (UniqueName: \"kubernetes.io/projected/394ed76c-84ac-4612-9acb-a7e36cb68d89-kube-api-access-6qhrx\") pod \"394ed76c-84ac-4612-9acb-a7e36cb68d89\" (UID: \"394ed76c-84ac-4612-9acb-a7e36cb68d89\") "
Oct 31 12:59:52.444306 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:52.444012    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e33b790-15e5-4754-929f-c0a7330b9361-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e33b790-15e5-4754-929f-c0a7330b9361-istio-envoy\") pod \"1e33b790-15e5-4754-929f-c0a7330b9361\" (UID: \"1e33b790-15e5-4754-929f-c0a7330b9361\") "
Oct 31 12:59:52.444306 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:52.444143    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e33b790-15e5-4754-929f-c0a7330b9361-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e33b790-15e5-4754-929f-c0a7330b9361-istiod-ca-cert\") pod \"1e33b790-15e5-4754-929f-c0a7330b9361\" (UID: \"1e33b790-15e5-4754-929f-c0a7330b9361\") "
Oct 31 12:59:52.444306 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:52.444223    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e33b790-15e5-4754-929f-c0a7330b9361-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e33b790-15e5-4754-929f-c0a7330b9361-workload-socket\") pod \"1e33b790-15e5-4754-929f-c0a7330b9361\" (UID: \"1e33b790-15e5-4754-929f-c0a7330b9361\") "
Oct 31 12:59:52.444507 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:52.444304    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/394ed76c-84ac-4612-9acb-a7e36cb68d89-home\" (UniqueName: \"kubernetes.io/empty-dir/394ed76c-84ac-4612-9acb-a7e36cb68d89-home\") pod \"394ed76c-84ac-4612-9acb-a7e36cb68d89\" (UID: \"394ed76c-84ac-4612-9acb-a7e36cb68d89\") "
Oct 31 12:59:52.444507 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:52.444390    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/394ed76c-84ac-4612-9acb-a7e36cb68d89-dshm\" (UniqueName: \"kubernetes.io/empty-dir/394ed76c-84ac-4612-9acb-a7e36cb68d89-dshm\") pod \"394ed76c-84ac-4612-9acb-a7e36cb68d89\" (UID: \"394ed76c-84ac-4612-9acb-a7e36cb68d89\") "
Oct 31 12:59:52.444507 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:52.444449    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e33b790-15e5-4754-929f-c0a7330b9361-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e33b790-15e5-4754-929f-c0a7330b9361-workload-certs\") pod \"1e33b790-15e5-4754-929f-c0a7330b9361\" (UID: \"1e33b790-15e5-4754-929f-c0a7330b9361\") "
Oct 31 12:59:52.444684 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:52.444548    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e33b790-15e5-4754-929f-c0a7330b9361-kube-api-access-f9lpm\" (UniqueName: \"kubernetes.io/projected/1e33b790-15e5-4754-929f-c0a7330b9361-kube-api-access-f9lpm\") pod \"1e33b790-15e5-4754-929f-c0a7330b9361\" (UID: \"1e33b790-15e5-4754-929f-c0a7330b9361\") "
Oct 31 12:59:52.444684 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:52.444587    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e33b790-15e5-4754-929f-c0a7330b9361-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e33b790-15e5-4754-929f-c0a7330b9361-istio-data\") pod \"1e33b790-15e5-4754-929f-c0a7330b9361\" (UID: \"1e33b790-15e5-4754-929f-c0a7330b9361\") "
Oct 31 12:59:52.444684 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:52.444630    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e33b790-15e5-4754-929f-c0a7330b9361-istio-token\" (UniqueName: \"kubernetes.io/projected/1e33b790-15e5-4754-929f-c0a7330b9361-istio-token\") pod \"1e33b790-15e5-4754-929f-c0a7330b9361\" (UID: \"1e33b790-15e5-4754-929f-c0a7330b9361\") "
Oct 31 12:59:52.444684 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:52.444659    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e33b790-15e5-4754-929f-c0a7330b9361-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e33b790-15e5-4754-929f-c0a7330b9361-credential-socket\") pod \"1e33b790-15e5-4754-929f-c0a7330b9361\" (UID: \"1e33b790-15e5-4754-929f-c0a7330b9361\") "
Oct 31 12:59:52.444903 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:52.444688    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/394ed76c-84ac-4612-9acb-a7e36cb68d89-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/394ed76c-84ac-4612-9acb-a7e36cb68d89-model-cache\") pod \"394ed76c-84ac-4612-9acb-a7e36cb68d89\" (UID: \"394ed76c-84ac-4612-9acb-a7e36cb68d89\") "
Oct 31 12:59:52.446956 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:52.446643    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e33b790-15e5-4754-929f-c0a7330b9361-workload-socket" pod "1e33b790-15e5-4754-929f-c0a7330b9361" (UID: "1e33b790-15e5-4754-929f-c0a7330b9361"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:59:52.446956 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:52.446781    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e33b790-15e5-4754-929f-c0a7330b9361-istiod-ca-cert" pod "1e33b790-15e5-4754-929f-c0a7330b9361" (UID: "1e33b790-15e5-4754-929f-c0a7330b9361"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:59:52.448838 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:52.447613    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e33b790-15e5-4754-929f-c0a7330b9361-istio-data" pod "1e33b790-15e5-4754-929f-c0a7330b9361" (UID: "1e33b790-15e5-4754-929f-c0a7330b9361"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:59:52.448838 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:52.447945    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/394ed76c-84ac-4612-9acb-a7e36cb68d89-home" pod "394ed76c-84ac-4612-9acb-a7e36cb68d89" (UID: "394ed76c-84ac-4612-9acb-a7e36cb68d89"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:59:52.448838 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:52.448303    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e33b790-15e5-4754-929f-c0a7330b9361-workload-certs" pod "1e33b790-15e5-4754-929f-c0a7330b9361" (UID: "1e33b790-15e5-4754-929f-c0a7330b9361"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:59:52.448838 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:52.448367    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e33b790-15e5-4754-929f-c0a7330b9361-credential-socket" pod "1e33b790-15e5-4754-929f-c0a7330b9361" (UID: "1e33b790-15e5-4754-929f-c0a7330b9361"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:59:52.454912 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:52.454881    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/394ed76c-84ac-4612-9acb-a7e36cb68d89-tmp-dir" pod "394ed76c-84ac-4612-9acb-a7e36cb68d89" (UID: "394ed76c-84ac-4612-9acb-a7e36cb68d89"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:59:52.455082 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:52.455059    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/394ed76c-84ac-4612-9acb-a7e36cb68d89-model-cache" pod "394ed76c-84ac-4612-9acb-a7e36cb68d89" (UID: "394ed76c-84ac-4612-9acb-a7e36cb68d89"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:59:52.455783 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:52.455743    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/394ed76c-84ac-4612-9acb-a7e36cb68d89-tls-certs" pod "394ed76c-84ac-4612-9acb-a7e36cb68d89" (UID: "394ed76c-84ac-4612-9acb-a7e36cb68d89"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:59:52.456449 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:52.456414    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/394ed76c-84ac-4612-9acb-a7e36cb68d89-kube-api-access-6qhrx" pod "394ed76c-84ac-4612-9acb-a7e36cb68d89" (UID: "394ed76c-84ac-4612-9acb-a7e36cb68d89"). InnerVolumeSpecName "kube-api-access-6qhrx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:59:52.456749 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:52.456693    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e33b790-15e5-4754-929f-c0a7330b9361-kube-api-access-f9lpm" pod "1e33b790-15e5-4754-929f-c0a7330b9361" (UID: "1e33b790-15e5-4754-929f-c0a7330b9361"). InnerVolumeSpecName "kube-api-access-f9lpm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:59:52.456838 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:52.456786    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/394ed76c-84ac-4612-9acb-a7e36cb68d89-dshm" pod "394ed76c-84ac-4612-9acb-a7e36cb68d89" (UID: "394ed76c-84ac-4612-9acb-a7e36cb68d89"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:59:52.457014 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:52.456987    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e33b790-15e5-4754-929f-c0a7330b9361-istio-envoy" pod "1e33b790-15e5-4754-929f-c0a7330b9361" (UID: "1e33b790-15e5-4754-929f-c0a7330b9361"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:59:52.459888 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:52.459861    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1e33b790-15e5-4754-929f-c0a7330b9361-istio-podinfo" pod "1e33b790-15e5-4754-929f-c0a7330b9361" (UID: "1e33b790-15e5-4754-929f-c0a7330b9361"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 12:59:52.460053 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:52.460021    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e33b790-15e5-4754-929f-c0a7330b9361-istio-token" pod "1e33b790-15e5-4754-929f-c0a7330b9361" (UID: "1e33b790-15e5-4754-929f-c0a7330b9361"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:59:52.546015 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:52.545979    5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e33b790-15e5-4754-929f-c0a7330b9361-workload-socket\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:52.546015 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:52.546013    5060 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/394ed76c-84ac-4612-9acb-a7e36cb68d89-home\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:52.546326 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:52.546025    5060 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/394ed76c-84ac-4612-9acb-a7e36cb68d89-dshm\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:52.546326 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:52.546037    5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e33b790-15e5-4754-929f-c0a7330b9361-workload-certs\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:52.546326 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:52.546049    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9lpm\" (UniqueName: \"kubernetes.io/projected/1e33b790-15e5-4754-929f-c0a7330b9361-kube-api-access-f9lpm\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:52.546326 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:52.546061    5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e33b790-15e5-4754-929f-c0a7330b9361-istio-data\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:52.546326 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:52.546072    5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e33b790-15e5-4754-929f-c0a7330b9361-istio-token\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:52.546326 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:52.546086    5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e33b790-15e5-4754-929f-c0a7330b9361-credential-socket\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:52.546326 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:52.546098    5060 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/394ed76c-84ac-4612-9acb-a7e36cb68d89-model-cache\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:52.546326 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:52.546109    5060 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/394ed76c-84ac-4612-9acb-a7e36cb68d89-tmp-dir\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:52.546326 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:52.546123    5060 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/394ed76c-84ac-4612-9acb-a7e36cb68d89-tls-certs\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:52.546326 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:52.546136    5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e33b790-15e5-4754-929f-c0a7330b9361-istio-podinfo\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:52.546326 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:52.546150    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6qhrx\" (UniqueName: \"kubernetes.io/projected/394ed76c-84ac-4612-9acb-a7e36cb68d89-kube-api-access-6qhrx\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:52.546326 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:52.546162    5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e33b790-15e5-4754-929f-c0a7330b9361-istio-envoy\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:52.546326 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:52.546173    5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e33b790-15e5-4754-929f-c0a7330b9361-istiod-ca-cert\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:52.901790 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:52.901696    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-b25d3b-66f84f5d8f-6w4n2" podUID="85cd1407-54f9-493a-b8fd-9e638a6615a5" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 12:59:52.936164 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:52.936125    5060 generic.go:356] "Generic (PLEG): container finished" podID="394ed76c-84ac-4612-9acb-a7e36cb68d89" containerID="1270a228fe9e90c1c467bed01ee44027f4ab4b23919044785878fe418213c5bd" exitCode=0
Oct 31 12:59:52.936363 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:52.936183    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-2-w5-b25d3b-kserve-76f798d59f-vvzhg"
Oct 31 12:59:52.936363 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:52.936209    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-2-w5-b25d3b-kserve-76f798d59f-vvzhg" event={"ID":"394ed76c-84ac-4612-9acb-a7e36cb68d89","Type":"ContainerDied","Data":"1270a228fe9e90c1c467bed01ee44027f4ab4b23919044785878fe418213c5bd"}
Oct 31 12:59:52.936363 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:52.936247    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-2-w5-b25d3b-kserve-76f798d59f-vvzhg" event={"ID":"394ed76c-84ac-4612-9acb-a7e36cb68d89","Type":"ContainerDied","Data":"6cf96c6ede1f75a8b30adcf4b90c62aae92014994fb1bbd968771a672caba75a"}
Oct 31 12:59:52.936363 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:52.936272    5060 scope.go:122] "RemoveContainer" containerID="1270a228fe9e90c1c467bed01ee44027f4ab4b23919044785878fe418213c5bd"
Oct 31 12:59:52.937866 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:52.937837    5060 generic.go:356] "Generic (PLEG): container finished" podID="1e33b790-15e5-4754-929f-c0a7330b9361" containerID="61d1f66705dbeb17fd9cbd694f56a41316654ebc7e876f422e77fca134c16d3f" exitCode=0
Oct 31 12:59:52.938002 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:52.937864    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-v2sv5" event={"ID":"1e33b790-15e5-4754-929f-c0a7330b9361","Type":"ContainerDied","Data":"61d1f66705dbeb17fd9cbd694f56a41316654ebc7e876f422e77fca134c16d3f"}
Oct 31 12:59:52.938002 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:52.937885    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-v2sv5"
Oct 31 12:59:52.938002 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:52.937909    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-v2sv5" event={"ID":"1e33b790-15e5-4754-929f-c0a7330b9361","Type":"ContainerDied","Data":"fa7d629062693e7f49fbb860fb02ae52298084218816068bdc9c8973af8f3fa6"}
Oct 31 12:59:52.949842 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:52.949818    5060 scope.go:122] "RemoveContainer" containerID="1270a228fe9e90c1c467bed01ee44027f4ab4b23919044785878fe418213c5bd"
Oct 31 12:59:52.950158 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:59:52.950139    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1270a228fe9e90c1c467bed01ee44027f4ab4b23919044785878fe418213c5bd\": container with ID starting with 1270a228fe9e90c1c467bed01ee44027f4ab4b23919044785878fe418213c5bd not found: ID does not exist" containerID="1270a228fe9e90c1c467bed01ee44027f4ab4b23919044785878fe418213c5bd"
Oct 31 12:59:52.950206 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:52.950165    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1270a228fe9e90c1c467bed01ee44027f4ab4b23919044785878fe418213c5bd"} err="failed to get container status \"1270a228fe9e90c1c467bed01ee44027f4ab4b23919044785878fe418213c5bd\": rpc error: code = NotFound desc = could not find container \"1270a228fe9e90c1c467bed01ee44027f4ab4b23919044785878fe418213c5bd\": container with ID starting with 1270a228fe9e90c1c467bed01ee44027f4ab4b23919044785878fe418213c5bd not found: ID does not exist"
Oct 31 12:59:52.950206 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:52.950182    5060 scope.go:122] "RemoveContainer" containerID="61d1f66705dbeb17fd9cbd694f56a41316654ebc7e876f422e77fca134c16d3f"
Oct 31 12:59:52.963263 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:52.963239    5060 scope.go:122] "RemoveContainer" containerID="61d1f66705dbeb17fd9cbd694f56a41316654ebc7e876f422e77fca134c16d3f"
Oct 31 12:59:52.963668 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:59:52.963634    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61d1f66705dbeb17fd9cbd694f56a41316654ebc7e876f422e77fca134c16d3f\": container with ID starting with 61d1f66705dbeb17fd9cbd694f56a41316654ebc7e876f422e77fca134c16d3f not found: ID does not exist" containerID="61d1f66705dbeb17fd9cbd694f56a41316654ebc7e876f422e77fca134c16d3f"
Oct 31 12:59:52.963862 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:52.963698    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61d1f66705dbeb17fd9cbd694f56a41316654ebc7e876f422e77fca134c16d3f"} err="failed to get container status \"61d1f66705dbeb17fd9cbd694f56a41316654ebc7e876f422e77fca134c16d3f\": rpc error: code = NotFound desc = could not find container \"61d1f66705dbeb17fd9cbd694f56a41316654ebc7e876f422e77fca134c16d3f\": container with ID starting with 61d1f66705dbeb17fd9cbd694f56a41316654ebc7e876f422e77fca134c16d3f not found: ID does not exist"
Oct 31 12:59:52.967830 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:52.967695    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-b25d3b/distinct-2-w5-b25d3b-kserve-76f798d59f-vvzhg"]
Oct 31 12:59:52.972448 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:52.972419    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-b25d3b/distinct-2-w5-b25d3b-kserve-76f798d59f-vvzhg"]
Oct 31 12:59:52.984006 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:52.983979    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-v2sv5"]
Oct 31 12:59:52.989146 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:52.989122    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-v2sv5"]
Oct 31 12:59:53.048236 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:53.048201    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e33b790-15e5-4754-929f-c0a7330b9361" path="/var/lib/kubelet/pods/1e33b790-15e5-4754-929f-c0a7330b9361/volumes"
Oct 31 12:59:53.048723 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:53.048708    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="394ed76c-84ac-4612-9acb-a7e36cb68d89" path="/var/lib/kubelet/pods/394ed76c-84ac-4612-9acb-a7e36cb68d89/volumes"
Oct 31 12:59:54.441850 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:54.441813    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-8faad6-s64xf"]
Oct 31 12:59:54.442367 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:54.442353    5060 cpu_manager.go:414] "containerMap: removing container" podUID="0c93ff68-7e63-40b2-882b-8b2aed13e982" containerName="istio-proxy"
Oct 31 12:59:54.442426 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:54.442367    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c93ff68-7e63-40b2-882b-8b2aed13e982" containerName="istio-proxy"
Oct 31 12:59:54.442426 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:54.442386    5060 cpu_manager.go:414] "containerMap: removing container" podUID="394ed76c-84ac-4612-9acb-a7e36cb68d89" containerName="main"
Oct 31 12:59:54.442426 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:54.442392    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="394ed76c-84ac-4612-9acb-a7e36cb68d89" containerName="main"
Oct 31 12:59:54.442426 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:54.442403    5060 cpu_manager.go:414] "containerMap: removing container" podUID="0c93ff68-7e63-40b2-882b-8b2aed13e982" containerName="istio-proxy"
Oct 31 12:59:54.442426 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:54.442410    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c93ff68-7e63-40b2-882b-8b2aed13e982" containerName="istio-proxy"
Oct 31 12:59:54.442426 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:54.442420    5060 cpu_manager.go:414] "containerMap: removing container" podUID="1e33b790-15e5-4754-929f-c0a7330b9361" containerName="istio-proxy"
Oct 31 12:59:54.442426 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:54.442425    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e33b790-15e5-4754-929f-c0a7330b9361" containerName="istio-proxy"
Oct 31 12:59:54.442691 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:54.442441    5060 cpu_manager.go:414] "containerMap: removing container" podUID="61ba0787-c638-4376-86d9-4c1c90dd8b6d" containerName="revoke-keys"
Oct 31 12:59:54.442691 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:54.442447    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61ba0787-c638-4376-86d9-4c1c90dd8b6d" containerName="revoke-keys"
Oct 31 12:59:54.442691 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:54.442455    5060 cpu_manager.go:414] "containerMap: removing container" podUID="b5d32cf4-f969-45b2-a94c-273809cbf9ed" containerName="revoke-keys"
Oct 31 12:59:54.442691 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:54.442460    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5d32cf4-f969-45b2-a94c-273809cbf9ed" containerName="revoke-keys"
Oct 31 12:59:54.442691 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:54.442563    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c93ff68-7e63-40b2-882b-8b2aed13e982" containerName="istio-proxy"
Oct 31 12:59:54.442691 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:54.442575    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="394ed76c-84ac-4612-9acb-a7e36cb68d89" containerName="main"
Oct 31 12:59:54.442691 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:54.442582    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c93ff68-7e63-40b2-882b-8b2aed13e982" containerName="istio-proxy"
Oct 31 12:59:54.442691 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:54.442589    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e33b790-15e5-4754-929f-c0a7330b9361" containerName="istio-proxy"
Oct 31 12:59:54.442691 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:54.442596    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="61ba0787-c638-4376-86d9-4c1c90dd8b6d" containerName="revoke-keys"
Oct 31 12:59:54.442691 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:54.442603    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5d32cf4-f969-45b2-a94c-273809cbf9ed" containerName="revoke-keys"
Oct 31 12:59:54.447520 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:54.447474    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-8faad6-s64xf"
Oct 31 12:59:54.452019 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:54.451988    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 12:59:54.452239 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:54.452011    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 12:59:54.452367 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:54.452059    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 12:59:54.452460 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:54.452108    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7wsmb\""
Oct 31 12:59:54.456241 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:54.456213    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-8faad6-s64xf"]
Oct 31 12:59:54.577144 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:54.577110    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78c3c8ea-96af-4239-b430-75173a1e3fba-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-8faad6-s64xf\" (UID: \"78c3c8ea-96af-4239-b430-75173a1e3fba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-8faad6-s64xf"
Oct 31 12:59:54.577316 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:54.577163    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sx2kz\" (UniqueName: \"kubernetes.io/projected/78c3c8ea-96af-4239-b430-75173a1e3fba-kube-api-access-sx2kz\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-8faad6-s64xf\" (UID: \"78c3c8ea-96af-4239-b430-75173a1e3fba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-8faad6-s64xf"
Oct 31 12:59:54.577398 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:54.577370    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78c3c8ea-96af-4239-b430-75173a1e3fba-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-8faad6-s64xf\" (UID: \"78c3c8ea-96af-4239-b430-75173a1e3fba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-8faad6-s64xf"
Oct 31 12:59:54.678612 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:54.678575    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sx2kz\" (UniqueName: \"kubernetes.io/projected/78c3c8ea-96af-4239-b430-75173a1e3fba-kube-api-access-sx2kz\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-8faad6-s64xf\" (UID: \"78c3c8ea-96af-4239-b430-75173a1e3fba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-8faad6-s64xf"
Oct 31 12:59:54.678867 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:54.678718    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78c3c8ea-96af-4239-b430-75173a1e3fba-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-8faad6-s64xf\" (UID: \"78c3c8ea-96af-4239-b430-75173a1e3fba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-8faad6-s64xf"
Oct 31 12:59:54.678867 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:54.678857    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78c3c8ea-96af-4239-b430-75173a1e3fba-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-8faad6-s64xf\" (UID: \"78c3c8ea-96af-4239-b430-75173a1e3fba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-8faad6-s64xf"
Oct 31 12:59:54.679629 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:54.679597    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78c3c8ea-96af-4239-b430-75173a1e3fba-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-8faad6-s64xf\" (UID: \"78c3c8ea-96af-4239-b430-75173a1e3fba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-8faad6-s64xf"
Oct 31 12:59:54.681359 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:54.681337    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78c3c8ea-96af-4239-b430-75173a1e3fba-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-8faad6-s64xf\" (UID: \"78c3c8ea-96af-4239-b430-75173a1e3fba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-8faad6-s64xf"
Oct 31 12:59:54.691921 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:54.691867    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sx2kz\" (UniqueName: \"kubernetes.io/projected/78c3c8ea-96af-4239-b430-75173a1e3fba-kube-api-access-sx2kz\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-8faad6-s64xf\" (UID: \"78c3c8ea-96af-4239-b430-75173a1e3fba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-8faad6-s64xf"
Oct 31 12:59:54.767594 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:54.767550    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-8faad6-s64xf"
Oct 31 12:59:55.080091 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:55.080065    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-8faad6-s64xf"]
Oct 31 12:59:55.082331 ip-10-0-130-190 kubenswrapper[5060]: W1031 12:59:55.082304    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78c3c8ea_96af_4239_b430_75173a1e3fba.slice/crio-5fac44039dd4a60fd4c9771710703d94182de2be844239e2f27df106fb92c4ac WatchSource:0}: Error finding container 5fac44039dd4a60fd4c9771710703d94182de2be844239e2f27df106fb92c4ac: Status 404 returned error can't find the container with id 5fac44039dd4a60fd4c9771710703d94182de2be844239e2f27df106fb92c4ac
Oct 31 12:59:55.974849 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:55.974820    5060 generic.go:356] "Generic (PLEG): container finished" podID="78c3c8ea-96af-4239-b430-75173a1e3fba" containerID="9beb8a228a0b9d09f03bd6061e406c42cfc070ae42d4225e719e6ad4b88b85e7" exitCode=0
Oct 31 12:59:55.975243 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:55.974902    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-8faad6-s64xf" event={"ID":"78c3c8ea-96af-4239-b430-75173a1e3fba","Type":"ContainerDied","Data":"9beb8a228a0b9d09f03bd6061e406c42cfc070ae42d4225e719e6ad4b88b85e7"}
Oct 31 12:59:55.975243 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:55.974951    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-8faad6-s64xf" event={"ID":"78c3c8ea-96af-4239-b430-75173a1e3fba","Type":"ContainerStarted","Data":"5fac44039dd4a60fd4c9771710703d94182de2be844239e2f27df106fb92c4ac"}
Oct 31 12:59:56.207186 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:56.207096    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-9b17b9-t2n9k"]
Oct 31 12:59:56.214343 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:56.214309    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-9b17b9-t2n9k"]
Oct 31 12:59:57.050985 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:57.050952    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5d32cf4-f969-45b2-a94c-273809cbf9ed" path="/var/lib/kubelet/pods/b5d32cf4-f969-45b2-a94c-273809cbf9ed/volumes"
Oct 31 12:59:57.161044 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:57.161012    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-8faad6-s64xf"
Oct 31 12:59:57.310882 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:57.310853    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78c3c8ea-96af-4239-b430-75173a1e3fba-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78c3c8ea-96af-4239-b430-75173a1e3fba-maas-api-openshift-ca\") pod \"78c3c8ea-96af-4239-b430-75173a1e3fba\" (UID: \"78c3c8ea-96af-4239-b430-75173a1e3fba\") "
Oct 31 12:59:57.311073 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:57.310902    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78c3c8ea-96af-4239-b430-75173a1e3fba-kube-api-access-sx2kz\" (UniqueName: \"kubernetes.io/projected/78c3c8ea-96af-4239-b430-75173a1e3fba-kube-api-access-sx2kz\") pod \"78c3c8ea-96af-4239-b430-75173a1e3fba\" (UID: \"78c3c8ea-96af-4239-b430-75173a1e3fba\") "
Oct 31 12:59:57.311073 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:57.310986    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/78c3c8ea-96af-4239-b430-75173a1e3fba-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78c3c8ea-96af-4239-b430-75173a1e3fba-maas-api-xks-ca\") pod \"78c3c8ea-96af-4239-b430-75173a1e3fba\" (UID: \"78c3c8ea-96af-4239-b430-75173a1e3fba\") "
Oct 31 12:59:57.311296 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:57.311262    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78c3c8ea-96af-4239-b430-75173a1e3fba-maas-api-openshift-ca" pod "78c3c8ea-96af-4239-b430-75173a1e3fba" (UID: "78c3c8ea-96af-4239-b430-75173a1e3fba"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:59:57.311538 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:57.311433    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78c3c8ea-96af-4239-b430-75173a1e3fba-maas-api-openshift-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:57.313182 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:57.313160    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78c3c8ea-96af-4239-b430-75173a1e3fba-kube-api-access-sx2kz" pod "78c3c8ea-96af-4239-b430-75173a1e3fba" (UID: "78c3c8ea-96af-4239-b430-75173a1e3fba"). InnerVolumeSpecName "kube-api-access-sx2kz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:59:57.313285 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:57.313202    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78c3c8ea-96af-4239-b430-75173a1e3fba-maas-api-xks-ca" pod "78c3c8ea-96af-4239-b430-75173a1e3fba" (UID: "78c3c8ea-96af-4239-b430-75173a1e3fba"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:59:57.412313 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:57.412282    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78c3c8ea-96af-4239-b430-75173a1e3fba-maas-api-xks-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:57.412313 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:57.412311    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sx2kz\" (UniqueName: \"kubernetes.io/projected/78c3c8ea-96af-4239-b430-75173a1e3fba-kube-api-access-sx2kz\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:57.997880 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:57.997842    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-8faad6-s64xf" event={"ID":"78c3c8ea-96af-4239-b430-75173a1e3fba","Type":"ContainerDied","Data":"5fac44039dd4a60fd4c9771710703d94182de2be844239e2f27df106fb92c4ac"}
Oct 31 12:59:57.997880 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:57.997876    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-8faad6-s64xf"
Oct 31 12:59:57.998096 ip-10-0-130-190 kubenswrapper[5060]: I1031 12:59:57.997899    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5fac44039dd4a60fd4c9771710703d94182de2be844239e2f27df106fb92c4ac"
Oct 31 12:59:59.922009 ip-10-0-130-190 kubenswrapper[5060]: E1031 12:59:59.921970    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7phb" containerName="extractor"
Oct 31 13:00:00.144082 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:00.144044    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-2d6201-29890860-8p4sf"]
Oct 31 13:00:00.144631 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:00.144614    5060 cpu_manager.go:414] "containerMap: removing container" podUID="78c3c8ea-96af-4239-b430-75173a1e3fba" containerName="revoke-keys"
Oct 31 13:00:00.144631 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:00.144630    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78c3c8ea-96af-4239-b430-75173a1e3fba" containerName="revoke-keys"
Oct 31 13:00:00.144743 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:00.144737    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="78c3c8ea-96af-4239-b430-75173a1e3fba" containerName="revoke-keys"
Oct 31 13:00:00.148708 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:00.148683    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-2d6201-29890860-8p4sf"
Oct 31 13:00:00.153677 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:00.153608    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 13:00:00.154190 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:00.154168    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c67de2-29890860-bpnn6"]
Oct 31 13:00:00.158932 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:00.158901    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c67de2-29890860-bpnn6"
Oct 31 13:00:00.159135 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:00.159114    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-8faad6-29890860-wv6gj"]
Oct 31 13:00:00.163735 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:00.163711    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-8faad6-29890860-wv6gj"
Oct 31 13:00:00.164444 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:00.164420    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-2d6201-29890860-8p4sf"]
Oct 31 13:00:00.176681 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:00.176556    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-8faad6-29890860-wv6gj"]
Oct 31 13:00:00.183586 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:00.183557    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c67de2-29890860-bpnn6"]
Oct 31 13:00:00.225680 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:00.225649    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7wsmb\""
Oct 31 13:00:00.245584 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:00.245544    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lmxt\" (UniqueName: \"kubernetes.io/projected/fd6cc1a2-16d5-4e08-af17-6dceac1af4a1-kube-api-access-9lmxt\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-8faad6-29890860-wv6gj\" (UID: \"fd6cc1a2-16d5-4e08-af17-6dceac1af4a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-8faad6-29890860-wv6gj"
Oct 31 13:00:00.245766 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:00.245679    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8wbj\" (UniqueName: \"kubernetes.io/projected/a1a17f30-378e-4b5c-81c4-a6f50503b909-kube-api-access-z8wbj\") pod \"maas-api-key-cleanup-e2e-worker-w4-c67de2-29890860-bpnn6\" (UID: \"a1a17f30-378e-4b5c-81c4-a6f50503b909\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c67de2-29890860-bpnn6"
Oct 31 13:00:00.245851 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:00.245777    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1a17f30-378e-4b5c-81c4-a6f50503b909-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-c67de2-29890860-bpnn6\" (UID: \"a1a17f30-378e-4b5c-81c4-a6f50503b909\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c67de2-29890860-bpnn6"
Oct 31 13:00:00.245851 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:00.245826    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd6cc1a2-16d5-4e08-af17-6dceac1af4a1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-8faad6-29890860-wv6gj\" (UID: \"fd6cc1a2-16d5-4e08-af17-6dceac1af4a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-8faad6-29890860-wv6gj"
Oct 31 13:00:00.245968 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:00.245881    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jvvj\" (UniqueName: \"kubernetes.io/projected/9a318cfb-1970-4ec2-8488-07de8420ba8e-kube-api-access-8jvvj\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-2d6201-29890860-8p4sf\" (UID: \"9a318cfb-1970-4ec2-8488-07de8420ba8e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-2d6201-29890860-8p4sf"
Oct 31 13:00:00.245968 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:00.245909    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a318cfb-1970-4ec2-8488-07de8420ba8e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-2d6201-29890860-8p4sf\" (UID: \"9a318cfb-1970-4ec2-8488-07de8420ba8e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-2d6201-29890860-8p4sf"
Oct 31 13:00:00.253637 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:00.253609    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 13:00:00.347716 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:00.347673    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1a17f30-378e-4b5c-81c4-a6f50503b909-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-c67de2-29890860-bpnn6\" (UID: \"a1a17f30-378e-4b5c-81c4-a6f50503b909\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c67de2-29890860-bpnn6"
Oct 31 13:00:00.347973 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:00.347740    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd6cc1a2-16d5-4e08-af17-6dceac1af4a1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-8faad6-29890860-wv6gj\" (UID: \"fd6cc1a2-16d5-4e08-af17-6dceac1af4a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-8faad6-29890860-wv6gj"
Oct 31 13:00:00.347973 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:00.347787    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jvvj\" (UniqueName: \"kubernetes.io/projected/9a318cfb-1970-4ec2-8488-07de8420ba8e-kube-api-access-8jvvj\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-2d6201-29890860-8p4sf\" (UID: \"9a318cfb-1970-4ec2-8488-07de8420ba8e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-2d6201-29890860-8p4sf"
Oct 31 13:00:00.347973 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:00.347827    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a318cfb-1970-4ec2-8488-07de8420ba8e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-2d6201-29890860-8p4sf\" (UID: \"9a318cfb-1970-4ec2-8488-07de8420ba8e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-2d6201-29890860-8p4sf"
Oct 31 13:00:00.347973 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:00.347872    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lmxt\" (UniqueName: \"kubernetes.io/projected/fd6cc1a2-16d5-4e08-af17-6dceac1af4a1-kube-api-access-9lmxt\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-8faad6-29890860-wv6gj\" (UID: \"fd6cc1a2-16d5-4e08-af17-6dceac1af4a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-8faad6-29890860-wv6gj"
Oct 31 13:00:00.348211 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:00.348007    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8wbj\" (UniqueName: \"kubernetes.io/projected/a1a17f30-378e-4b5c-81c4-a6f50503b909-kube-api-access-z8wbj\") pod \"maas-api-key-cleanup-e2e-worker-w4-c67de2-29890860-bpnn6\" (UID: \"a1a17f30-378e-4b5c-81c4-a6f50503b909\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c67de2-29890860-bpnn6"
Oct 31 13:00:00.349275 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:00.349244    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1a17f30-378e-4b5c-81c4-a6f50503b909-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-c67de2-29890860-bpnn6\" (UID: \"a1a17f30-378e-4b5c-81c4-a6f50503b909\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c67de2-29890860-bpnn6"
Oct 31 13:00:00.349401 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:00.349292    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a318cfb-1970-4ec2-8488-07de8420ba8e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-2d6201-29890860-8p4sf\" (UID: \"9a318cfb-1970-4ec2-8488-07de8420ba8e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-2d6201-29890860-8p4sf"
Oct 31 13:00:00.349401 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:00.349333    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd6cc1a2-16d5-4e08-af17-6dceac1af4a1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-8faad6-29890860-wv6gj\" (UID: \"fd6cc1a2-16d5-4e08-af17-6dceac1af4a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-8faad6-29890860-wv6gj"
Oct 31 13:00:00.363002 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:00.362969    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8wbj\" (UniqueName: \"kubernetes.io/projected/a1a17f30-378e-4b5c-81c4-a6f50503b909-kube-api-access-z8wbj\") pod \"maas-api-key-cleanup-e2e-worker-w4-c67de2-29890860-bpnn6\" (UID: \"a1a17f30-378e-4b5c-81c4-a6f50503b909\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c67de2-29890860-bpnn6"
Oct 31 13:00:00.363393 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:00.363369    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jvvj\" (UniqueName: \"kubernetes.io/projected/9a318cfb-1970-4ec2-8488-07de8420ba8e-kube-api-access-8jvvj\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-2d6201-29890860-8p4sf\" (UID: \"9a318cfb-1970-4ec2-8488-07de8420ba8e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-2d6201-29890860-8p4sf"
Oct 31 13:00:00.364331 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:00.364308    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lmxt\" (UniqueName: \"kubernetes.io/projected/fd6cc1a2-16d5-4e08-af17-6dceac1af4a1-kube-api-access-9lmxt\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-8faad6-29890860-wv6gj\" (UID: \"fd6cc1a2-16d5-4e08-af17-6dceac1af4a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-8faad6-29890860-wv6gj"
Oct 31 13:00:00.475602 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:00.475514    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-2d6201-29890860-8p4sf"
Oct 31 13:00:00.492037 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:00.492008    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c67de2-29890860-bpnn6"
Oct 31 13:00:00.498474 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:00.498447    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-8faad6-29890860-wv6gj"
Oct 31 13:00:00.785013 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:00.784974    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-2d6201-29890860-8p4sf"]
Oct 31 13:00:00.788825 ip-10-0-130-190 kubenswrapper[5060]: W1031 13:00:00.788780    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a318cfb_1970_4ec2_8488_07de8420ba8e.slice/crio-1409d0deb873140f2a9c7bc11131dae26d91d4d1fb975abf25bc09743bfe3d6d WatchSource:0}: Error finding container 1409d0deb873140f2a9c7bc11131dae26d91d4d1fb975abf25bc09743bfe3d6d: Status 404 returned error can't find the container with id 1409d0deb873140f2a9c7bc11131dae26d91d4d1fb975abf25bc09743bfe3d6d
Oct 31 13:00:00.811666 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:00.811642    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c67de2-29890860-bpnn6"]
Oct 31 13:00:00.814064 ip-10-0-130-190 kubenswrapper[5060]: W1031 13:00:00.814039    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1a17f30_378e_4b5c_81c4_a6f50503b909.slice/crio-ca322abb4a51e9d21274ee5310a5671b93b086bcecc4c39e44faae5fc1b29430 WatchSource:0}: Error finding container ca322abb4a51e9d21274ee5310a5671b93b086bcecc4c39e44faae5fc1b29430: Status 404 returned error can't find the container with id ca322abb4a51e9d21274ee5310a5671b93b086bcecc4c39e44faae5fc1b29430
Oct 31 13:00:00.830912 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:00.830889    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-8faad6-29890860-wv6gj"]
Oct 31 13:00:00.832724 ip-10-0-130-190 kubenswrapper[5060]: W1031 13:00:00.832700    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd6cc1a2_16d5_4e08_af17_6dceac1af4a1.slice/crio-4ecff9b4a9c39ddc5ef4febef013bd412ccf597ba5af5890e4d5304804d9770b WatchSource:0}: Error finding container 4ecff9b4a9c39ddc5ef4febef013bd412ccf597ba5af5890e4d5304804d9770b: Status 404 returned error can't find the container with id 4ecff9b4a9c39ddc5ef4febef013bd412ccf597ba5af5890e4d5304804d9770b
Oct 31 13:00:01.026101 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:01.026058    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c67de2-29890860-bpnn6" event={"ID":"a1a17f30-378e-4b5c-81c4-a6f50503b909","Type":"ContainerStarted","Data":"ca322abb4a51e9d21274ee5310a5671b93b086bcecc4c39e44faae5fc1b29430"}
Oct 31 13:00:01.027558 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:01.027522    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-2d6201-29890860-8p4sf" event={"ID":"9a318cfb-1970-4ec2-8488-07de8420ba8e","Type":"ContainerStarted","Data":"1409d0deb873140f2a9c7bc11131dae26d91d4d1fb975abf25bc09743bfe3d6d"}
Oct 31 13:00:01.028903 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:01.028882    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-8faad6-29890860-wv6gj" event={"ID":"fd6cc1a2-16d5-4e08-af17-6dceac1af4a1","Type":"ContainerStarted","Data":"4ecff9b4a9c39ddc5ef4febef013bd412ccf597ba5af5890e4d5304804d9770b"}
Oct 31 13:00:02.037764 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:02.037726    5060 generic.go:356] "Generic (PLEG): container finished" podID="a1a17f30-378e-4b5c-81c4-a6f50503b909" containerID="d4c6b0d2cb41b90b3aa67648bff3c94ece59bbf13e77b6e5336ca49a0b4a4fe1" exitCode=60
Oct 31 13:00:02.038218 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:02.037810    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c67de2-29890860-bpnn6" event={"ID":"a1a17f30-378e-4b5c-81c4-a6f50503b909","Type":"ContainerDied","Data":"d4c6b0d2cb41b90b3aa67648bff3c94ece59bbf13e77b6e5336ca49a0b4a4fe1"}
Oct 31 13:00:02.038218 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:02.038073    5060 scope.go:122] "RemoveContainer" containerID="d4c6b0d2cb41b90b3aa67648bff3c94ece59bbf13e77b6e5336ca49a0b4a4fe1"
Oct 31 13:00:02.039598 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:02.039574    5060 generic.go:356] "Generic (PLEG): container finished" podID="9a318cfb-1970-4ec2-8488-07de8420ba8e" containerID="dad751b4062b5d33f4530725a31afb47b340e11faa79e4d987fcda1f0780c7f4" exitCode=60
Oct 31 13:00:02.039698 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:02.039684    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-2d6201-29890860-8p4sf" event={"ID":"9a318cfb-1970-4ec2-8488-07de8420ba8e","Type":"ContainerDied","Data":"dad751b4062b5d33f4530725a31afb47b340e11faa79e4d987fcda1f0780c7f4"}
Oct 31 13:00:02.039861 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:02.039845    5060 scope.go:122] "RemoveContainer" containerID="dad751b4062b5d33f4530725a31afb47b340e11faa79e4d987fcda1f0780c7f4"
Oct 31 13:00:02.051880 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:02.043607    5060 generic.go:356] "Generic (PLEG): container finished" podID="fd6cc1a2-16d5-4e08-af17-6dceac1af4a1" containerID="ecbe26d10c932430e89d6ddecb4305445928cf4afcd1a73ae2182d0bdd37295a" exitCode=60
Oct 31 13:00:02.051880 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:02.043700    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-8faad6-29890860-wv6gj" event={"ID":"fd6cc1a2-16d5-4e08-af17-6dceac1af4a1","Type":"ContainerDied","Data":"ecbe26d10c932430e89d6ddecb4305445928cf4afcd1a73ae2182d0bdd37295a"}
Oct 31 13:00:02.051880 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:02.044125    5060 scope.go:122] "RemoveContainer" containerID="ecbe26d10c932430e89d6ddecb4305445928cf4afcd1a73ae2182d0bdd37295a"
Oct 31 13:00:02.901442 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:02.901404    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-b25d3b-66f84f5d8f-6w4n2" podUID="85cd1407-54f9-493a-b8fd-9e638a6615a5" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 13:00:03.053225 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:03.053193    5060 generic.go:356] "Generic (PLEG): container finished" podID="a1a17f30-378e-4b5c-81c4-a6f50503b909" containerID="9d39d890ee8dfe598ba2549a8bc6856e2ede6c8ce15d7dc85594699327aaecdd" exitCode=60
Oct 31 13:00:03.053724 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:03.053270    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c67de2-29890860-bpnn6" event={"ID":"a1a17f30-378e-4b5c-81c4-a6f50503b909","Type":"ContainerDied","Data":"9d39d890ee8dfe598ba2549a8bc6856e2ede6c8ce15d7dc85594699327aaecdd"}
Oct 31 13:00:03.053724 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:03.053333    5060 scope.go:122] "RemoveContainer" containerID="d4c6b0d2cb41b90b3aa67648bff3c94ece59bbf13e77b6e5336ca49a0b4a4fe1"
Oct 31 13:00:03.053724 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:03.053530    5060 scope.go:122] "RemoveContainer" containerID="9d39d890ee8dfe598ba2549a8bc6856e2ede6c8ce15d7dc85594699327aaecdd"
Oct 31 13:00:03.053901 ip-10-0-130-190 kubenswrapper[5060]: E1031 13:00:03.053824    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-c67de2-29890860-bpnn6_odh-ai-gateway-infra(a1a17f30-378e-4b5c-81c4-a6f50503b909)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c67de2-29890860-bpnn6" podUID="a1a17f30-378e-4b5c-81c4-a6f50503b909"
Oct 31 13:00:03.055660 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:03.055632    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-2d6201-29890860-8p4sf" event={"ID":"9a318cfb-1970-4ec2-8488-07de8420ba8e","Type":"ContainerStarted","Data":"4cc9a7768d074bd391ced05082ee5b4be8a70ef6fa1e21c295a8599b32016e53"}
Oct 31 13:00:03.057998 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:03.057973    5060 generic.go:356] "Generic (PLEG): container finished" podID="fd6cc1a2-16d5-4e08-af17-6dceac1af4a1" containerID="e9fffc5f357c3b129fc576044452da203126f580776f4169e8c83c8b675dd2ca" exitCode=60
Oct 31 13:00:03.058116 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:03.058068    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-8faad6-29890860-wv6gj" event={"ID":"fd6cc1a2-16d5-4e08-af17-6dceac1af4a1","Type":"ContainerDied","Data":"e9fffc5f357c3b129fc576044452da203126f580776f4169e8c83c8b675dd2ca"}
Oct 31 13:00:03.058288 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:03.058272    5060 scope.go:122] "RemoveContainer" containerID="e9fffc5f357c3b129fc576044452da203126f580776f4169e8c83c8b675dd2ca"
Oct 31 13:00:03.058533 ip-10-0-130-190 kubenswrapper[5060]: E1031 13:00:03.058511    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-8faad6-29890860-wv6gj_odh-ai-gateway-infra(fd6cc1a2-16d5-4e08-af17-6dceac1af4a1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-8faad6-29890860-wv6gj" podUID="fd6cc1a2-16d5-4e08-af17-6dceac1af4a1"
Oct 31 13:00:03.092450 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:03.092395    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-2d6201-29890860-8p4sf" podStartSLOduration=3.092378637 podStartE2EDuration="3.092378637s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:00:03.088619405 +0000 UTC m=+1966.191687116" watchObservedRunningTime="2026-10-31 13:00:03.092378637 +0000 UTC m=+1966.195446350"
Oct 31 13:00:03.099131 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:03.099112    5060 scope.go:122] "RemoveContainer" containerID="ecbe26d10c932430e89d6ddecb4305445928cf4afcd1a73ae2182d0bdd37295a"
Oct 31 13:00:04.071067 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:04.071033    5060 scope.go:122] "RemoveContainer" containerID="9d39d890ee8dfe598ba2549a8bc6856e2ede6c8ce15d7dc85594699327aaecdd"
Oct 31 13:00:04.071611 ip-10-0-130-190 kubenswrapper[5060]: E1031 13:00:04.071363    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-c67de2-29890860-bpnn6_odh-ai-gateway-infra(a1a17f30-378e-4b5c-81c4-a6f50503b909)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c67de2-29890860-bpnn6" podUID="a1a17f30-378e-4b5c-81c4-a6f50503b909"
Oct 31 13:00:04.072964 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:04.072932    5060 generic.go:356] "Generic (PLEG): container finished" podID="9a318cfb-1970-4ec2-8488-07de8420ba8e" containerID="4cc9a7768d074bd391ced05082ee5b4be8a70ef6fa1e21c295a8599b32016e53" exitCode=60
Oct 31 13:00:04.073110 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:04.072992    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-2d6201-29890860-8p4sf" event={"ID":"9a318cfb-1970-4ec2-8488-07de8420ba8e","Type":"ContainerDied","Data":"4cc9a7768d074bd391ced05082ee5b4be8a70ef6fa1e21c295a8599b32016e53"}
Oct 31 13:00:04.073110 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:04.073053    5060 scope.go:122] "RemoveContainer" containerID="dad751b4062b5d33f4530725a31afb47b340e11faa79e4d987fcda1f0780c7f4"
Oct 31 13:00:04.073324 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:04.073301    5060 scope.go:122] "RemoveContainer" containerID="4cc9a7768d074bd391ced05082ee5b4be8a70ef6fa1e21c295a8599b32016e53"
Oct 31 13:00:04.073663 ip-10-0-130-190 kubenswrapper[5060]: E1031 13:00:04.073640    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-2d6201-29890860-8p4sf_odh-ai-gateway-infra(9a318cfb-1970-4ec2-8488-07de8420ba8e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-2d6201-29890860-8p4sf" podUID="9a318cfb-1970-4ec2-8488-07de8420ba8e"
Oct 31 13:00:04.075401 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:04.075378    5060 scope.go:122] "RemoveContainer" containerID="e9fffc5f357c3b129fc576044452da203126f580776f4169e8c83c8b675dd2ca"
Oct 31 13:00:04.075672 ip-10-0-130-190 kubenswrapper[5060]: E1031 13:00:04.075649    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-8faad6-29890860-wv6gj_odh-ai-gateway-infra(fd6cc1a2-16d5-4e08-af17-6dceac1af4a1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-8faad6-29890860-wv6gj" podUID="fd6cc1a2-16d5-4e08-af17-6dceac1af4a1"
Oct 31 13:00:04.363564 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:04.347449    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-9b17b9-openshift-default-598d5bf87-rr64l"]
Oct 31 13:00:04.363564 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:04.347719    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w6-9b17b9-openshift-default-598d5bf87-rr64l" podUID="fcde778f-8f1a-4c1a-a497-6c66733d54ff" containerName="istio-proxy" containerID="cri-o://5a4614f248369d23f04985ae910ff69be39a38a7d5d1959fd4ff018ca7e272ab" gracePeriod=30
Oct 31 13:00:04.611682 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:04.611640    5060 patch_prober.go:29] interesting pod/e2e-shared-a-w6-9b17b9-openshift-default-598d5bf87-rr64l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 13:00:04.611880 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:04.611707    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w6-9b17b9-openshift-default-598d5bf87-rr64l" podUID="fcde778f-8f1a-4c1a-a497-6c66733d54ff" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 13:00:04.889939 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:04.889898    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-8faad6-29890860-wv6gj"]
Oct 31 13:00:05.134068 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:05.134028    5060 scope.go:122] "RemoveContainer" containerID="4cc9a7768d074bd391ced05082ee5b4be8a70ef6fa1e21c295a8599b32016e53"
Oct 31 13:00:05.134590 ip-10-0-130-190 kubenswrapper[5060]: E1031 13:00:05.134336    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-2d6201-29890860-8p4sf_odh-ai-gateway-infra(9a318cfb-1970-4ec2-8488-07de8420ba8e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-2d6201-29890860-8p4sf" podUID="9a318cfb-1970-4ec2-8488-07de8420ba8e"
Oct 31 13:00:05.320273 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:05.320244    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-8faad6-29890860-wv6gj"
Oct 31 13:00:05.423612 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:05.423578    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd6cc1a2-16d5-4e08-af17-6dceac1af4a1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd6cc1a2-16d5-4e08-af17-6dceac1af4a1-maas-api-service-ca\") pod \"fd6cc1a2-16d5-4e08-af17-6dceac1af4a1\" (UID: \"fd6cc1a2-16d5-4e08-af17-6dceac1af4a1\") "
Oct 31 13:00:05.423813 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:05.423648    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd6cc1a2-16d5-4e08-af17-6dceac1af4a1-kube-api-access-9lmxt\" (UniqueName: \"kubernetes.io/projected/fd6cc1a2-16d5-4e08-af17-6dceac1af4a1-kube-api-access-9lmxt\") pod \"fd6cc1a2-16d5-4e08-af17-6dceac1af4a1\" (UID: \"fd6cc1a2-16d5-4e08-af17-6dceac1af4a1\") "
Oct 31 13:00:05.424006 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:05.423969    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd6cc1a2-16d5-4e08-af17-6dceac1af4a1-maas-api-service-ca" pod "fd6cc1a2-16d5-4e08-af17-6dceac1af4a1" (UID: "fd6cc1a2-16d5-4e08-af17-6dceac1af4a1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:00:05.425957 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:05.425930    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd6cc1a2-16d5-4e08-af17-6dceac1af4a1-kube-api-access-9lmxt" pod "fd6cc1a2-16d5-4e08-af17-6dceac1af4a1" (UID: "fd6cc1a2-16d5-4e08-af17-6dceac1af4a1"). InnerVolumeSpecName "kube-api-access-9lmxt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:00:05.525459 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:05.525426    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd6cc1a2-16d5-4e08-af17-6dceac1af4a1-maas-api-service-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:05.525459 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:05.525460    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9lmxt\" (UniqueName: \"kubernetes.io/projected/fd6cc1a2-16d5-4e08-af17-6dceac1af4a1-kube-api-access-9lmxt\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:06.152238 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:06.152200    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-8faad6-29890860-wv6gj"
Oct 31 13:00:06.152238 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:06.152211    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-8faad6-29890860-wv6gj" event={"ID":"fd6cc1a2-16d5-4e08-af17-6dceac1af4a1","Type":"ContainerDied","Data":"4ecff9b4a9c39ddc5ef4febef013bd412ccf597ba5af5890e4d5304804d9770b"}
Oct 31 13:00:06.152812 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:06.152293    5060 scope.go:122] "RemoveContainer" containerID="e9fffc5f357c3b129fc576044452da203126f580776f4169e8c83c8b675dd2ca"
Oct 31 13:00:06.195544 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:06.195498    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-8faad6-29890860-wv6gj"]
Oct 31 13:00:06.199247 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:06.199218    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-8faad6-29890860-wv6gj"]
Oct 31 13:00:07.049804 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:07.049769    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd6cc1a2-16d5-4e08-af17-6dceac1af4a1" path="/var/lib/kubelet/pods/fd6cc1a2-16d5-4e08-af17-6dceac1af4a1/volumes"
Oct 31 13:00:07.546886 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:07.546857    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-b25d3b-66f84f5d8f-6w4n2"
Oct 31 13:00:07.551248 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:07.551224    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85cd1407-54f9-493a-b8fd-9e638a6615a5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85cd1407-54f9-493a-b8fd-9e638a6615a5-plugins-config-volume\") pod \"85cd1407-54f9-493a-b8fd-9e638a6615a5\" (UID: \"85cd1407-54f9-493a-b8fd-9e638a6615a5\") "
Oct 31 13:00:07.551419 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:07.551405    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85cd1407-54f9-493a-b8fd-9e638a6615a5-kube-api-access-c9bzr\" (UniqueName: \"kubernetes.io/projected/85cd1407-54f9-493a-b8fd-9e638a6615a5-kube-api-access-c9bzr\") pod \"85cd1407-54f9-493a-b8fd-9e638a6615a5\" (UID: \"85cd1407-54f9-493a-b8fd-9e638a6615a5\") "
Oct 31 13:00:07.551611 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:07.551589    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85cd1407-54f9-493a-b8fd-9e638a6615a5-plugins-config-volume" pod "85cd1407-54f9-493a-b8fd-9e638a6615a5" (UID: "85cd1407-54f9-493a-b8fd-9e638a6615a5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:00:07.551831 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:07.551811    5060 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85cd1407-54f9-493a-b8fd-9e638a6615a5-plugins-config-volume\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:07.553534 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:07.553508    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85cd1407-54f9-493a-b8fd-9e638a6615a5-kube-api-access-c9bzr" pod "85cd1407-54f9-493a-b8fd-9e638a6615a5" (UID: "85cd1407-54f9-493a-b8fd-9e638a6615a5"). InnerVolumeSpecName "kube-api-access-c9bzr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:00:07.653344 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:07.653177    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9bzr\" (UniqueName: \"kubernetes.io/projected/85cd1407-54f9-493a-b8fd-9e638a6615a5-kube-api-access-c9bzr\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:07.672771 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:07.669991    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-fcn5x"]
Oct 31 13:00:07.672771 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:07.670222    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-fcn5x" podUID="aae46d99-cc6d-4be9-868b-8c4dda15a7a7" containerName="istio-proxy" containerID="cri-o://147a513a685d7a0c89a75e10e6b013e2bf878ffdfedb1103a44b4588a8e07e27" gracePeriod=30
Oct 31 13:00:07.674839 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:07.674810    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-ffvcd"]
Oct 31 13:00:07.675066 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:07.675041    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-ffvcd" podUID="a72336d5-83db-484a-a300-de68dc5a37c4" containerName="istio-proxy" containerID="cri-o://26c609efe5737af2f6c4b4b2a014ebee99b278f61bb417df55fd22f54f40072a" gracePeriod=30
Oct 31 13:00:08.072957 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:08.072919    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9a5be90-mtmc2"]
Oct 31 13:00:08.092324 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:08.092286    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9a5be90-mtmc2"]
Oct 31 13:00:08.171084 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:08.171045    5060 generic.go:356] "Generic (PLEG): container finished" podID="85cd1407-54f9-493a-b8fd-9e638a6615a5" containerID="751646ab827e81ab96b1b3b08200b865727aa93d32279c669f803a16eb4d57ba" exitCode=0
Oct 31 13:00:08.171275 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:08.171209    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-b25d3b-66f84f5d8f-6w4n2" event={"ID":"85cd1407-54f9-493a-b8fd-9e638a6615a5","Type":"ContainerDied","Data":"751646ab827e81ab96b1b3b08200b865727aa93d32279c669f803a16eb4d57ba"}
Oct 31 13:00:08.171367 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:08.171290    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-b25d3b-66f84f5d8f-6w4n2" event={"ID":"85cd1407-54f9-493a-b8fd-9e638a6615a5","Type":"ContainerDied","Data":"7fb845f41e3876323691e0e1bb5ed767c63209ca39d22d360f9ecedf915c3c49"}
Oct 31 13:00:08.171367 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:08.171321    5060 scope.go:122] "RemoveContainer" containerID="751646ab827e81ab96b1b3b08200b865727aa93d32279c669f803a16eb4d57ba"
Oct 31 13:00:08.171525 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:08.171433    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-b25d3b-66f84f5d8f-6w4n2"
Oct 31 13:00:08.191027 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:08.190996    5060 scope.go:122] "RemoveContainer" containerID="751646ab827e81ab96b1b3b08200b865727aa93d32279c669f803a16eb4d57ba"
Oct 31 13:00:08.191336 ip-10-0-130-190 kubenswrapper[5060]: E1031 13:00:08.191305    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"751646ab827e81ab96b1b3b08200b865727aa93d32279c669f803a16eb4d57ba\": container with ID starting with 751646ab827e81ab96b1b3b08200b865727aa93d32279c669f803a16eb4d57ba not found: ID does not exist" containerID="751646ab827e81ab96b1b3b08200b865727aa93d32279c669f803a16eb4d57ba"
Oct 31 13:00:08.191439 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:08.191349    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"751646ab827e81ab96b1b3b08200b865727aa93d32279c669f803a16eb4d57ba"} err="failed to get container status \"751646ab827e81ab96b1b3b08200b865727aa93d32279c669f803a16eb4d57ba\": rpc error: code = NotFound desc = could not find container \"751646ab827e81ab96b1b3b08200b865727aa93d32279c669f803a16eb4d57ba\": container with ID starting with 751646ab827e81ab96b1b3b08200b865727aa93d32279c669f803a16eb4d57ba not found: ID does not exist"
Oct 31 13:00:08.199838 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:08.199809    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-b25d3b-66f84f5d8f-6w4n2"]
Oct 31 13:00:08.210022 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:08.209993    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-b25d3b-66f84f5d8f-6w4n2"]
Oct 31 13:00:08.603706 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:08.603674    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-8faad6-s64xf"]
Oct 31 13:00:08.610111 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:08.609944    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-8faad6-s64xf"]
Oct 31 13:00:09.051772 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:09.051739    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f4d9298-c434-4443-8014-a1175d8ddb08" path="/var/lib/kubelet/pods/6f4d9298-c434-4443-8014-a1175d8ddb08/volumes"
Oct 31 13:00:09.052242 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:09.052225    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78c3c8ea-96af-4239-b430-75173a1e3fba" path="/var/lib/kubelet/pods/78c3c8ea-96af-4239-b430-75173a1e3fba/volumes"
Oct 31 13:00:09.052676 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:09.052658    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85cd1407-54f9-493a-b8fd-9e638a6615a5" path="/var/lib/kubelet/pods/85cd1407-54f9-493a-b8fd-9e638a6615a5/volumes"
Oct 31 13:00:09.634933 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:09.634903    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-9b17b9-openshift-default-598d5bf87-rr64l"
Oct 31 13:00:09.675472 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:09.675383    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fcde778f-8f1a-4c1a-a497-6c66733d54ff-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcde778f-8f1a-4c1a-a497-6c66733d54ff-istio-podinfo\") pod \"fcde778f-8f1a-4c1a-a497-6c66733d54ff\" (UID: \"fcde778f-8f1a-4c1a-a497-6c66733d54ff\") "
Oct 31 13:00:09.675664 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:09.675479    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fcde778f-8f1a-4c1a-a497-6c66733d54ff-istio-token\" (UniqueName: \"kubernetes.io/projected/fcde778f-8f1a-4c1a-a497-6c66733d54ff-istio-token\") pod \"fcde778f-8f1a-4c1a-a497-6c66733d54ff\" (UID: \"fcde778f-8f1a-4c1a-a497-6c66733d54ff\") "
Oct 31 13:00:09.675664 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:09.675574    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcde778f-8f1a-4c1a-a497-6c66733d54ff-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcde778f-8f1a-4c1a-a497-6c66733d54ff-istio-data\") pod \"fcde778f-8f1a-4c1a-a497-6c66733d54ff\" (UID: \"fcde778f-8f1a-4c1a-a497-6c66733d54ff\") "
Oct 31 13:00:09.675664 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:09.675631    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fcde778f-8f1a-4c1a-a497-6c66733d54ff-kube-api-access-gb8hv\" (UniqueName: \"kubernetes.io/projected/fcde778f-8f1a-4c1a-a497-6c66733d54ff-kube-api-access-gb8hv\") pod \"fcde778f-8f1a-4c1a-a497-6c66733d54ff\" (UID: \"fcde778f-8f1a-4c1a-a497-6c66733d54ff\") "
Oct 31 13:00:09.675910 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:09.675691    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcde778f-8f1a-4c1a-a497-6c66733d54ff-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcde778f-8f1a-4c1a-a497-6c66733d54ff-istio-envoy\") pod \"fcde778f-8f1a-4c1a-a497-6c66733d54ff\" (UID: \"fcde778f-8f1a-4c1a-a497-6c66733d54ff\") "
Oct 31 13:00:09.675910 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:09.675728    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcde778f-8f1a-4c1a-a497-6c66733d54ff-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcde778f-8f1a-4c1a-a497-6c66733d54ff-workload-socket\") pod \"fcde778f-8f1a-4c1a-a497-6c66733d54ff\" (UID: \"fcde778f-8f1a-4c1a-a497-6c66733d54ff\") "
Oct 31 13:00:09.675910 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:09.675829    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fcde778f-8f1a-4c1a-a497-6c66733d54ff-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcde778f-8f1a-4c1a-a497-6c66733d54ff-istiod-ca-cert\") pod \"fcde778f-8f1a-4c1a-a497-6c66733d54ff\" (UID: \"fcde778f-8f1a-4c1a-a497-6c66733d54ff\") "
Oct 31 13:00:09.675910 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:09.675869    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcde778f-8f1a-4c1a-a497-6c66733d54ff-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcde778f-8f1a-4c1a-a497-6c66733d54ff-credential-socket\") pod \"fcde778f-8f1a-4c1a-a497-6c66733d54ff\" (UID: \"fcde778f-8f1a-4c1a-a497-6c66733d54ff\") "
Oct 31 13:00:09.675910 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:09.675892    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcde778f-8f1a-4c1a-a497-6c66733d54ff-istio-data" pod "fcde778f-8f1a-4c1a-a497-6c66733d54ff" (UID: "fcde778f-8f1a-4c1a-a497-6c66733d54ff"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:00:09.675910 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:09.675908    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcde778f-8f1a-4c1a-a497-6c66733d54ff-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcde778f-8f1a-4c1a-a497-6c66733d54ff-workload-certs\") pod \"fcde778f-8f1a-4c1a-a497-6c66733d54ff\" (UID: \"fcde778f-8f1a-4c1a-a497-6c66733d54ff\") "
Oct 31 13:00:09.676242 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:09.676112    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcde778f-8f1a-4c1a-a497-6c66733d54ff-workload-socket" pod "fcde778f-8f1a-4c1a-a497-6c66733d54ff" (UID: "fcde778f-8f1a-4c1a-a497-6c66733d54ff"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:00:09.676361 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:09.676337    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcde778f-8f1a-4c1a-a497-6c66733d54ff-workload-certs" pod "fcde778f-8f1a-4c1a-a497-6c66733d54ff" (UID: "fcde778f-8f1a-4c1a-a497-6c66733d54ff"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:00:09.676483 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:09.676466    5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcde778f-8f1a-4c1a-a497-6c66733d54ff-workload-certs\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:09.676483 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:09.676512    5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcde778f-8f1a-4c1a-a497-6c66733d54ff-istio-data\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:09.676683 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:09.676529    5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcde778f-8f1a-4c1a-a497-6c66733d54ff-workload-socket\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:09.676683 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:09.676660    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcde778f-8f1a-4c1a-a497-6c66733d54ff-istiod-ca-cert" pod "fcde778f-8f1a-4c1a-a497-6c66733d54ff" (UID: "fcde778f-8f1a-4c1a-a497-6c66733d54ff"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:00:09.676947 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:09.676923    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcde778f-8f1a-4c1a-a497-6c66733d54ff-credential-socket" pod "fcde778f-8f1a-4c1a-a497-6c66733d54ff" (UID: "fcde778f-8f1a-4c1a-a497-6c66733d54ff"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:00:09.678257 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:09.678225    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcde778f-8f1a-4c1a-a497-6c66733d54ff-istio-token" pod "fcde778f-8f1a-4c1a-a497-6c66733d54ff" (UID: "fcde778f-8f1a-4c1a-a497-6c66733d54ff"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:00:09.678656 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:09.678635    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcde778f-8f1a-4c1a-a497-6c66733d54ff-istio-envoy" pod "fcde778f-8f1a-4c1a-a497-6c66733d54ff" (UID: "fcde778f-8f1a-4c1a-a497-6c66733d54ff"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:00:09.679787 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:09.679768    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fcde778f-8f1a-4c1a-a497-6c66733d54ff-istio-podinfo" pod "fcde778f-8f1a-4c1a-a497-6c66733d54ff" (UID: "fcde778f-8f1a-4c1a-a497-6c66733d54ff"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 13:00:09.679866 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:09.679842    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcde778f-8f1a-4c1a-a497-6c66733d54ff-kube-api-access-gb8hv" pod "fcde778f-8f1a-4c1a-a497-6c66733d54ff" (UID: "fcde778f-8f1a-4c1a-a497-6c66733d54ff"). InnerVolumeSpecName "kube-api-access-gb8hv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:00:09.777654 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:09.777613    5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcde778f-8f1a-4c1a-a497-6c66733d54ff-istiod-ca-cert\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:09.777654 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:09.777648    5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcde778f-8f1a-4c1a-a497-6c66733d54ff-credential-socket\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:09.777654 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:09.777662    5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcde778f-8f1a-4c1a-a497-6c66733d54ff-istio-podinfo\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:09.777909 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:09.777675    5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fcde778f-8f1a-4c1a-a497-6c66733d54ff-istio-token\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:09.777909 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:09.777690    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gb8hv\" (UniqueName: \"kubernetes.io/projected/fcde778f-8f1a-4c1a-a497-6c66733d54ff-kube-api-access-gb8hv\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:09.777909 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:09.777704    5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcde778f-8f1a-4c1a-a497-6c66733d54ff-istio-envoy\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:10.193448 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:10.193417    5060 generic.go:356] "Generic (PLEG): container finished" podID="fcde778f-8f1a-4c1a-a497-6c66733d54ff" containerID="5a4614f248369d23f04985ae910ff69be39a38a7d5d1959fd4ff018ca7e272ab" exitCode=0
Oct 31 13:00:10.193680 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:10.193484    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-9b17b9-openshift-default-598d5bf87-rr64l"
Oct 31 13:00:10.193680 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:10.193516    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-9b17b9-openshift-default-598d5bf87-rr64l" event={"ID":"fcde778f-8f1a-4c1a-a497-6c66733d54ff","Type":"ContainerDied","Data":"5a4614f248369d23f04985ae910ff69be39a38a7d5d1959fd4ff018ca7e272ab"}
Oct 31 13:00:10.193680 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:10.193642    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-9b17b9-openshift-default-598d5bf87-rr64l" event={"ID":"fcde778f-8f1a-4c1a-a497-6c66733d54ff","Type":"ContainerDied","Data":"05c06bd58510d98086307a019cd0561d638b4de7e331357b49880153f1df33ee"}
Oct 31 13:00:10.193680 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:10.193668    5060 scope.go:122] "RemoveContainer" containerID="5a4614f248369d23f04985ae910ff69be39a38a7d5d1959fd4ff018ca7e272ab"
Oct 31 13:00:10.206358 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:10.206339    5060 scope.go:122] "RemoveContainer" containerID="be48cd4e14df08136b036625e90d6dac400cba28c17756e71993064162ace3e4"
Oct 31 13:00:10.220764 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:10.220740    5060 scope.go:122] "RemoveContainer" containerID="5a4614f248369d23f04985ae910ff69be39a38a7d5d1959fd4ff018ca7e272ab"
Oct 31 13:00:10.220843 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:10.220811    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-9b17b9-openshift-default-598d5bf87-rr64l"]
Oct 31 13:00:10.221101 ip-10-0-130-190 kubenswrapper[5060]: E1031 13:00:10.221074    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a4614f248369d23f04985ae910ff69be39a38a7d5d1959fd4ff018ca7e272ab\": container with ID starting with 5a4614f248369d23f04985ae910ff69be39a38a7d5d1959fd4ff018ca7e272ab not found: ID does not exist" containerID="5a4614f248369d23f04985ae910ff69be39a38a7d5d1959fd4ff018ca7e272ab"
Oct 31 13:00:10.221153 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:10.221113    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a4614f248369d23f04985ae910ff69be39a38a7d5d1959fd4ff018ca7e272ab"} err="failed to get container status \"5a4614f248369d23f04985ae910ff69be39a38a7d5d1959fd4ff018ca7e272ab\": rpc error: code = NotFound desc = could not find container \"5a4614f248369d23f04985ae910ff69be39a38a7d5d1959fd4ff018ca7e272ab\": container with ID starting with 5a4614f248369d23f04985ae910ff69be39a38a7d5d1959fd4ff018ca7e272ab not found: ID does not exist"
Oct 31 13:00:10.221153 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:10.221138    5060 scope.go:122] "RemoveContainer" containerID="be48cd4e14df08136b036625e90d6dac400cba28c17756e71993064162ace3e4"
Oct 31 13:00:10.221410 ip-10-0-130-190 kubenswrapper[5060]: E1031 13:00:10.221390    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be48cd4e14df08136b036625e90d6dac400cba28c17756e71993064162ace3e4\": container with ID starting with be48cd4e14df08136b036625e90d6dac400cba28c17756e71993064162ace3e4 not found: ID does not exist" containerID="be48cd4e14df08136b036625e90d6dac400cba28c17756e71993064162ace3e4"
Oct 31 13:00:10.221479 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:10.221416    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be48cd4e14df08136b036625e90d6dac400cba28c17756e71993064162ace3e4"} err="failed to get container status \"be48cd4e14df08136b036625e90d6dac400cba28c17756e71993064162ace3e4\": rpc error: code = NotFound desc = could not find container \"be48cd4e14df08136b036625e90d6dac400cba28c17756e71993064162ace3e4\": container with ID starting with be48cd4e14df08136b036625e90d6dac400cba28c17756e71993064162ace3e4 not found: ID does not exist"
Oct 31 13:00:10.229161 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:10.229131    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-9b17b9-openshift-default-598d5bf87-rr64l"]
Oct 31 13:00:11.048281 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:11.048245    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fcde778f-8f1a-4c1a-a497-6c66733d54ff" path="/var/lib/kubelet/pods/fcde778f-8f1a-4c1a-a497-6c66733d54ff/volumes"
Oct 31 13:00:12.985652 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:12.985617    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-fcn5x"
Oct 31 13:00:13.010431 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:13.010399    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-ffvcd"
Oct 31 13:00:13.013294 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:13.013266    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aae46d99-cc6d-4be9-868b-8c4dda15a7a7-kube-api-access-prvgm\" (UniqueName: \"kubernetes.io/projected/aae46d99-cc6d-4be9-868b-8c4dda15a7a7-kube-api-access-prvgm\") pod \"aae46d99-cc6d-4be9-868b-8c4dda15a7a7\" (UID: \"aae46d99-cc6d-4be9-868b-8c4dda15a7a7\") "
Oct 31 13:00:13.013441 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:13.013364    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aae46d99-cc6d-4be9-868b-8c4dda15a7a7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aae46d99-cc6d-4be9-868b-8c4dda15a7a7-istio-data\") pod \"aae46d99-cc6d-4be9-868b-8c4dda15a7a7\" (UID: \"aae46d99-cc6d-4be9-868b-8c4dda15a7a7\") "
Oct 31 13:00:13.013441 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:13.013401    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aae46d99-cc6d-4be9-868b-8c4dda15a7a7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aae46d99-cc6d-4be9-868b-8c4dda15a7a7-workload-socket\") pod \"aae46d99-cc6d-4be9-868b-8c4dda15a7a7\" (UID: \"aae46d99-cc6d-4be9-868b-8c4dda15a7a7\") "
Oct 31 13:00:13.013441 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:13.013433    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aae46d99-cc6d-4be9-868b-8c4dda15a7a7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aae46d99-cc6d-4be9-868b-8c4dda15a7a7-istio-envoy\") pod \"aae46d99-cc6d-4be9-868b-8c4dda15a7a7\" (UID: \"aae46d99-cc6d-4be9-868b-8c4dda15a7a7\") "
Oct 31 13:00:13.013672 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:13.013559    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aae46d99-cc6d-4be9-868b-8c4dda15a7a7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aae46d99-cc6d-4be9-868b-8c4dda15a7a7-credential-socket\") pod \"aae46d99-cc6d-4be9-868b-8c4dda15a7a7\" (UID: \"aae46d99-cc6d-4be9-868b-8c4dda15a7a7\") "
Oct 31 13:00:13.013672 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:13.013636    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aae46d99-cc6d-4be9-868b-8c4dda15a7a7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aae46d99-cc6d-4be9-868b-8c4dda15a7a7-istio-podinfo\") pod \"aae46d99-cc6d-4be9-868b-8c4dda15a7a7\" (UID: \"aae46d99-cc6d-4be9-868b-8c4dda15a7a7\") "
Oct 31 13:00:13.013672 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:13.013664    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aae46d99-cc6d-4be9-868b-8c4dda15a7a7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aae46d99-cc6d-4be9-868b-8c4dda15a7a7-istiod-ca-cert\") pod \"aae46d99-cc6d-4be9-868b-8c4dda15a7a7\" (UID: \"aae46d99-cc6d-4be9-868b-8c4dda15a7a7\") "
Oct 31 13:00:13.013837 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:13.013697    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aae46d99-cc6d-4be9-868b-8c4dda15a7a7-istio-token\" (UniqueName: \"kubernetes.io/projected/aae46d99-cc6d-4be9-868b-8c4dda15a7a7-istio-token\") pod \"aae46d99-cc6d-4be9-868b-8c4dda15a7a7\" (UID: \"aae46d99-cc6d-4be9-868b-8c4dda15a7a7\") "
Oct 31 13:00:13.013837 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:13.013745    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aae46d99-cc6d-4be9-868b-8c4dda15a7a7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aae46d99-cc6d-4be9-868b-8c4dda15a7a7-workload-certs\") pod \"aae46d99-cc6d-4be9-868b-8c4dda15a7a7\" (UID: \"aae46d99-cc6d-4be9-868b-8c4dda15a7a7\") "
Oct 31 13:00:13.014362 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:13.014334    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aae46d99-cc6d-4be9-868b-8c4dda15a7a7-workload-certs" pod "aae46d99-cc6d-4be9-868b-8c4dda15a7a7" (UID: "aae46d99-cc6d-4be9-868b-8c4dda15a7a7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:00:13.014598 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:13.014574    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aae46d99-cc6d-4be9-868b-8c4dda15a7a7-credential-socket" pod "aae46d99-cc6d-4be9-868b-8c4dda15a7a7" (UID: "aae46d99-cc6d-4be9-868b-8c4dda15a7a7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:00:13.015420 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:13.015389    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aae46d99-cc6d-4be9-868b-8c4dda15a7a7-istiod-ca-cert" pod "aae46d99-cc6d-4be9-868b-8c4dda15a7a7" (UID: "aae46d99-cc6d-4be9-868b-8c4dda15a7a7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:00:13.015717 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:13.015690    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aae46d99-cc6d-4be9-868b-8c4dda15a7a7-istio-data" pod "aae46d99-cc6d-4be9-868b-8c4dda15a7a7" (UID: "aae46d99-cc6d-4be9-868b-8c4dda15a7a7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:00:13.016281 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:13.016249    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aae46d99-cc6d-4be9-868b-8c4dda15a7a7-workload-socket" pod "aae46d99-cc6d-4be9-868b-8c4dda15a7a7" (UID: "aae46d99-cc6d-4be9-868b-8c4dda15a7a7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:00:13.016376 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:13.016312    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aae46d99-cc6d-4be9-868b-8c4dda15a7a7-istio-envoy" pod "aae46d99-cc6d-4be9-868b-8c4dda15a7a7" (UID: "aae46d99-cc6d-4be9-868b-8c4dda15a7a7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:00:13.017884 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:13.017859    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aae46d99-cc6d-4be9-868b-8c4dda15a7a7-istio-podinfo" pod "aae46d99-cc6d-4be9-868b-8c4dda15a7a7" (UID: "aae46d99-cc6d-4be9-868b-8c4dda15a7a7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 13:00:13.018190 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:13.018162    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aae46d99-cc6d-4be9-868b-8c4dda15a7a7-kube-api-access-prvgm" pod "aae46d99-cc6d-4be9-868b-8c4dda15a7a7" (UID: "aae46d99-cc6d-4be9-868b-8c4dda15a7a7"). InnerVolumeSpecName "kube-api-access-prvgm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:00:13.019111 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:13.019093    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aae46d99-cc6d-4be9-868b-8c4dda15a7a7-istio-token" pod "aae46d99-cc6d-4be9-868b-8c4dda15a7a7" (UID: "aae46d99-cc6d-4be9-868b-8c4dda15a7a7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:00:13.041314 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:13.041288    5060 scope.go:122] "RemoveContainer" containerID="9d39d890ee8dfe598ba2549a8bc6856e2ede6c8ce15d7dc85594699327aaecdd"
Oct 31 13:00:13.115000 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:13.114969    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a72336d5-83db-484a-a300-de68dc5a37c4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a72336d5-83db-484a-a300-de68dc5a37c4-istiod-ca-cert\") pod \"a72336d5-83db-484a-a300-de68dc5a37c4\" (UID: \"a72336d5-83db-484a-a300-de68dc5a37c4\") "
Oct 31 13:00:13.115161 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:13.115007    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a72336d5-83db-484a-a300-de68dc5a37c4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a72336d5-83db-484a-a300-de68dc5a37c4-istio-envoy\") pod \"a72336d5-83db-484a-a300-de68dc5a37c4\" (UID: \"a72336d5-83db-484a-a300-de68dc5a37c4\") "
Oct 31 13:00:13.115161 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:13.115088    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a72336d5-83db-484a-a300-de68dc5a37c4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a72336d5-83db-484a-a300-de68dc5a37c4-workload-socket\") pod \"a72336d5-83db-484a-a300-de68dc5a37c4\" (UID: \"a72336d5-83db-484a-a300-de68dc5a37c4\") "
Oct 31 13:00:13.115161 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:13.115118    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a72336d5-83db-484a-a300-de68dc5a37c4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a72336d5-83db-484a-a300-de68dc5a37c4-istio-data\") pod \"a72336d5-83db-484a-a300-de68dc5a37c4\" (UID: \"a72336d5-83db-484a-a300-de68dc5a37c4\") "
Oct 31 13:00:13.115347 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:13.115161    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a72336d5-83db-484a-a300-de68dc5a37c4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a72336d5-83db-484a-a300-de68dc5a37c4-istio-podinfo\") pod \"a72336d5-83db-484a-a300-de68dc5a37c4\" (UID: \"a72336d5-83db-484a-a300-de68dc5a37c4\") "
Oct 31 13:00:13.115347 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:13.115204    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a72336d5-83db-484a-a300-de68dc5a37c4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a72336d5-83db-484a-a300-de68dc5a37c4-workload-certs\") pod \"a72336d5-83db-484a-a300-de68dc5a37c4\" (UID: \"a72336d5-83db-484a-a300-de68dc5a37c4\") "
Oct 31 13:00:13.115347 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:13.115249    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a72336d5-83db-484a-a300-de68dc5a37c4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a72336d5-83db-484a-a300-de68dc5a37c4-credential-socket\") pod \"a72336d5-83db-484a-a300-de68dc5a37c4\" (UID: \"a72336d5-83db-484a-a300-de68dc5a37c4\") "
Oct 31 13:00:13.115522 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:13.115344    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a72336d5-83db-484a-a300-de68dc5a37c4-kube-api-access-25tcp\" (UniqueName: \"kubernetes.io/projected/a72336d5-83db-484a-a300-de68dc5a37c4-kube-api-access-25tcp\") pod \"a72336d5-83db-484a-a300-de68dc5a37c4\" (UID: \"a72336d5-83db-484a-a300-de68dc5a37c4\") "
Oct 31 13:00:13.115522 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:13.115369    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a72336d5-83db-484a-a300-de68dc5a37c4-istiod-ca-cert" pod "a72336d5-83db-484a-a300-de68dc5a37c4" (UID: "a72336d5-83db-484a-a300-de68dc5a37c4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:00:13.115522 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:13.115416    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a72336d5-83db-484a-a300-de68dc5a37c4-workload-socket" pod "a72336d5-83db-484a-a300-de68dc5a37c4" (UID: "a72336d5-83db-484a-a300-de68dc5a37c4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:00:13.115522 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:13.115429    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a72336d5-83db-484a-a300-de68dc5a37c4-istio-token\" (UniqueName: \"kubernetes.io/projected/a72336d5-83db-484a-a300-de68dc5a37c4-istio-token\") pod \"a72336d5-83db-484a-a300-de68dc5a37c4\" (UID: \"a72336d5-83db-484a-a300-de68dc5a37c4\") "
Oct 31 13:00:13.115522 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:13.115473    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a72336d5-83db-484a-a300-de68dc5a37c4-istio-data" pod "a72336d5-83db-484a-a300-de68dc5a37c4" (UID: "a72336d5-83db-484a-a300-de68dc5a37c4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:00:13.115745 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:13.115675    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a72336d5-83db-484a-a300-de68dc5a37c4-workload-certs" pod "a72336d5-83db-484a-a300-de68dc5a37c4" (UID: "a72336d5-83db-484a-a300-de68dc5a37c4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:00:13.115745 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:13.115692    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a72336d5-83db-484a-a300-de68dc5a37c4-credential-socket" pod "a72336d5-83db-484a-a300-de68dc5a37c4" (UID: "a72336d5-83db-484a-a300-de68dc5a37c4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:00:13.115988 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:13.115969    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-prvgm\" (UniqueName: \"kubernetes.io/projected/aae46d99-cc6d-4be9-868b-8c4dda15a7a7-kube-api-access-prvgm\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:13.116051 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:13.115989    5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a72336d5-83db-484a-a300-de68dc5a37c4-workload-certs\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:13.116051 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:13.116000    5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a72336d5-83db-484a-a300-de68dc5a37c4-credential-socket\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:13.116051 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:13.116009    5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aae46d99-cc6d-4be9-868b-8c4dda15a7a7-istio-data\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:13.116051 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:13.116017    5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aae46d99-cc6d-4be9-868b-8c4dda15a7a7-workload-socket\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:13.116051 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:13.116027    5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aae46d99-cc6d-4be9-868b-8c4dda15a7a7-istio-envoy\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:13.116051 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:13.116035    5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aae46d99-cc6d-4be9-868b-8c4dda15a7a7-credential-socket\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:13.116051 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:13.116044    5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a72336d5-83db-484a-a300-de68dc5a37c4-istiod-ca-cert\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:13.116051 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:13.116052    5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aae46d99-cc6d-4be9-868b-8c4dda15a7a7-istio-podinfo\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:13.116544 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:13.116061    5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aae46d99-cc6d-4be9-868b-8c4dda15a7a7-istiod-ca-cert\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:13.116544 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:13.116088    5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aae46d99-cc6d-4be9-868b-8c4dda15a7a7-istio-token\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:13.116544 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:13.116110    5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aae46d99-cc6d-4be9-868b-8c4dda15a7a7-workload-certs\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:13.116544 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:13.116126    5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a72336d5-83db-484a-a300-de68dc5a37c4-workload-socket\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:13.116544 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:13.116143    5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a72336d5-83db-484a-a300-de68dc5a37c4-istio-data\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:13.117530 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:13.117477    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a72336d5-83db-484a-a300-de68dc5a37c4-istio-envoy" pod "a72336d5-83db-484a-a300-de68dc5a37c4" (UID: "a72336d5-83db-484a-a300-de68dc5a37c4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:00:13.117630 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:13.117589    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a72336d5-83db-484a-a300-de68dc5a37c4-istio-podinfo" pod "a72336d5-83db-484a-a300-de68dc5a37c4" (UID: "a72336d5-83db-484a-a300-de68dc5a37c4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 13:00:13.117832 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:13.117813    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a72336d5-83db-484a-a300-de68dc5a37c4-istio-token" pod "a72336d5-83db-484a-a300-de68dc5a37c4" (UID: "a72336d5-83db-484a-a300-de68dc5a37c4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:00:13.117940 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:13.117924    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a72336d5-83db-484a-a300-de68dc5a37c4-kube-api-access-25tcp" pod "a72336d5-83db-484a-a300-de68dc5a37c4" (UID: "a72336d5-83db-484a-a300-de68dc5a37c4"). InnerVolumeSpecName "kube-api-access-25tcp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:00:13.217317 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:13.217231    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-25tcp\" (UniqueName: \"kubernetes.io/projected/a72336d5-83db-484a-a300-de68dc5a37c4-kube-api-access-25tcp\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:13.217317 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:13.217258    5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a72336d5-83db-484a-a300-de68dc5a37c4-istio-token\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:13.217317 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:13.217268    5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a72336d5-83db-484a-a300-de68dc5a37c4-istio-envoy\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:13.217317 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:13.217279    5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a72336d5-83db-484a-a300-de68dc5a37c4-istio-podinfo\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:13.220273 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:13.220243    5060 generic.go:356] "Generic (PLEG): container finished" podID="a72336d5-83db-484a-a300-de68dc5a37c4" containerID="26c609efe5737af2f6c4b4b2a014ebee99b278f61bb417df55fd22f54f40072a" exitCode=0
Oct 31 13:00:13.220412 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:13.220296    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-ffvcd"
Oct 31 13:00:13.220412 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:13.220325    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-ffvcd" event={"ID":"a72336d5-83db-484a-a300-de68dc5a37c4","Type":"ContainerDied","Data":"26c609efe5737af2f6c4b4b2a014ebee99b278f61bb417df55fd22f54f40072a"}
Oct 31 13:00:13.220412 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:13.220384    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-ffvcd" event={"ID":"a72336d5-83db-484a-a300-de68dc5a37c4","Type":"ContainerDied","Data":"d7e26be7fecc6dc60a542fe2e45be74ddd09159a9adaa7d4b8d3f39657a7a2fc"}
Oct 31 13:00:13.220571 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:13.220414    5060 scope.go:122] "RemoveContainer" containerID="26c609efe5737af2f6c4b4b2a014ebee99b278f61bb417df55fd22f54f40072a"
Oct 31 13:00:13.222026 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:13.222001    5060 generic.go:356] "Generic (PLEG): container finished" podID="aae46d99-cc6d-4be9-868b-8c4dda15a7a7" containerID="147a513a685d7a0c89a75e10e6b013e2bf878ffdfedb1103a44b4588a8e07e27" exitCode=0
Oct 31 13:00:13.222169 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:13.222046    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-fcn5x"
Oct 31 13:00:13.222169 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:13.222102    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-fcn5x" event={"ID":"aae46d99-cc6d-4be9-868b-8c4dda15a7a7","Type":"ContainerDied","Data":"147a513a685d7a0c89a75e10e6b013e2bf878ffdfedb1103a44b4588a8e07e27"}
Oct 31 13:00:13.222169 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:13.222148    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-fcn5x" event={"ID":"aae46d99-cc6d-4be9-868b-8c4dda15a7a7","Type":"ContainerDied","Data":"35c5fcef0e641246cde02165fdfdff980cc110c1fff1018e2e07767b548d35a9"}
Oct 31 13:00:13.232666 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:13.232646    5060 scope.go:122] "RemoveContainer" containerID="26c609efe5737af2f6c4b4b2a014ebee99b278f61bb417df55fd22f54f40072a"
Oct 31 13:00:13.232960 ip-10-0-130-190 kubenswrapper[5060]: E1031 13:00:13.232940    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26c609efe5737af2f6c4b4b2a014ebee99b278f61bb417df55fd22f54f40072a\": container with ID starting with 26c609efe5737af2f6c4b4b2a014ebee99b278f61bb417df55fd22f54f40072a not found: ID does not exist" containerID="26c609efe5737af2f6c4b4b2a014ebee99b278f61bb417df55fd22f54f40072a"
Oct 31 13:00:13.233014 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:13.232969    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26c609efe5737af2f6c4b4b2a014ebee99b278f61bb417df55fd22f54f40072a"} err="failed to get container status \"26c609efe5737af2f6c4b4b2a014ebee99b278f61bb417df55fd22f54f40072a\": rpc error: code = NotFound desc = could not find container \"26c609efe5737af2f6c4b4b2a014ebee99b278f61bb417df55fd22f54f40072a\": container with ID starting with 26c609efe5737af2f6c4b4b2a014ebee99b278f61bb417df55fd22f54f40072a not found: ID does not exist"
Oct 31 13:00:13.233014 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:13.232986    5060 scope.go:122] "RemoveContainer" containerID="147a513a685d7a0c89a75e10e6b013e2bf878ffdfedb1103a44b4588a8e07e27"
Oct 31 13:00:13.243687 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:13.243668    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-fcn5x"]
Oct 31 13:00:13.243738 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:13.243709    5060 scope.go:122] "RemoveContainer" containerID="147a513a685d7a0c89a75e10e6b013e2bf878ffdfedb1103a44b4588a8e07e27"
Oct 31 13:00:13.244023 ip-10-0-130-190 kubenswrapper[5060]: E1031 13:00:13.244005    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"147a513a685d7a0c89a75e10e6b013e2bf878ffdfedb1103a44b4588a8e07e27\": container with ID starting with 147a513a685d7a0c89a75e10e6b013e2bf878ffdfedb1103a44b4588a8e07e27 not found: ID does not exist" containerID="147a513a685d7a0c89a75e10e6b013e2bf878ffdfedb1103a44b4588a8e07e27"
Oct 31 13:00:13.244066 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:13.244030    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"147a513a685d7a0c89a75e10e6b013e2bf878ffdfedb1103a44b4588a8e07e27"} err="failed to get container status \"147a513a685d7a0c89a75e10e6b013e2bf878ffdfedb1103a44b4588a8e07e27\": rpc error: code = NotFound desc = could not find container \"147a513a685d7a0c89a75e10e6b013e2bf878ffdfedb1103a44b4588a8e07e27\": container with ID starting with 147a513a685d7a0c89a75e10e6b013e2bf878ffdfedb1103a44b4588a8e07e27 not found: ID does not exist"
Oct 31 13:00:13.250400 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:13.250377    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-fcn5x"]
Oct 31 13:00:13.263221 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:13.263083    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-ffvcd"]
Oct 31 13:00:13.266971 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:13.266768    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-ffvcd"]
Oct 31 13:00:14.241739 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:14.241697    5060 generic.go:356] "Generic (PLEG): container finished" podID="a1a17f30-378e-4b5c-81c4-a6f50503b909" containerID="2b6534d2398bb00cb1d9069a3e7c8b9ccc54b66bd658aa9e6046cc5578b1e36e" exitCode=60
Oct 31 13:00:14.242470 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:14.242427    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c67de2-29890860-bpnn6" event={"ID":"a1a17f30-378e-4b5c-81c4-a6f50503b909","Type":"ContainerDied","Data":"2b6534d2398bb00cb1d9069a3e7c8b9ccc54b66bd658aa9e6046cc5578b1e36e"}
Oct 31 13:00:14.242659 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:14.242578    5060 scope.go:122] "RemoveContainer" containerID="9d39d890ee8dfe598ba2549a8bc6856e2ede6c8ce15d7dc85594699327aaecdd"
Oct 31 13:00:14.243075 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:14.243053    5060 scope.go:122] "RemoveContainer" containerID="2b6534d2398bb00cb1d9069a3e7c8b9ccc54b66bd658aa9e6046cc5578b1e36e"
Oct 31 13:00:14.243558 ip-10-0-130-190 kubenswrapper[5060]: E1031 13:00:14.243479    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-c67de2-29890860-bpnn6_odh-ai-gateway-infra(a1a17f30-378e-4b5c-81c4-a6f50503b909)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c67de2-29890860-bpnn6" podUID="a1a17f30-378e-4b5c-81c4-a6f50503b909"
Oct 31 13:00:15.050906 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:15.050868    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a72336d5-83db-484a-a300-de68dc5a37c4" path="/var/lib/kubelet/pods/a72336d5-83db-484a-a300-de68dc5a37c4/volumes"
Oct 31 13:00:15.051522 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:15.051479    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aae46d99-cc6d-4be9-868b-8c4dda15a7a7" path="/var/lib/kubelet/pods/aae46d99-cc6d-4be9-868b-8c4dda15a7a7/volumes"
Oct 31 13:00:15.171251 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:15.171220    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-2d6201-29890860-8p4sf"]
Oct 31 13:00:15.291482 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:15.291443    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c67de2-29890860-bpnn6"]
Oct 31 13:00:15.354612 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:15.352982    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-2d6201-29890860-8p4sf"
Oct 31 13:00:15.453977 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:15.453934    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a318cfb-1970-4ec2-8488-07de8420ba8e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a318cfb-1970-4ec2-8488-07de8420ba8e-maas-api-service-ca\") pod \"9a318cfb-1970-4ec2-8488-07de8420ba8e\" (UID: \"9a318cfb-1970-4ec2-8488-07de8420ba8e\") "
Oct 31 13:00:15.454169 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:15.454136    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a318cfb-1970-4ec2-8488-07de8420ba8e-kube-api-access-8jvvj\" (UniqueName: \"kubernetes.io/projected/9a318cfb-1970-4ec2-8488-07de8420ba8e-kube-api-access-8jvvj\") pod \"9a318cfb-1970-4ec2-8488-07de8420ba8e\" (UID: \"9a318cfb-1970-4ec2-8488-07de8420ba8e\") "
Oct 31 13:00:15.454423 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:15.454301    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a318cfb-1970-4ec2-8488-07de8420ba8e-maas-api-service-ca" pod "9a318cfb-1970-4ec2-8488-07de8420ba8e" (UID: "9a318cfb-1970-4ec2-8488-07de8420ba8e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:00:15.454687 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:15.454655    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a318cfb-1970-4ec2-8488-07de8420ba8e-maas-api-service-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:15.458429 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:15.457716    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a318cfb-1970-4ec2-8488-07de8420ba8e-kube-api-access-8jvvj" pod "9a318cfb-1970-4ec2-8488-07de8420ba8e" (UID: "9a318cfb-1970-4ec2-8488-07de8420ba8e"). InnerVolumeSpecName "kube-api-access-8jvvj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:00:15.514230 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:15.514204    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c67de2-29890860-bpnn6"
Oct 31 13:00:15.555034 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:15.555003    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1a17f30-378e-4b5c-81c4-a6f50503b909-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1a17f30-378e-4b5c-81c4-a6f50503b909-maas-api-service-ca\") pod \"a1a17f30-378e-4b5c-81c4-a6f50503b909\" (UID: \"a1a17f30-378e-4b5c-81c4-a6f50503b909\") "
Oct 31 13:00:15.555244 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:15.555169    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1a17f30-378e-4b5c-81c4-a6f50503b909-kube-api-access-z8wbj\" (UniqueName: \"kubernetes.io/projected/a1a17f30-378e-4b5c-81c4-a6f50503b909-kube-api-access-z8wbj\") pod \"a1a17f30-378e-4b5c-81c4-a6f50503b909\" (UID: \"a1a17f30-378e-4b5c-81c4-a6f50503b909\") "
Oct 31 13:00:15.555721 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:15.555418    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1a17f30-378e-4b5c-81c4-a6f50503b909-maas-api-service-ca" pod "a1a17f30-378e-4b5c-81c4-a6f50503b909" (UID: "a1a17f30-378e-4b5c-81c4-a6f50503b909"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:00:15.555865 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:15.555724    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1a17f30-378e-4b5c-81c4-a6f50503b909-maas-api-service-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:15.555865 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:15.555748    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jvvj\" (UniqueName: \"kubernetes.io/projected/9a318cfb-1970-4ec2-8488-07de8420ba8e-kube-api-access-8jvvj\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:15.558106 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:15.558075    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1a17f30-378e-4b5c-81c4-a6f50503b909-kube-api-access-z8wbj" pod "a1a17f30-378e-4b5c-81c4-a6f50503b909" (UID: "a1a17f30-378e-4b5c-81c4-a6f50503b909"). InnerVolumeSpecName "kube-api-access-z8wbj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:00:15.656607 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:15.656482    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8wbj\" (UniqueName: \"kubernetes.io/projected/a1a17f30-378e-4b5c-81c4-a6f50503b909-kube-api-access-z8wbj\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:16.272728 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:16.272698    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c67de2-29890860-bpnn6"
Oct 31 13:00:16.272728 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:16.272718    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c67de2-29890860-bpnn6" event={"ID":"a1a17f30-378e-4b5c-81c4-a6f50503b909","Type":"ContainerDied","Data":"ca322abb4a51e9d21274ee5310a5671b93b086bcecc4c39e44faae5fc1b29430"}
Oct 31 13:00:16.273016 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:16.272785    5060 scope.go:122] "RemoveContainer" containerID="2b6534d2398bb00cb1d9069a3e7c8b9ccc54b66bd658aa9e6046cc5578b1e36e"
Oct 31 13:00:16.274602 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:16.274573    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-2d6201-29890860-8p4sf" event={"ID":"9a318cfb-1970-4ec2-8488-07de8420ba8e","Type":"ContainerDied","Data":"1409d0deb873140f2a9c7bc11131dae26d91d4d1fb975abf25bc09743bfe3d6d"}
Oct 31 13:00:16.274602 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:16.274600    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-2d6201-29890860-8p4sf"
Oct 31 13:00:16.285252 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:16.285228    5060 scope.go:122] "RemoveContainer" containerID="4cc9a7768d074bd391ced05082ee5b4be8a70ef6fa1e21c295a8599b32016e53"
Oct 31 13:00:16.318958 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:16.318930    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c67de2-29890860-bpnn6"]
Oct 31 13:00:16.328633 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:16.328605    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c67de2-29890860-bpnn6"]
Oct 31 13:00:16.350341 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:16.350313    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-2d6201-29890860-8p4sf"]
Oct 31 13:00:16.359473 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:16.359449    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-2d6201-29890860-8p4sf"]
Oct 31 13:00:17.050445 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:17.050415    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a318cfb-1970-4ec2-8488-07de8420ba8e" path="/var/lib/kubelet/pods/9a318cfb-1970-4ec2-8488-07de8420ba8e/volumes"
Oct 31 13:00:17.050858 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:17.050843    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1a17f30-378e-4b5c-81c4-a6f50503b909" path="/var/lib/kubelet/pods/a1a17f30-378e-4b5c-81c4-a6f50503b909/volumes"
Oct 31 13:00:17.051154 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:17.051140    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-55360e0c-vhnxl"]
Oct 31 13:00:17.051193 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:17.051169    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-55360e0c-vhnxl"]
Oct 31 13:00:17.681474 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:17.681434    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-8faad6-openshift-default-9ff4fdc49-fc7fk"]
Oct 31 13:00:17.682040 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:17.681728    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-8faad6-openshift-default-9ff4fdc49-fc7fk" podUID="4d1ddd1b-8dc4-4706-911f-18376b4b2b68" containerName="istio-proxy" containerID="cri-o://e1fd78abfe67144fdad2fa06e6ca38520bc79562bf27dcd46d0c0565a00f842c" gracePeriod=30
Oct 31 13:00:18.023268 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:18.023241    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f231812-xgbw7"]
Oct 31 13:00:18.026244 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:18.026215    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f231812-xgbw7"]
Oct 31 13:00:18.331455 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:18.331381    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-c67de2-px6f4"]
Oct 31 13:00:18.331877 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:18.331863    5060 cpu_manager.go:414] "containerMap: removing container" podUID="fd6cc1a2-16d5-4e08-af17-6dceac1af4a1" containerName="cleanup"
Oct 31 13:00:18.331928 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:18.331878    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd6cc1a2-16d5-4e08-af17-6dceac1af4a1" containerName="cleanup"
Oct 31 13:00:18.331928 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:18.331888    5060 cpu_manager.go:414] "containerMap: removing container" podUID="a72336d5-83db-484a-a300-de68dc5a37c4" containerName="istio-proxy"
Oct 31 13:00:18.331928 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:18.331893    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a72336d5-83db-484a-a300-de68dc5a37c4" containerName="istio-proxy"
Oct 31 13:00:18.331928 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:18.331901    5060 cpu_manager.go:414] "containerMap: removing container" podUID="a1a17f30-378e-4b5c-81c4-a6f50503b909" containerName="cleanup"
Oct 31 13:00:18.331928 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:18.331906    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1a17f30-378e-4b5c-81c4-a6f50503b909" containerName="cleanup"
Oct 31 13:00:18.331928 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:18.331919    5060 cpu_manager.go:414] "containerMap: removing container" podUID="9a318cfb-1970-4ec2-8488-07de8420ba8e" containerName="cleanup"
Oct 31 13:00:18.331928 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:18.331924    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a318cfb-1970-4ec2-8488-07de8420ba8e" containerName="cleanup"
Oct 31 13:00:18.332184 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:18.331934    5060 cpu_manager.go:414] "containerMap: removing container" podUID="fcde778f-8f1a-4c1a-a497-6c66733d54ff" containerName="istio-proxy"
Oct 31 13:00:18.332184 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:18.331939    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fcde778f-8f1a-4c1a-a497-6c66733d54ff" containerName="istio-proxy"
Oct 31 13:00:18.332184 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:18.331945    5060 cpu_manager.go:414] "containerMap: removing container" podUID="a1a17f30-378e-4b5c-81c4-a6f50503b909" containerName="cleanup"
Oct 31 13:00:18.332184 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:18.331949    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1a17f30-378e-4b5c-81c4-a6f50503b909" containerName="cleanup"
Oct 31 13:00:18.332184 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:18.331958    5060 cpu_manager.go:414] "containerMap: removing container" podUID="9a318cfb-1970-4ec2-8488-07de8420ba8e" containerName="cleanup"
Oct 31 13:00:18.332184 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:18.331964    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a318cfb-1970-4ec2-8488-07de8420ba8e" containerName="cleanup"
Oct 31 13:00:18.332184 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:18.331972    5060 cpu_manager.go:414] "containerMap: removing container" podUID="fd6cc1a2-16d5-4e08-af17-6dceac1af4a1" containerName="cleanup"
Oct 31 13:00:18.332184 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:18.331976    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd6cc1a2-16d5-4e08-af17-6dceac1af4a1" containerName="cleanup"
Oct 31 13:00:18.332184 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:18.331982    5060 cpu_manager.go:414] "containerMap: removing container" podUID="fcde778f-8f1a-4c1a-a497-6c66733d54ff" containerName="istio-proxy"
Oct 31 13:00:18.332184 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:18.331987    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fcde778f-8f1a-4c1a-a497-6c66733d54ff" containerName="istio-proxy"
Oct 31 13:00:18.332184 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:18.332002    5060 cpu_manager.go:414] "containerMap: removing container" podUID="aae46d99-cc6d-4be9-868b-8c4dda15a7a7" containerName="istio-proxy"
Oct 31 13:00:18.332184 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:18.332006    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aae46d99-cc6d-4be9-868b-8c4dda15a7a7" containerName="istio-proxy"
Oct 31 13:00:18.332184 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:18.332021    5060 cpu_manager.go:414] "containerMap: removing container" podUID="85cd1407-54f9-493a-b8fd-9e638a6615a5" containerName="payload-processing"
Oct 31 13:00:18.332184 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:18.332026    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85cd1407-54f9-493a-b8fd-9e638a6615a5" containerName="payload-processing"
Oct 31 13:00:18.332184 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:18.332094    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd6cc1a2-16d5-4e08-af17-6dceac1af4a1" containerName="cleanup"
Oct 31 13:00:18.332184 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:18.332101    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="fcde778f-8f1a-4c1a-a497-6c66733d54ff" containerName="istio-proxy"
Oct 31 13:00:18.332184 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:18.332108    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="aae46d99-cc6d-4be9-868b-8c4dda15a7a7" containerName="istio-proxy"
Oct 31 13:00:18.332184 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:18.332116    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="85cd1407-54f9-493a-b8fd-9e638a6615a5" containerName="payload-processing"
Oct 31 13:00:18.332184 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:18.332125    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd6cc1a2-16d5-4e08-af17-6dceac1af4a1" containerName="cleanup"
Oct 31 13:00:18.332184 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:18.332131    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="a72336d5-83db-484a-a300-de68dc5a37c4" containerName="istio-proxy"
Oct 31 13:00:18.332184 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:18.332137    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1a17f30-378e-4b5c-81c4-a6f50503b909" containerName="cleanup"
Oct 31 13:00:18.332184 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:18.332142    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a318cfb-1970-4ec2-8488-07de8420ba8e" containerName="cleanup"
Oct 31 13:00:18.332184 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:18.332148    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1a17f30-378e-4b5c-81c4-a6f50503b909" containerName="cleanup"
Oct 31 13:00:18.332184 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:18.332155    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a318cfb-1970-4ec2-8488-07de8420ba8e" containerName="cleanup"
Oct 31 13:00:18.337066 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:18.337047    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-c67de2-px6f4"
Oct 31 13:00:18.340856 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:18.340650    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 13:00:18.340856 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:18.340742    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 13:00:18.345291 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:18.345257    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-c67de2-px6f4"]
Oct 31 13:00:18.359809 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:18.359775    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7wsmb\""
Oct 31 13:00:18.359964 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:18.359827    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 13:00:18.384597 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:18.384561    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsgzh\" (UniqueName: \"kubernetes.io/projected/cc6253cf-8a68-46d7-ba90-6402318088d8-kube-api-access-qsgzh\") pod \"maas-api-revoke-keys-e2e-worker-w4-c67de2-px6f4\" (UID: \"cc6253cf-8a68-46d7-ba90-6402318088d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-c67de2-px6f4"
Oct 31 13:00:18.384753 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:18.384619    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc6253cf-8a68-46d7-ba90-6402318088d8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-c67de2-px6f4\" (UID: \"cc6253cf-8a68-46d7-ba90-6402318088d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-c67de2-px6f4"
Oct 31 13:00:18.384821 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:18.384775    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc6253cf-8a68-46d7-ba90-6402318088d8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-c67de2-px6f4\" (UID: \"cc6253cf-8a68-46d7-ba90-6402318088d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-c67de2-px6f4"
Oct 31 13:00:18.485817 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:18.485781    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc6253cf-8a68-46d7-ba90-6402318088d8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-c67de2-px6f4\" (UID: \"cc6253cf-8a68-46d7-ba90-6402318088d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-c67de2-px6f4"
Oct 31 13:00:18.486020 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:18.485896    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qsgzh\" (UniqueName: \"kubernetes.io/projected/cc6253cf-8a68-46d7-ba90-6402318088d8-kube-api-access-qsgzh\") pod \"maas-api-revoke-keys-e2e-worker-w4-c67de2-px6f4\" (UID: \"cc6253cf-8a68-46d7-ba90-6402318088d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-c67de2-px6f4"
Oct 31 13:00:18.486020 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:18.485938    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc6253cf-8a68-46d7-ba90-6402318088d8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-c67de2-px6f4\" (UID: \"cc6253cf-8a68-46d7-ba90-6402318088d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-c67de2-px6f4"
Oct 31 13:00:18.486703 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:18.486680    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc6253cf-8a68-46d7-ba90-6402318088d8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-c67de2-px6f4\" (UID: \"cc6253cf-8a68-46d7-ba90-6402318088d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-c67de2-px6f4"
Oct 31 13:00:18.488477 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:18.488449    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc6253cf-8a68-46d7-ba90-6402318088d8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-c67de2-px6f4\" (UID: \"cc6253cf-8a68-46d7-ba90-6402318088d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-c67de2-px6f4"
Oct 31 13:00:18.498451 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:18.498426    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsgzh\" (UniqueName: \"kubernetes.io/projected/cc6253cf-8a68-46d7-ba90-6402318088d8-kube-api-access-qsgzh\") pod \"maas-api-revoke-keys-e2e-worker-w4-c67de2-px6f4\" (UID: \"cc6253cf-8a68-46d7-ba90-6402318088d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-c67de2-px6f4"
Oct 31 13:00:18.653995 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:18.653908    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-c67de2-px6f4"
Oct 31 13:00:18.939686 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:18.939655    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-c67de2-px6f4"]
Oct 31 13:00:18.940914 ip-10-0-130-190 kubenswrapper[5060]: W1031 13:00:18.940880    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc6253cf_8a68_46d7_ba90_6402318088d8.slice/crio-35f33e01f1933efb2571cbe45d4ea6a0bb2016364ff5b02e788b6451060fb8e2 WatchSource:0}: Error finding container 35f33e01f1933efb2571cbe45d4ea6a0bb2016364ff5b02e788b6451060fb8e2: Status 404 returned error can't find the container with id 35f33e01f1933efb2571cbe45d4ea6a0bb2016364ff5b02e788b6451060fb8e2
Oct 31 13:00:19.048210 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:19.048172    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00572ed5-fa0b-4297-8087-24d4d632f5fb" path="/var/lib/kubelet/pods/00572ed5-fa0b-4297-8087-24d4d632f5fb/volumes"
Oct 31 13:00:19.048718 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:19.048699    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e616d24-af13-4f8f-93f0-bae103fceca8" path="/var/lib/kubelet/pods/3e616d24-af13-4f8f-93f0-bae103fceca8/volumes"
Oct 31 13:00:19.307154 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:19.307113    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-c67de2-px6f4" event={"ID":"cc6253cf-8a68-46d7-ba90-6402318088d8","Type":"ContainerStarted","Data":"35f33e01f1933efb2571cbe45d4ea6a0bb2016364ff5b02e788b6451060fb8e2"}
Oct 31 13:00:20.317590 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:20.317549    5060 generic.go:356] "Generic (PLEG): container finished" podID="cc6253cf-8a68-46d7-ba90-6402318088d8" containerID="0a6da146988fd4d3c56f5c0c02e9fbc13603f4f47d5d91edf450faad7a84a23f" exitCode=0
Oct 31 13:00:20.318040 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:20.317643    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-c67de2-px6f4" event={"ID":"cc6253cf-8a68-46d7-ba90-6402318088d8","Type":"ContainerDied","Data":"0a6da146988fd4d3c56f5c0c02e9fbc13603f4f47d5d91edf450faad7a84a23f"}
Oct 31 13:00:21.476565 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:21.476537    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-c67de2-px6f4"
Oct 31 13:00:21.515812 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:21.515783    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc6253cf-8a68-46d7-ba90-6402318088d8-kube-api-access-qsgzh\" (UniqueName: \"kubernetes.io/projected/cc6253cf-8a68-46d7-ba90-6402318088d8-kube-api-access-qsgzh\") pod \"cc6253cf-8a68-46d7-ba90-6402318088d8\" (UID: \"cc6253cf-8a68-46d7-ba90-6402318088d8\") "
Oct 31 13:00:21.515974 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:21.515865    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc6253cf-8a68-46d7-ba90-6402318088d8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc6253cf-8a68-46d7-ba90-6402318088d8-maas-api-openshift-ca\") pod \"cc6253cf-8a68-46d7-ba90-6402318088d8\" (UID: \"cc6253cf-8a68-46d7-ba90-6402318088d8\") "
Oct 31 13:00:21.515974 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:21.515927    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cc6253cf-8a68-46d7-ba90-6402318088d8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc6253cf-8a68-46d7-ba90-6402318088d8-maas-api-xks-ca\") pod \"cc6253cf-8a68-46d7-ba90-6402318088d8\" (UID: \"cc6253cf-8a68-46d7-ba90-6402318088d8\") "
Oct 31 13:00:21.516250 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:21.516224    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc6253cf-8a68-46d7-ba90-6402318088d8-maas-api-openshift-ca" pod "cc6253cf-8a68-46d7-ba90-6402318088d8" (UID: "cc6253cf-8a68-46d7-ba90-6402318088d8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:00:21.518280 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:21.518242    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc6253cf-8a68-46d7-ba90-6402318088d8-maas-api-xks-ca" pod "cc6253cf-8a68-46d7-ba90-6402318088d8" (UID: "cc6253cf-8a68-46d7-ba90-6402318088d8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:00:21.518445 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:21.518423    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc6253cf-8a68-46d7-ba90-6402318088d8-kube-api-access-qsgzh" pod "cc6253cf-8a68-46d7-ba90-6402318088d8" (UID: "cc6253cf-8a68-46d7-ba90-6402318088d8"). InnerVolumeSpecName "kube-api-access-qsgzh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:00:21.616951 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:21.616876    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc6253cf-8a68-46d7-ba90-6402318088d8-maas-api-xks-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:21.616951 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:21.616906    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qsgzh\" (UniqueName: \"kubernetes.io/projected/cc6253cf-8a68-46d7-ba90-6402318088d8-kube-api-access-qsgzh\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:21.616951 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:21.616915    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc6253cf-8a68-46d7-ba90-6402318088d8-maas-api-openshift-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:22.339286 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:22.339239    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-c67de2-px6f4" event={"ID":"cc6253cf-8a68-46d7-ba90-6402318088d8","Type":"ContainerDied","Data":"35f33e01f1933efb2571cbe45d4ea6a0bb2016364ff5b02e788b6451060fb8e2"}
Oct 31 13:00:22.339521 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:22.339372    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="35f33e01f1933efb2571cbe45d4ea6a0bb2016364ff5b02e788b6451060fb8e2"
Oct 31 13:00:22.339521 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:22.339473    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-c67de2-px6f4"
Oct 31 13:00:22.942182 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:22.942155    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-8faad6-openshift-default-9ff4fdc49-fc7fk"
Oct 31 13:00:23.032649 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:23.032616    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4d1ddd1b-8dc4-4706-911f-18376b4b2b68-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d1ddd1b-8dc4-4706-911f-18376b4b2b68-istio-podinfo\") pod \"4d1ddd1b-8dc4-4706-911f-18376b4b2b68\" (UID: \"4d1ddd1b-8dc4-4706-911f-18376b4b2b68\") "
Oct 31 13:00:23.032869 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:23.032658    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d1ddd1b-8dc4-4706-911f-18376b4b2b68-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d1ddd1b-8dc4-4706-911f-18376b4b2b68-istiod-ca-cert\") pod \"4d1ddd1b-8dc4-4706-911f-18376b4b2b68\" (UID: \"4d1ddd1b-8dc4-4706-911f-18376b4b2b68\") "
Oct 31 13:00:23.032869 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:23.032703    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d1ddd1b-8dc4-4706-911f-18376b4b2b68-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d1ddd1b-8dc4-4706-911f-18376b4b2b68-credential-socket\") pod \"4d1ddd1b-8dc4-4706-911f-18376b4b2b68\" (UID: \"4d1ddd1b-8dc4-4706-911f-18376b4b2b68\") "
Oct 31 13:00:23.032869 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:23.032773    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d1ddd1b-8dc4-4706-911f-18376b4b2b68-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d1ddd1b-8dc4-4706-911f-18376b4b2b68-istio-data\") pod \"4d1ddd1b-8dc4-4706-911f-18376b4b2b68\" (UID: \"4d1ddd1b-8dc4-4706-911f-18376b4b2b68\") "
Oct 31 13:00:23.032869 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:23.032814    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d1ddd1b-8dc4-4706-911f-18376b4b2b68-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d1ddd1b-8dc4-4706-911f-18376b4b2b68-workload-certs\") pod \"4d1ddd1b-8dc4-4706-911f-18376b4b2b68\" (UID: \"4d1ddd1b-8dc4-4706-911f-18376b4b2b68\") "
Oct 31 13:00:23.032869 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:23.032837    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d1ddd1b-8dc4-4706-911f-18376b4b2b68-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d1ddd1b-8dc4-4706-911f-18376b4b2b68-istio-envoy\") pod \"4d1ddd1b-8dc4-4706-911f-18376b4b2b68\" (UID: \"4d1ddd1b-8dc4-4706-911f-18376b4b2b68\") "
Oct 31 13:00:23.032869 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:23.032863    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d1ddd1b-8dc4-4706-911f-18376b4b2b68-istio-token\" (UniqueName: \"kubernetes.io/projected/4d1ddd1b-8dc4-4706-911f-18376b4b2b68-istio-token\") pod \"4d1ddd1b-8dc4-4706-911f-18376b4b2b68\" (UID: \"4d1ddd1b-8dc4-4706-911f-18376b4b2b68\") "
Oct 31 13:00:23.033270 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:23.032927    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d1ddd1b-8dc4-4706-911f-18376b4b2b68-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d1ddd1b-8dc4-4706-911f-18376b4b2b68-workload-socket\") pod \"4d1ddd1b-8dc4-4706-911f-18376b4b2b68\" (UID: \"4d1ddd1b-8dc4-4706-911f-18376b4b2b68\") "
Oct 31 13:00:23.033270 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:23.032969    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d1ddd1b-8dc4-4706-911f-18376b4b2b68-kube-api-access-bnq7x\" (UniqueName: \"kubernetes.io/projected/4d1ddd1b-8dc4-4706-911f-18376b4b2b68-kube-api-access-bnq7x\") pod \"4d1ddd1b-8dc4-4706-911f-18376b4b2b68\" (UID: \"4d1ddd1b-8dc4-4706-911f-18376b4b2b68\") "
Oct 31 13:00:23.033270 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:23.033032    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d1ddd1b-8dc4-4706-911f-18376b4b2b68-credential-socket" pod "4d1ddd1b-8dc4-4706-911f-18376b4b2b68" (UID: "4d1ddd1b-8dc4-4706-911f-18376b4b2b68"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:00:23.033270 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:23.033116    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d1ddd1b-8dc4-4706-911f-18376b4b2b68-istiod-ca-cert" pod "4d1ddd1b-8dc4-4706-911f-18376b4b2b68" (UID: "4d1ddd1b-8dc4-4706-911f-18376b4b2b68"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:00:23.034669 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:23.033449    5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d1ddd1b-8dc4-4706-911f-18376b4b2b68-istiod-ca-cert\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:23.034669 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:23.033474    5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d1ddd1b-8dc4-4706-911f-18376b4b2b68-credential-socket\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:23.034669 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:23.033778    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d1ddd1b-8dc4-4706-911f-18376b4b2b68-istio-data" pod "4d1ddd1b-8dc4-4706-911f-18376b4b2b68" (UID: "4d1ddd1b-8dc4-4706-911f-18376b4b2b68"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:00:23.034669 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:23.033977    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d1ddd1b-8dc4-4706-911f-18376b4b2b68-workload-socket" pod "4d1ddd1b-8dc4-4706-911f-18376b4b2b68" (UID: "4d1ddd1b-8dc4-4706-911f-18376b4b2b68"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:00:23.034669 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:23.034157    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d1ddd1b-8dc4-4706-911f-18376b4b2b68-workload-certs" pod "4d1ddd1b-8dc4-4706-911f-18376b4b2b68" (UID: "4d1ddd1b-8dc4-4706-911f-18376b4b2b68"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:00:23.035684 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:23.035657    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4d1ddd1b-8dc4-4706-911f-18376b4b2b68-istio-podinfo" pod "4d1ddd1b-8dc4-4706-911f-18376b4b2b68" (UID: "4d1ddd1b-8dc4-4706-911f-18376b4b2b68"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 13:00:23.037277 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:23.037249    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d1ddd1b-8dc4-4706-911f-18376b4b2b68-istio-envoy" pod "4d1ddd1b-8dc4-4706-911f-18376b4b2b68" (UID: "4d1ddd1b-8dc4-4706-911f-18376b4b2b68"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:00:23.038433 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:23.038408    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d1ddd1b-8dc4-4706-911f-18376b4b2b68-kube-api-access-bnq7x" pod "4d1ddd1b-8dc4-4706-911f-18376b4b2b68" (UID: "4d1ddd1b-8dc4-4706-911f-18376b4b2b68"). InnerVolumeSpecName "kube-api-access-bnq7x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:00:23.038854 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:23.038819    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d1ddd1b-8dc4-4706-911f-18376b4b2b68-istio-token" pod "4d1ddd1b-8dc4-4706-911f-18376b4b2b68" (UID: "4d1ddd1b-8dc4-4706-911f-18376b4b2b68"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:00:23.134240 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:23.134201    5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d1ddd1b-8dc4-4706-911f-18376b4b2b68-istio-podinfo\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:23.134240 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:23.134238    5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d1ddd1b-8dc4-4706-911f-18376b4b2b68-istio-data\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:23.134542 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:23.134253    5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d1ddd1b-8dc4-4706-911f-18376b4b2b68-workload-certs\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:23.134542 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:23.134267    5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d1ddd1b-8dc4-4706-911f-18376b4b2b68-istio-envoy\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:23.134542 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:23.134281    5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d1ddd1b-8dc4-4706-911f-18376b4b2b68-istio-token\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:23.134542 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:23.134295    5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d1ddd1b-8dc4-4706-911f-18376b4b2b68-workload-socket\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:23.134542 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:23.134309    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bnq7x\" (UniqueName: \"kubernetes.io/projected/4d1ddd1b-8dc4-4706-911f-18376b4b2b68-kube-api-access-bnq7x\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:23.152837 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:23.152802    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-c67de2/simulator-w4-c67de2-kserve-868679cf9b-gqjbx"]
Oct 31 13:00:23.348533 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:23.348415    5060 generic.go:356] "Generic (PLEG): container finished" podID="4d1ddd1b-8dc4-4706-911f-18376b4b2b68" containerID="e1fd78abfe67144fdad2fa06e6ca38520bc79562bf27dcd46d0c0565a00f842c" exitCode=0
Oct 31 13:00:23.348533 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:23.348468    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-8faad6-openshift-default-9ff4fdc49-fc7fk"
Oct 31 13:00:23.348744 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:23.348528    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-8faad6-openshift-default-9ff4fdc49-fc7fk" event={"ID":"4d1ddd1b-8dc4-4706-911f-18376b4b2b68","Type":"ContainerDied","Data":"e1fd78abfe67144fdad2fa06e6ca38520bc79562bf27dcd46d0c0565a00f842c"}
Oct 31 13:00:23.348744 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:23.348585    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-8faad6-openshift-default-9ff4fdc49-fc7fk" event={"ID":"4d1ddd1b-8dc4-4706-911f-18376b4b2b68","Type":"ContainerDied","Data":"b51a18b2b9cb14b4b74d2d6477a11fd1deb5fe4dc1f6fd729c471c8687e857dd"}
Oct 31 13:00:23.348744 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:23.348614    5060 scope.go:122] "RemoveContainer" containerID="e1fd78abfe67144fdad2fa06e6ca38520bc79562bf27dcd46d0c0565a00f842c"
Oct 31 13:00:23.360422 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:23.360401    5060 scope.go:122] "RemoveContainer" containerID="d7d9f76e56891d8d11d5191df4ed997977d680b6bdc8f71963300232ed4a9340"
Oct 31 13:00:23.370652 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:23.370627    5060 scope.go:122] "RemoveContainer" containerID="e1fd78abfe67144fdad2fa06e6ca38520bc79562bf27dcd46d0c0565a00f842c"
Oct 31 13:00:23.370923 ip-10-0-130-190 kubenswrapper[5060]: E1031 13:00:23.370902    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1fd78abfe67144fdad2fa06e6ca38520bc79562bf27dcd46d0c0565a00f842c\": container with ID starting with e1fd78abfe67144fdad2fa06e6ca38520bc79562bf27dcd46d0c0565a00f842c not found: ID does not exist" containerID="e1fd78abfe67144fdad2fa06e6ca38520bc79562bf27dcd46d0c0565a00f842c"
Oct 31 13:00:23.371007 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:23.370936    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1fd78abfe67144fdad2fa06e6ca38520bc79562bf27dcd46d0c0565a00f842c"} err="failed to get container status \"e1fd78abfe67144fdad2fa06e6ca38520bc79562bf27dcd46d0c0565a00f842c\": rpc error: code = NotFound desc = could not find container \"e1fd78abfe67144fdad2fa06e6ca38520bc79562bf27dcd46d0c0565a00f842c\": container with ID starting with e1fd78abfe67144fdad2fa06e6ca38520bc79562bf27dcd46d0c0565a00f842c not found: ID does not exist"
Oct 31 13:00:23.371007 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:23.370960    5060 scope.go:122] "RemoveContainer" containerID="d7d9f76e56891d8d11d5191df4ed997977d680b6bdc8f71963300232ed4a9340"
Oct 31 13:00:23.371199 ip-10-0-130-190 kubenswrapper[5060]: E1031 13:00:23.371179    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7d9f76e56891d8d11d5191df4ed997977d680b6bdc8f71963300232ed4a9340\": container with ID starting with d7d9f76e56891d8d11d5191df4ed997977d680b6bdc8f71963300232ed4a9340 not found: ID does not exist" containerID="d7d9f76e56891d8d11d5191df4ed997977d680b6bdc8f71963300232ed4a9340"
Oct 31 13:00:23.371293 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:23.371205    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7d9f76e56891d8d11d5191df4ed997977d680b6bdc8f71963300232ed4a9340"} err="failed to get container status \"d7d9f76e56891d8d11d5191df4ed997977d680b6bdc8f71963300232ed4a9340\": rpc error: code = NotFound desc = could not find container \"d7d9f76e56891d8d11d5191df4ed997977d680b6bdc8f71963300232ed4a9340\": container with ID starting with d7d9f76e56891d8d11d5191df4ed997977d680b6bdc8f71963300232ed4a9340 not found: ID does not exist"
Oct 31 13:00:23.382481 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:23.382433    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-8faad6-openshift-default-9ff4fdc49-fc7fk"]
Oct 31 13:00:23.385682 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:23.385659    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-8faad6-openshift-default-9ff4fdc49-fc7fk"]
Oct 31 13:00:25.052202 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:25.052162    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d1ddd1b-8dc4-4706-911f-18376b4b2b68" path="/var/lib/kubelet/pods/4d1ddd1b-8dc4-4706-911f-18376b4b2b68/volumes"
Oct 31 13:00:25.188574 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:25.188542    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-l8n27"]
Oct 31 13:00:25.188763 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:25.188730    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-l8n27" podUID="c5c79f52-3049-4611-a935-ef2f4eb5e0bd" containerName="istio-proxy" containerID="cri-o://996dfd8501bc25b0bbf2d5acaacca3f2bb6ca231454aee25616f89192cb48a5d" gracePeriod=30
Oct 31 13:00:26.056508 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:26.056456    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b30aae92-c2vmq"]
Oct 31 13:00:26.064423 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:26.064389    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b30aae92-c2vmq"]
Oct 31 13:00:26.285950 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:26.285918    5060 scope.go:122] "RemoveContainer" containerID="56971a4a3a477e8095fef84a7d644d486e7fda1a8ac07cd046f5df7327080b1c"
Oct 31 13:00:26.296308 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:26.296280    5060 scope.go:122] "RemoveContainer" containerID="2fba625dcc73a5ede98b5f28994db8578f0f4e7593068688c7c250c220fee059"
Oct 31 13:00:26.307332 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:26.307310    5060 scope.go:122] "RemoveContainer" containerID="f1b54df8213f1c477052f21e28ec5fc98f2842f44810408a13c4b1bc3c275e07"
Oct 31 13:00:26.316437 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:26.316419    5060 scope.go:122] "RemoveContainer" containerID="955e669d2572d6c92e972dc8dfc208d142fa4331bbbacdf311993f1a752e35c9"
Oct 31 13:00:26.325576 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:26.325553    5060 scope.go:122] "RemoveContainer" containerID="223c129273c47e87f1c3e2d957c390796115b371212736d6cecd42833b9091a7"
Oct 31 13:00:26.334287 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:26.334268    5060 scope.go:122] "RemoveContainer" containerID="342a263e5f70b61f3edd4c3c8dbeb77e2de25a21c0fd87613713fbfbf5e5dbea"
Oct 31 13:00:26.343081 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:26.343065    5060 scope.go:122] "RemoveContainer" containerID="d7bca43b9b15ecff60de5ccd7e6e5372115f4a5f0704486e24f8c2ca77f7b331"
Oct 31 13:00:27.047800 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:27.047761    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68c38c1d-abc4-44c0-9280-c378cb1b4590" path="/var/lib/kubelet/pods/68c38c1d-abc4-44c0-9280-c378cb1b4590/volumes"
Oct 31 13:00:28.453596 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:28.453563    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-c67de2-7d5dc5649c-62vgx"]
Oct 31 13:00:28.474076 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:28.474036    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-c67de2-5656c49759-8llqd"]
Oct 31 13:00:29.163313 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:29.163282    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4918fc6d-zr525"]
Oct 31 13:00:29.163821 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:29.163806    5060 cpu_manager.go:414] "containerMap: removing container" podUID="4d1ddd1b-8dc4-4706-911f-18376b4b2b68" containerName="istio-proxy"
Oct 31 13:00:29.163879 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:29.163822    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d1ddd1b-8dc4-4706-911f-18376b4b2b68" containerName="istio-proxy"
Oct 31 13:00:29.163879 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:29.163835    5060 cpu_manager.go:414] "containerMap: removing container" podUID="4d1ddd1b-8dc4-4706-911f-18376b4b2b68" containerName="istio-proxy"
Oct 31 13:00:29.163879 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:29.163841    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d1ddd1b-8dc4-4706-911f-18376b4b2b68" containerName="istio-proxy"
Oct 31 13:00:29.163879 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:29.163853    5060 cpu_manager.go:414] "containerMap: removing container" podUID="cc6253cf-8a68-46d7-ba90-6402318088d8" containerName="revoke-keys"
Oct 31 13:00:29.163879 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:29.163858    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc6253cf-8a68-46d7-ba90-6402318088d8" containerName="revoke-keys"
Oct 31 13:00:29.163879 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:29.163871    5060 cpu_manager.go:414] "containerMap: removing container" podUID="a1a17f30-378e-4b5c-81c4-a6f50503b909" containerName="cleanup"
Oct 31 13:00:29.163879 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:29.163876    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1a17f30-378e-4b5c-81c4-a6f50503b909" containerName="cleanup"
Oct 31 13:00:29.164112 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:29.163948    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d1ddd1b-8dc4-4706-911f-18376b4b2b68" containerName="istio-proxy"
Oct 31 13:00:29.164112 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:29.163959    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d1ddd1b-8dc4-4706-911f-18376b4b2b68" containerName="istio-proxy"
Oct 31 13:00:29.164112 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:29.163966    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc6253cf-8a68-46d7-ba90-6402318088d8" containerName="revoke-keys"
Oct 31 13:00:29.164112 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:29.163976    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1a17f30-378e-4b5c-81c4-a6f50503b909" containerName="cleanup"
Oct 31 13:00:29.164112 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:29.163983    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="fcde778f-8f1a-4c1a-a497-6c66733d54ff" containerName="istio-proxy"
Oct 31 13:00:29.170688 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:29.170664    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4918fc6d-zr525"
Oct 31 13:00:29.176523 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:29.174828    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 13:00:29.176523 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:29.174930    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 13:00:29.176523 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:29.175102    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7wsmb\""
Oct 31 13:00:29.176523 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:29.175157    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 13:00:29.178340 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:29.178291    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4918fc6d-zr525"]
Oct 31 13:00:29.299960 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:29.299924    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ncsg\" (UniqueName: \"kubernetes.io/projected/a5b0a48d-4c1d-4fef-a172-8e608048f155-kube-api-access-4ncsg\") pod \"maas-api-revoke-sub-4918fc6d-zr525\" (UID: \"a5b0a48d-4c1d-4fef-a172-8e608048f155\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4918fc6d-zr525"
Oct 31 13:00:29.300154 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:29.300073    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5b0a48d-4c1d-4fef-a172-8e608048f155-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4918fc6d-zr525\" (UID: \"a5b0a48d-4c1d-4fef-a172-8e608048f155\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4918fc6d-zr525"
Oct 31 13:00:29.300154 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:29.300123    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5b0a48d-4c1d-4fef-a172-8e608048f155-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4918fc6d-zr525\" (UID: \"a5b0a48d-4c1d-4fef-a172-8e608048f155\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4918fc6d-zr525"
Oct 31 13:00:29.401177 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:29.401145    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ncsg\" (UniqueName: \"kubernetes.io/projected/a5b0a48d-4c1d-4fef-a172-8e608048f155-kube-api-access-4ncsg\") pod \"maas-api-revoke-sub-4918fc6d-zr525\" (UID: \"a5b0a48d-4c1d-4fef-a172-8e608048f155\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4918fc6d-zr525"
Oct 31 13:00:29.401368 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:29.401233    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5b0a48d-4c1d-4fef-a172-8e608048f155-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4918fc6d-zr525\" (UID: \"a5b0a48d-4c1d-4fef-a172-8e608048f155\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4918fc6d-zr525"
Oct 31 13:00:29.401368 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:29.401269    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5b0a48d-4c1d-4fef-a172-8e608048f155-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4918fc6d-zr525\" (UID: \"a5b0a48d-4c1d-4fef-a172-8e608048f155\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4918fc6d-zr525"
Oct 31 13:00:29.402088 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:29.402059    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5b0a48d-4c1d-4fef-a172-8e608048f155-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4918fc6d-zr525\" (UID: \"a5b0a48d-4c1d-4fef-a172-8e608048f155\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4918fc6d-zr525"
Oct 31 13:00:29.403983 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:29.403955    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5b0a48d-4c1d-4fef-a172-8e608048f155-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4918fc6d-zr525\" (UID: \"a5b0a48d-4c1d-4fef-a172-8e608048f155\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4918fc6d-zr525"
Oct 31 13:00:29.413749 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:29.413687    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ncsg\" (UniqueName: \"kubernetes.io/projected/a5b0a48d-4c1d-4fef-a172-8e608048f155-kube-api-access-4ncsg\") pod \"maas-api-revoke-sub-4918fc6d-zr525\" (UID: \"a5b0a48d-4c1d-4fef-a172-8e608048f155\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4918fc6d-zr525"
Oct 31 13:00:29.489142 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:29.489096    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4918fc6d-zr525"
Oct 31 13:00:29.776362 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:29.776324    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4918fc6d-zr525"]
Oct 31 13:00:29.779950 ip-10-0-130-190 kubenswrapper[5060]: W1031 13:00:29.779914    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5b0a48d_4c1d_4fef_a172_8e608048f155.slice/crio-56ff2143ea20a13342a38b56e4a0b7508d72ea0546a3ee2e7046b4442fe029ed WatchSource:0}: Error finding container 56ff2143ea20a13342a38b56e4a0b7508d72ea0546a3ee2e7046b4442fe029ed: Status 404 returned error can't find the container with id 56ff2143ea20a13342a38b56e4a0b7508d72ea0546a3ee2e7046b4442fe029ed
Oct 31 13:00:29.921942 ip-10-0-130-190 kubenswrapper[5060]: E1031 13:00:29.921908    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7phb" containerName="extractor"
Oct 31 13:00:30.424886 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:30.424845    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4918fc6d-zr525" event={"ID":"a5b0a48d-4c1d-4fef-a172-8e608048f155","Type":"ContainerStarted","Data":"633c4a8874cd6c68b1bd9b37c36fa00c02bcf97cf4e27419b166b768dade1bab"}
Oct 31 13:00:30.425070 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:30.424917    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4918fc6d-zr525" event={"ID":"a5b0a48d-4c1d-4fef-a172-8e608048f155","Type":"ContainerStarted","Data":"56ff2143ea20a13342a38b56e4a0b7508d72ea0546a3ee2e7046b4442fe029ed"}
Oct 31 13:00:30.427342 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:30.427315    5060 generic.go:356] "Generic (PLEG): container finished" podID="c5c79f52-3049-4611-a935-ef2f4eb5e0bd" containerID="996dfd8501bc25b0bbf2d5acaacca3f2bb6ca231454aee25616f89192cb48a5d" exitCode=0
Oct 31 13:00:30.427508 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:30.427364    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-l8n27" event={"ID":"c5c79f52-3049-4611-a935-ef2f4eb5e0bd","Type":"ContainerDied","Data":"996dfd8501bc25b0bbf2d5acaacca3f2bb6ca231454aee25616f89192cb48a5d"}
Oct 31 13:00:30.446359 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:30.446300    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4918fc6d-zr525" podStartSLOduration=1.446282477 podStartE2EDuration="1.446282477s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:00:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:00:30.442662934 +0000 UTC m=+1993.545730647" watchObservedRunningTime="2026-10-31 13:00:30.446282477 +0000 UTC m=+1993.549350190"
Oct 31 13:00:30.474685 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:30.474660    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-l8n27"
Oct 31 13:00:30.612929 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:30.612899    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5c79f52-3049-4611-a935-ef2f4eb5e0bd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5c79f52-3049-4611-a935-ef2f4eb5e0bd-workload-socket\") pod \"c5c79f52-3049-4611-a935-ef2f4eb5e0bd\" (UID: \"c5c79f52-3049-4611-a935-ef2f4eb5e0bd\") "
Oct 31 13:00:30.613321 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:30.612965    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5c79f52-3049-4611-a935-ef2f4eb5e0bd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5c79f52-3049-4611-a935-ef2f4eb5e0bd-istiod-ca-cert\") pod \"c5c79f52-3049-4611-a935-ef2f4eb5e0bd\" (UID: \"c5c79f52-3049-4611-a935-ef2f4eb5e0bd\") "
Oct 31 13:00:30.613321 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:30.613004    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5c79f52-3049-4611-a935-ef2f4eb5e0bd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5c79f52-3049-4611-a935-ef2f4eb5e0bd-istio-envoy\") pod \"c5c79f52-3049-4611-a935-ef2f4eb5e0bd\" (UID: \"c5c79f52-3049-4611-a935-ef2f4eb5e0bd\") "
Oct 31 13:00:30.613321 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:30.613030    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c5c79f52-3049-4611-a935-ef2f4eb5e0bd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5c79f52-3049-4611-a935-ef2f4eb5e0bd-istio-podinfo\") pod \"c5c79f52-3049-4611-a935-ef2f4eb5e0bd\" (UID: \"c5c79f52-3049-4611-a935-ef2f4eb5e0bd\") "
Oct 31 13:00:30.613321 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:30.613077    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5c79f52-3049-4611-a935-ef2f4eb5e0bd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5c79f52-3049-4611-a935-ef2f4eb5e0bd-credential-socket\") pod \"c5c79f52-3049-4611-a935-ef2f4eb5e0bd\" (UID: \"c5c79f52-3049-4611-a935-ef2f4eb5e0bd\") "
Oct 31 13:00:30.613321 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:30.613131    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5c79f52-3049-4611-a935-ef2f4eb5e0bd-istio-token\" (UniqueName: \"kubernetes.io/projected/c5c79f52-3049-4611-a935-ef2f4eb5e0bd-istio-token\") pod \"c5c79f52-3049-4611-a935-ef2f4eb5e0bd\" (UID: \"c5c79f52-3049-4611-a935-ef2f4eb5e0bd\") "
Oct 31 13:00:30.613321 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:30.613182    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5c79f52-3049-4611-a935-ef2f4eb5e0bd-kube-api-access-nhqld\" (UniqueName: \"kubernetes.io/projected/c5c79f52-3049-4611-a935-ef2f4eb5e0bd-kube-api-access-nhqld\") pod \"c5c79f52-3049-4611-a935-ef2f4eb5e0bd\" (UID: \"c5c79f52-3049-4611-a935-ef2f4eb5e0bd\") "
Oct 31 13:00:30.613321 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:30.613195    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5c79f52-3049-4611-a935-ef2f4eb5e0bd-workload-socket" pod "c5c79f52-3049-4611-a935-ef2f4eb5e0bd" (UID: "c5c79f52-3049-4611-a935-ef2f4eb5e0bd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:00:30.613780 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:30.613359    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5c79f52-3049-4611-a935-ef2f4eb5e0bd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5c79f52-3049-4611-a935-ef2f4eb5e0bd-workload-certs\") pod \"c5c79f52-3049-4611-a935-ef2f4eb5e0bd\" (UID: \"c5c79f52-3049-4611-a935-ef2f4eb5e0bd\") "
Oct 31 13:00:30.613780 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:30.613395    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5c79f52-3049-4611-a935-ef2f4eb5e0bd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5c79f52-3049-4611-a935-ef2f4eb5e0bd-istio-data\") pod \"c5c79f52-3049-4611-a935-ef2f4eb5e0bd\" (UID: \"c5c79f52-3049-4611-a935-ef2f4eb5e0bd\") "
Oct 31 13:00:30.613780 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:30.613449    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5c79f52-3049-4611-a935-ef2f4eb5e0bd-istiod-ca-cert" pod "c5c79f52-3049-4611-a935-ef2f4eb5e0bd" (UID: "c5c79f52-3049-4611-a935-ef2f4eb5e0bd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:00:30.613780 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:30.613458    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5c79f52-3049-4611-a935-ef2f4eb5e0bd-credential-socket" pod "c5c79f52-3049-4611-a935-ef2f4eb5e0bd" (UID: "c5c79f52-3049-4611-a935-ef2f4eb5e0bd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:00:30.613780 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:30.613771    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5c79f52-3049-4611-a935-ef2f4eb5e0bd-workload-certs" pod "c5c79f52-3049-4611-a935-ef2f4eb5e0bd" (UID: "c5c79f52-3049-4611-a935-ef2f4eb5e0bd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:00:30.614055 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:30.613785    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5c79f52-3049-4611-a935-ef2f4eb5e0bd-istio-data" pod "c5c79f52-3049-4611-a935-ef2f4eb5e0bd" (UID: "c5c79f52-3049-4611-a935-ef2f4eb5e0bd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:00:30.614055 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:30.613841    5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5c79f52-3049-4611-a935-ef2f4eb5e0bd-credential-socket\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:30.614055 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:30.613865    5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5c79f52-3049-4611-a935-ef2f4eb5e0bd-workload-socket\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:30.614055 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:30.613881    5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5c79f52-3049-4611-a935-ef2f4eb5e0bd-istiod-ca-cert\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:30.615301 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:30.615279    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5c79f52-3049-4611-a935-ef2f4eb5e0bd-istio-envoy" pod "c5c79f52-3049-4611-a935-ef2f4eb5e0bd" (UID: "c5c79f52-3049-4611-a935-ef2f4eb5e0bd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:00:30.615527 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:30.615468    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5c79f52-3049-4611-a935-ef2f4eb5e0bd-istio-token" pod "c5c79f52-3049-4611-a935-ef2f4eb5e0bd" (UID: "c5c79f52-3049-4611-a935-ef2f4eb5e0bd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:00:30.615527 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:30.615479    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5c79f52-3049-4611-a935-ef2f4eb5e0bd-kube-api-access-nhqld" pod "c5c79f52-3049-4611-a935-ef2f4eb5e0bd" (UID: "c5c79f52-3049-4611-a935-ef2f4eb5e0bd"). InnerVolumeSpecName "kube-api-access-nhqld". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:00:30.615648 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:30.615551    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c5c79f52-3049-4611-a935-ef2f4eb5e0bd-istio-podinfo" pod "c5c79f52-3049-4611-a935-ef2f4eb5e0bd" (UID: "c5c79f52-3049-4611-a935-ef2f4eb5e0bd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 13:00:30.714928 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:30.714894    5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5c79f52-3049-4611-a935-ef2f4eb5e0bd-istio-envoy\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:30.714928 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:30.714923    5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5c79f52-3049-4611-a935-ef2f4eb5e0bd-istio-podinfo\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:30.714928 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:30.714933    5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5c79f52-3049-4611-a935-ef2f4eb5e0bd-istio-token\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:30.715176 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:30.714944    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhqld\" (UniqueName: \"kubernetes.io/projected/c5c79f52-3049-4611-a935-ef2f4eb5e0bd-kube-api-access-nhqld\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:30.715176 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:30.714955    5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5c79f52-3049-4611-a935-ef2f4eb5e0bd-workload-certs\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:30.715176 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:30.714964    5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5c79f52-3049-4611-a935-ef2f4eb5e0bd-istio-data\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:31.436448 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:31.436416    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-l8n27"
Oct 31 13:00:31.436644 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:31.436434    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-l8n27" event={"ID":"c5c79f52-3049-4611-a935-ef2f4eb5e0bd","Type":"ContainerDied","Data":"ac2088dd0e1a1315aa49042883f143c77917c1ecb4e308455521685e37af5a96"}
Oct 31 13:00:31.436644 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:31.436532    5060 scope.go:122] "RemoveContainer" containerID="996dfd8501bc25b0bbf2d5acaacca3f2bb6ca231454aee25616f89192cb48a5d"
Oct 31 13:00:31.438599 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:31.438558    5060 generic.go:356] "Generic (PLEG): container finished" podID="a5b0a48d-4c1d-4fef-a172-8e608048f155" containerID="633c4a8874cd6c68b1bd9b37c36fa00c02bcf97cf4e27419b166b768dade1bab" exitCode=0
Oct 31 13:00:31.438739 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:31.438705    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4918fc6d-zr525" event={"ID":"a5b0a48d-4c1d-4fef-a172-8e608048f155","Type":"ContainerDied","Data":"633c4a8874cd6c68b1bd9b37c36fa00c02bcf97cf4e27419b166b768dade1bab"}
Oct 31 13:00:31.480166 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:31.480117    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-l8n27"]
Oct 31 13:00:31.487142 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:31.487113    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-l8n27"]
Oct 31 13:00:32.617202 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:32.617173    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4918fc6d-zr525"
Oct 31 13:00:32.737653 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:32.737563    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5b0a48d-4c1d-4fef-a172-8e608048f155-kube-api-access-4ncsg\" (UniqueName: \"kubernetes.io/projected/a5b0a48d-4c1d-4fef-a172-8e608048f155-kube-api-access-4ncsg\") pod \"a5b0a48d-4c1d-4fef-a172-8e608048f155\" (UID: \"a5b0a48d-4c1d-4fef-a172-8e608048f155\") "
Oct 31 13:00:32.737653 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:32.737600    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5b0a48d-4c1d-4fef-a172-8e608048f155-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5b0a48d-4c1d-4fef-a172-8e608048f155-maas-api-openshift-ca\") pod \"a5b0a48d-4c1d-4fef-a172-8e608048f155\" (UID: \"a5b0a48d-4c1d-4fef-a172-8e608048f155\") "
Oct 31 13:00:32.737653 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:32.737618    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a5b0a48d-4c1d-4fef-a172-8e608048f155-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5b0a48d-4c1d-4fef-a172-8e608048f155-maas-api-xks-ca\") pod \"a5b0a48d-4c1d-4fef-a172-8e608048f155\" (UID: \"a5b0a48d-4c1d-4fef-a172-8e608048f155\") "
Oct 31 13:00:32.737957 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:32.737933    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5b0a48d-4c1d-4fef-a172-8e608048f155-maas-api-openshift-ca" pod "a5b0a48d-4c1d-4fef-a172-8e608048f155" (UID: "a5b0a48d-4c1d-4fef-a172-8e608048f155"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:00:32.739900 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:32.739861    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5b0a48d-4c1d-4fef-a172-8e608048f155-maas-api-xks-ca" pod "a5b0a48d-4c1d-4fef-a172-8e608048f155" (UID: "a5b0a48d-4c1d-4fef-a172-8e608048f155"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:00:32.740030 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:32.739902    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5b0a48d-4c1d-4fef-a172-8e608048f155-kube-api-access-4ncsg" pod "a5b0a48d-4c1d-4fef-a172-8e608048f155" (UID: "a5b0a48d-4c1d-4fef-a172-8e608048f155"). InnerVolumeSpecName "kube-api-access-4ncsg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:00:32.838932 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:32.838892    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4ncsg\" (UniqueName: \"kubernetes.io/projected/a5b0a48d-4c1d-4fef-a172-8e608048f155-kube-api-access-4ncsg\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:32.838932 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:32.838928    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5b0a48d-4c1d-4fef-a172-8e608048f155-maas-api-openshift-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:32.838932 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:32.838939    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5b0a48d-4c1d-4fef-a172-8e608048f155-maas-api-xks-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:33.047200 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:33.047169    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5c79f52-3049-4611-a935-ef2f4eb5e0bd" path="/var/lib/kubelet/pods/c5c79f52-3049-4611-a935-ef2f4eb5e0bd/volumes"
Oct 31 13:00:33.454320 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:33.454233    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-c67de2-7d5dc5649c-62vgx" podUID="a6fa2170-8276-4138-9614-551ddc387fb8" containerName="payload-processing" containerID="cri-o://e96b8eab5f9c9c0a8f6acb59152966c8dc7b7f7eda603b139dc8644ab53ab73e" gracePeriod=25
Oct 31 13:00:33.457570 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:33.457533    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4918fc6d-zr525" event={"ID":"a5b0a48d-4c1d-4fef-a172-8e608048f155","Type":"ContainerDied","Data":"56ff2143ea20a13342a38b56e4a0b7508d72ea0546a3ee2e7046b4442fe029ed"}
Oct 31 13:00:33.457731 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:33.457574    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4918fc6d-zr525"
Oct 31 13:00:33.457731 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:33.457596    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="56ff2143ea20a13342a38b56e4a0b7508d72ea0546a3ee2e7046b4442fe029ed"
Oct 31 13:00:33.474552 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:33.474519    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c67de2-5656c49759-8llqd" podUID="d496cda1-2b40-48f1-8d49-a5ecd7f28a2b" containerName="payload-pre-processing" containerID="cri-o://1e7016a5833af9a9d8fd6407b4e62e48484ca6d04e5d21270de1aa7ff8736173" gracePeriod=25
Oct 31 13:00:36.722781 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:36.722745    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-67d410-2npv4"]
Oct 31 13:00:36.723218 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:36.723204    5060 cpu_manager.go:414] "containerMap: removing container" podUID="c5c79f52-3049-4611-a935-ef2f4eb5e0bd" containerName="istio-proxy"
Oct 31 13:00:36.723278 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:36.723220    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5c79f52-3049-4611-a935-ef2f4eb5e0bd" containerName="istio-proxy"
Oct 31 13:00:36.723278 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:36.723255    5060 cpu_manager.go:414] "containerMap: removing container" podUID="a5b0a48d-4c1d-4fef-a172-8e608048f155" containerName="revoke-keys"
Oct 31 13:00:36.723278 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:36.723262    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5b0a48d-4c1d-4fef-a172-8e608048f155" containerName="revoke-keys"
Oct 31 13:00:36.723388 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:36.723348    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5c79f52-3049-4611-a935-ef2f4eb5e0bd" containerName="istio-proxy"
Oct 31 13:00:36.723388 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:36.723362    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5b0a48d-4c1d-4fef-a172-8e608048f155" containerName="revoke-keys"
Oct 31 13:00:36.727917 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:36.727895    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-67d410-2npv4"
Oct 31 13:00:36.734349 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:36.734322    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 13:00:36.734561 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:36.734538    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 13:00:36.734650 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:36.734603    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 13:00:36.735688 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:36.735669    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7wsmb\""
Oct 31 13:00:36.739743 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:36.739723    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-67d410-2npv4"]
Oct 31 13:00:36.884642 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:36.884604    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e13e1c42-cae4-4f43-badd-24293a62300d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-67d410-2npv4\" (UID: \"e13e1c42-cae4-4f43-badd-24293a62300d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-67d410-2npv4"
Oct 31 13:00:36.884810 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:36.884659    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e13e1c42-cae4-4f43-badd-24293a62300d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-67d410-2npv4\" (UID: \"e13e1c42-cae4-4f43-badd-24293a62300d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-67d410-2npv4"
Oct 31 13:00:36.884810 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:36.884691    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4z2cw\" (UniqueName: \"kubernetes.io/projected/e13e1c42-cae4-4f43-badd-24293a62300d-kube-api-access-4z2cw\") pod \"maas-api-revoke-keys-e2e-worker-w6-67d410-2npv4\" (UID: \"e13e1c42-cae4-4f43-badd-24293a62300d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-67d410-2npv4"
Oct 31 13:00:36.986094 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:36.986010    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e13e1c42-cae4-4f43-badd-24293a62300d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-67d410-2npv4\" (UID: \"e13e1c42-cae4-4f43-badd-24293a62300d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-67d410-2npv4"
Oct 31 13:00:36.986094 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:36.986060    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e13e1c42-cae4-4f43-badd-24293a62300d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-67d410-2npv4\" (UID: \"e13e1c42-cae4-4f43-badd-24293a62300d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-67d410-2npv4"
Oct 31 13:00:36.986094 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:36.986085    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4z2cw\" (UniqueName: \"kubernetes.io/projected/e13e1c42-cae4-4f43-badd-24293a62300d-kube-api-access-4z2cw\") pod \"maas-api-revoke-keys-e2e-worker-w6-67d410-2npv4\" (UID: \"e13e1c42-cae4-4f43-badd-24293a62300d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-67d410-2npv4"
Oct 31 13:00:36.986685 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:36.986652    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e13e1c42-cae4-4f43-badd-24293a62300d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-67d410-2npv4\" (UID: \"e13e1c42-cae4-4f43-badd-24293a62300d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-67d410-2npv4"
Oct 31 13:00:36.988402 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:36.988374    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e13e1c42-cae4-4f43-badd-24293a62300d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-67d410-2npv4\" (UID: \"e13e1c42-cae4-4f43-badd-24293a62300d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-67d410-2npv4"
Oct 31 13:00:36.998652 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:36.998624    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4z2cw\" (UniqueName: \"kubernetes.io/projected/e13e1c42-cae4-4f43-badd-24293a62300d-kube-api-access-4z2cw\") pod \"maas-api-revoke-keys-e2e-worker-w6-67d410-2npv4\" (UID: \"e13e1c42-cae4-4f43-badd-24293a62300d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-67d410-2npv4"
Oct 31 13:00:37.045734 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:37.045707    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-67d410-2npv4"
Oct 31 13:00:37.328434 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:37.328404    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-67d410-2npv4"]
Oct 31 13:00:37.330793 ip-10-0-130-190 kubenswrapper[5060]: W1031 13:00:37.330750    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode13e1c42_cae4_4f43_badd_24293a62300d.slice/crio-8b724079d61e883dd019897a096e7d84afe90ce574b19ac6dda431ff38c34ee9 WatchSource:0}: Error finding container 8b724079d61e883dd019897a096e7d84afe90ce574b19ac6dda431ff38c34ee9: Status 404 returned error can't find the container with id 8b724079d61e883dd019897a096e7d84afe90ce574b19ac6dda431ff38c34ee9
Oct 31 13:00:37.494262 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:37.494220    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-67d410-2npv4" event={"ID":"e13e1c42-cae4-4f43-badd-24293a62300d","Type":"ContainerStarted","Data":"8b724079d61e883dd019897a096e7d84afe90ce574b19ac6dda431ff38c34ee9"}
Oct 31 13:00:38.173650 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:38.173603    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-c67de2/simulator-w4-c67de2-kserve-868679cf9b-gqjbx" podUID="e104d7da-1fb2-4c99-a480-b4ae84c4f273" containerName="main" containerID="cri-o://df554570f7de10a33f3fdb5dc1a575a7532829e536a1120d3f6d1e78374fed8d" gracePeriod=45
Oct 31 13:00:38.444977 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:38.444952    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c67de2/simulator-w4-c67de2-kserve-868679cf9b-gqjbx"
Oct 31 13:00:38.510207 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:38.510170    5060 generic.go:356] "Generic (PLEG): container finished" podID="e13e1c42-cae4-4f43-badd-24293a62300d" containerID="a281ef5accb16b0b9e3d81b3447776184f312bb530b36af79df4589523999ce8" exitCode=0
Oct 31 13:00:38.510378 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:38.510253    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-67d410-2npv4" event={"ID":"e13e1c42-cae4-4f43-badd-24293a62300d","Type":"ContainerDied","Data":"a281ef5accb16b0b9e3d81b3447776184f312bb530b36af79df4589523999ce8"}
Oct 31 13:00:38.512001 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:38.511979    5060 generic.go:356] "Generic (PLEG): container finished" podID="e104d7da-1fb2-4c99-a480-b4ae84c4f273" containerID="df554570f7de10a33f3fdb5dc1a575a7532829e536a1120d3f6d1e78374fed8d" exitCode=0
Oct 31 13:00:38.512126 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:38.512015    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c67de2/simulator-w4-c67de2-kserve-868679cf9b-gqjbx" event={"ID":"e104d7da-1fb2-4c99-a480-b4ae84c4f273","Type":"ContainerDied","Data":"df554570f7de10a33f3fdb5dc1a575a7532829e536a1120d3f6d1e78374fed8d"}
Oct 31 13:00:38.512126 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:38.512030    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c67de2/simulator-w4-c67de2-kserve-868679cf9b-gqjbx"
Oct 31 13:00:38.512126 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:38.512046    5060 scope.go:122] "RemoveContainer" containerID="df554570f7de10a33f3fdb5dc1a575a7532829e536a1120d3f6d1e78374fed8d"
Oct 31 13:00:38.512126 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:38.512036    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c67de2/simulator-w4-c67de2-kserve-868679cf9b-gqjbx" event={"ID":"e104d7da-1fb2-4c99-a480-b4ae84c4f273","Type":"ContainerDied","Data":"4293f1dbc4ec0aecf0e2a3502484912dd2f9c2296523f950c3a5f744c903fc26"}
Oct 31 13:00:38.525231 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:38.525213    5060 scope.go:122] "RemoveContainer" containerID="df554570f7de10a33f3fdb5dc1a575a7532829e536a1120d3f6d1e78374fed8d"
Oct 31 13:00:38.525576 ip-10-0-130-190 kubenswrapper[5060]: E1031 13:00:38.525551    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df554570f7de10a33f3fdb5dc1a575a7532829e536a1120d3f6d1e78374fed8d\": container with ID starting with df554570f7de10a33f3fdb5dc1a575a7532829e536a1120d3f6d1e78374fed8d not found: ID does not exist" containerID="df554570f7de10a33f3fdb5dc1a575a7532829e536a1120d3f6d1e78374fed8d"
Oct 31 13:00:38.525687 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:38.525588    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df554570f7de10a33f3fdb5dc1a575a7532829e536a1120d3f6d1e78374fed8d"} err="failed to get container status \"df554570f7de10a33f3fdb5dc1a575a7532829e536a1120d3f6d1e78374fed8d\": rpc error: code = NotFound desc = could not find container \"df554570f7de10a33f3fdb5dc1a575a7532829e536a1120d3f6d1e78374fed8d\": container with ID starting with df554570f7de10a33f3fdb5dc1a575a7532829e536a1120d3f6d1e78374fed8d not found: ID does not exist"
Oct 31 13:00:38.604251 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:38.604212    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e104d7da-1fb2-4c99-a480-b4ae84c4f273-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e104d7da-1fb2-4c99-a480-b4ae84c4f273-model-cache\") pod \"e104d7da-1fb2-4c99-a480-b4ae84c4f273\" (UID: \"e104d7da-1fb2-4c99-a480-b4ae84c4f273\") "
Oct 31 13:00:38.604426 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:38.604263    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e104d7da-1fb2-4c99-a480-b4ae84c4f273-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e104d7da-1fb2-4c99-a480-b4ae84c4f273-dshm\") pod \"e104d7da-1fb2-4c99-a480-b4ae84c4f273\" (UID: \"e104d7da-1fb2-4c99-a480-b4ae84c4f273\") "
Oct 31 13:00:38.604426 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:38.604299    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e104d7da-1fb2-4c99-a480-b4ae84c4f273-kube-api-access-gb4rx\" (UniqueName: \"kubernetes.io/projected/e104d7da-1fb2-4c99-a480-b4ae84c4f273-kube-api-access-gb4rx\") pod \"e104d7da-1fb2-4c99-a480-b4ae84c4f273\" (UID: \"e104d7da-1fb2-4c99-a480-b4ae84c4f273\") "
Oct 31 13:00:38.604426 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:38.604376    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e104d7da-1fb2-4c99-a480-b4ae84c4f273-home\" (UniqueName: \"kubernetes.io/empty-dir/e104d7da-1fb2-4c99-a480-b4ae84c4f273-home\") pod \"e104d7da-1fb2-4c99-a480-b4ae84c4f273\" (UID: \"e104d7da-1fb2-4c99-a480-b4ae84c4f273\") "
Oct 31 13:00:38.604651 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:38.604428    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e104d7da-1fb2-4c99-a480-b4ae84c4f273-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e104d7da-1fb2-4c99-a480-b4ae84c4f273-tmp-dir\") pod \"e104d7da-1fb2-4c99-a480-b4ae84c4f273\" (UID: \"e104d7da-1fb2-4c99-a480-b4ae84c4f273\") "
Oct 31 13:00:38.604651 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:38.604460    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e104d7da-1fb2-4c99-a480-b4ae84c4f273-tls-certs\" (UniqueName: \"kubernetes.io/secret/e104d7da-1fb2-4c99-a480-b4ae84c4f273-tls-certs\") pod \"e104d7da-1fb2-4c99-a480-b4ae84c4f273\" (UID: \"e104d7da-1fb2-4c99-a480-b4ae84c4f273\") "
Oct 31 13:00:38.604651 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:38.604546    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e104d7da-1fb2-4c99-a480-b4ae84c4f273-model-cache" pod "e104d7da-1fb2-4c99-a480-b4ae84c4f273" (UID: "e104d7da-1fb2-4c99-a480-b4ae84c4f273"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:00:38.604925 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:38.604867    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e104d7da-1fb2-4c99-a480-b4ae84c4f273-home" pod "e104d7da-1fb2-4c99-a480-b4ae84c4f273" (UID: "e104d7da-1fb2-4c99-a480-b4ae84c4f273"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:00:38.605239 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:38.605200    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e104d7da-1fb2-4c99-a480-b4ae84c4f273-tmp-dir" pod "e104d7da-1fb2-4c99-a480-b4ae84c4f273" (UID: "e104d7da-1fb2-4c99-a480-b4ae84c4f273"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:00:38.606828 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:38.606792    5060 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e104d7da-1fb2-4c99-a480-b4ae84c4f273-tmp-dir\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:38.606944 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:38.606845    5060 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e104d7da-1fb2-4c99-a480-b4ae84c4f273-model-cache\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:38.606944 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:38.606861    5060 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e104d7da-1fb2-4c99-a480-b4ae84c4f273-home\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:38.607680 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:38.607653    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e104d7da-1fb2-4c99-a480-b4ae84c4f273-dshm" pod "e104d7da-1fb2-4c99-a480-b4ae84c4f273" (UID: "e104d7da-1fb2-4c99-a480-b4ae84c4f273"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:00:38.607788 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:38.607678    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e104d7da-1fb2-4c99-a480-b4ae84c4f273-tls-certs" pod "e104d7da-1fb2-4c99-a480-b4ae84c4f273" (UID: "e104d7da-1fb2-4c99-a480-b4ae84c4f273"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:00:38.608091 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:38.608071    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e104d7da-1fb2-4c99-a480-b4ae84c4f273-kube-api-access-gb4rx" pod "e104d7da-1fb2-4c99-a480-b4ae84c4f273" (UID: "e104d7da-1fb2-4c99-a480-b4ae84c4f273"). InnerVolumeSpecName "kube-api-access-gb4rx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:00:38.707689 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:38.707604    5060 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e104d7da-1fb2-4c99-a480-b4ae84c4f273-tls-certs\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:38.707689 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:38.707634    5060 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e104d7da-1fb2-4c99-a480-b4ae84c4f273-dshm\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:38.707689 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:38.707648    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gb4rx\" (UniqueName: \"kubernetes.io/projected/e104d7da-1fb2-4c99-a480-b4ae84c4f273-kube-api-access-gb4rx\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:38.836691 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:38.836649    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-c67de2/simulator-w4-c67de2-kserve-868679cf9b-gqjbx"]
Oct 31 13:00:38.839558 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:38.839531    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-c67de2/simulator-w4-c67de2-kserve-868679cf9b-gqjbx"]
Oct 31 13:00:39.048373 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:39.048340    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e104d7da-1fb2-4c99-a480-b4ae84c4f273" path="/var/lib/kubelet/pods/e104d7da-1fb2-4c99-a480-b4ae84c4f273/volumes"
Oct 31 13:00:39.682153 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:39.682123    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-67d410-2npv4"
Oct 31 13:00:39.817996 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:39.817905    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e13e1c42-cae4-4f43-badd-24293a62300d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e13e1c42-cae4-4f43-badd-24293a62300d-maas-api-openshift-ca\") pod \"e13e1c42-cae4-4f43-badd-24293a62300d\" (UID: \"e13e1c42-cae4-4f43-badd-24293a62300d\") "
Oct 31 13:00:39.817996 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:39.817977    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e13e1c42-cae4-4f43-badd-24293a62300d-kube-api-access-4z2cw\" (UniqueName: \"kubernetes.io/projected/e13e1c42-cae4-4f43-badd-24293a62300d-kube-api-access-4z2cw\") pod \"e13e1c42-cae4-4f43-badd-24293a62300d\" (UID: \"e13e1c42-cae4-4f43-badd-24293a62300d\") "
Oct 31 13:00:39.818186 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:39.818128    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e13e1c42-cae4-4f43-badd-24293a62300d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e13e1c42-cae4-4f43-badd-24293a62300d-maas-api-xks-ca\") pod \"e13e1c42-cae4-4f43-badd-24293a62300d\" (UID: \"e13e1c42-cae4-4f43-badd-24293a62300d\") "
Oct 31 13:00:39.818346 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:39.818322    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e13e1c42-cae4-4f43-badd-24293a62300d-maas-api-openshift-ca" pod "e13e1c42-cae4-4f43-badd-24293a62300d" (UID: "e13e1c42-cae4-4f43-badd-24293a62300d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:00:39.818602 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:39.818575    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e13e1c42-cae4-4f43-badd-24293a62300d-maas-api-openshift-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:39.820360 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:39.820326    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e13e1c42-cae4-4f43-badd-24293a62300d-maas-api-xks-ca" pod "e13e1c42-cae4-4f43-badd-24293a62300d" (UID: "e13e1c42-cae4-4f43-badd-24293a62300d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:00:39.820463 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:39.820440    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e13e1c42-cae4-4f43-badd-24293a62300d-kube-api-access-4z2cw" pod "e13e1c42-cae4-4f43-badd-24293a62300d" (UID: "e13e1c42-cae4-4f43-badd-24293a62300d"). InnerVolumeSpecName "kube-api-access-4z2cw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:00:39.919505 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:39.919452    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4z2cw\" (UniqueName: \"kubernetes.io/projected/e13e1c42-cae4-4f43-badd-24293a62300d-kube-api-access-4z2cw\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:39.919694 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:39.919528    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e13e1c42-cae4-4f43-badd-24293a62300d-maas-api-xks-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:40.530298 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:40.530268    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-67d410-2npv4"
Oct 31 13:00:40.530552 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:40.530300    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-67d410-2npv4" event={"ID":"e13e1c42-cae4-4f43-badd-24293a62300d","Type":"ContainerDied","Data":"8b724079d61e883dd019897a096e7d84afe90ce574b19ac6dda431ff38c34ee9"}
Oct 31 13:00:40.530552 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:40.530349    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8b724079d61e883dd019897a096e7d84afe90ce574b19ac6dda431ff38c34ee9"
Oct 31 13:00:40.883730 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:40.883642    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c67de2-5656c49759-8llqd" podUID="d496cda1-2b40-48f1-8d49-a5ecd7f28a2b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 13:00:40.884538 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:40.884465    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-c67de2-7d5dc5649c-62vgx" podUID="a6fa2170-8276-4138-9614-551ddc387fb8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 13:00:45.024128 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:45.024086    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-c67de2-px6f4"]
Oct 31 13:00:45.030584 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:45.030549    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-c67de2-px6f4"]
Oct 31 13:00:45.047595 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:45.047562    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc6253cf-8a68-46d7-ba90-6402318088d8" path="/var/lib/kubelet/pods/cc6253cf-8a68-46d7-ba90-6402318088d8/volumes"
Oct 31 13:00:46.906750 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:46.906704    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-67d410-6c74c4dc47-k6m8r"]
Oct 31 13:00:46.923847 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:46.923812    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-67d410-5c9cd96795-tvtnh"]
Oct 31 13:00:47.362803 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:47.362767    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef807d2f-h9wvz"]
Oct 31 13:00:47.363547 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:47.363525    5060 cpu_manager.go:414] "containerMap: removing container" podUID="e13e1c42-cae4-4f43-badd-24293a62300d" containerName="revoke-keys"
Oct 31 13:00:47.363619 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:47.363550    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e13e1c42-cae4-4f43-badd-24293a62300d" containerName="revoke-keys"
Oct 31 13:00:47.363619 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:47.363565    5060 cpu_manager.go:414] "containerMap: removing container" podUID="e104d7da-1fb2-4c99-a480-b4ae84c4f273" containerName="main"
Oct 31 13:00:47.363619 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:47.363573    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e104d7da-1fb2-4c99-a480-b4ae84c4f273" containerName="main"
Oct 31 13:00:47.363720 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:47.363708    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="e13e1c42-cae4-4f43-badd-24293a62300d" containerName="revoke-keys"
Oct 31 13:00:47.363761 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:47.363722    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="e104d7da-1fb2-4c99-a480-b4ae84c4f273" containerName="main"
Oct 31 13:00:47.367779 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:47.367759    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef807d2f-h9wvz"
Oct 31 13:00:47.376871 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:47.376850    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 13:00:47.377067 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:47.376918    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef807d2f-h9wvz"]
Oct 31 13:00:47.377170 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:47.377000    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 13:00:47.398703 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:47.398662    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 13:00:47.398907 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:47.398861    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7wsmb\""
Oct 31 13:00:47.500719 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:47.500682    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48f68ea6-2cb0-4e83-8d61-db1532be4735-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ef807d2f-h9wvz\" (UID: \"48f68ea6-2cb0-4e83-8d61-db1532be4735\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef807d2f-h9wvz"
Oct 31 13:00:47.500921 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:47.500823    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqnn7\" (UniqueName: \"kubernetes.io/projected/48f68ea6-2cb0-4e83-8d61-db1532be4735-kube-api-access-gqnn7\") pod \"maas-api-revoke-sub-ef807d2f-h9wvz\" (UID: \"48f68ea6-2cb0-4e83-8d61-db1532be4735\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef807d2f-h9wvz"
Oct 31 13:00:47.500978 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:47.500945    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48f68ea6-2cb0-4e83-8d61-db1532be4735-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ef807d2f-h9wvz\" (UID: \"48f68ea6-2cb0-4e83-8d61-db1532be4735\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef807d2f-h9wvz"
Oct 31 13:00:47.604756 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:47.604718    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqnn7\" (UniqueName: \"kubernetes.io/projected/48f68ea6-2cb0-4e83-8d61-db1532be4735-kube-api-access-gqnn7\") pod \"maas-api-revoke-sub-ef807d2f-h9wvz\" (UID: \"48f68ea6-2cb0-4e83-8d61-db1532be4735\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef807d2f-h9wvz"
Oct 31 13:00:47.604966 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:47.604817    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48f68ea6-2cb0-4e83-8d61-db1532be4735-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ef807d2f-h9wvz\" (UID: \"48f68ea6-2cb0-4e83-8d61-db1532be4735\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef807d2f-h9wvz"
Oct 31 13:00:47.604966 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:47.604877    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48f68ea6-2cb0-4e83-8d61-db1532be4735-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ef807d2f-h9wvz\" (UID: \"48f68ea6-2cb0-4e83-8d61-db1532be4735\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef807d2f-h9wvz"
Oct 31 13:00:47.605781 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:47.605751    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48f68ea6-2cb0-4e83-8d61-db1532be4735-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ef807d2f-h9wvz\" (UID: \"48f68ea6-2cb0-4e83-8d61-db1532be4735\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef807d2f-h9wvz"
Oct 31 13:00:47.619159 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:47.619078    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48f68ea6-2cb0-4e83-8d61-db1532be4735-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ef807d2f-h9wvz\" (UID: \"48f68ea6-2cb0-4e83-8d61-db1532be4735\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef807d2f-h9wvz"
Oct 31 13:00:47.620417 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:47.620391    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqnn7\" (UniqueName: \"kubernetes.io/projected/48f68ea6-2cb0-4e83-8d61-db1532be4735-kube-api-access-gqnn7\") pod \"maas-api-revoke-sub-ef807d2f-h9wvz\" (UID: \"48f68ea6-2cb0-4e83-8d61-db1532be4735\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef807d2f-h9wvz"
Oct 31 13:00:47.690378 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:47.690343    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef807d2f-h9wvz"
Oct 31 13:00:47.985650 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:47.985623    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef807d2f-h9wvz"]
Oct 31 13:00:47.986643 ip-10-0-130-190 kubenswrapper[5060]: W1031 13:00:47.986616    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48f68ea6_2cb0_4e83_8d61_db1532be4735.slice/crio-f31635c7aa3bf37738d4a2751b83fcbb80d9720e2547c6e9c0dc60a881c3186d WatchSource:0}: Error finding container f31635c7aa3bf37738d4a2751b83fcbb80d9720e2547c6e9c0dc60a881c3186d: Status 404 returned error can't find the container with id f31635c7aa3bf37738d4a2751b83fcbb80d9720e2547c6e9c0dc60a881c3186d
Oct 31 13:00:48.592702 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:48.592663    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef807d2f-h9wvz" event={"ID":"48f68ea6-2cb0-4e83-8d61-db1532be4735","Type":"ContainerStarted","Data":"52adb4beb2b982405dba128837cf938918914beb34d14851c7e7c42cfef7217c"}
Oct 31 13:00:48.592912 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:48.592717    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef807d2f-h9wvz" event={"ID":"48f68ea6-2cb0-4e83-8d61-db1532be4735","Type":"ContainerStarted","Data":"f31635c7aa3bf37738d4a2751b83fcbb80d9720e2547c6e9c0dc60a881c3186d"}
Oct 31 13:00:49.609648 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:49.609604    5060 generic.go:356] "Generic (PLEG): container finished" podID="48f68ea6-2cb0-4e83-8d61-db1532be4735" containerID="52adb4beb2b982405dba128837cf938918914beb34d14851c7e7c42cfef7217c" exitCode=0
Oct 31 13:00:49.610140 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:49.609654    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef807d2f-h9wvz" event={"ID":"48f68ea6-2cb0-4e83-8d61-db1532be4735","Type":"ContainerDied","Data":"52adb4beb2b982405dba128837cf938918914beb34d14851c7e7c42cfef7217c"}
Oct 31 13:00:50.773478 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:50.773453    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef807d2f-h9wvz"
Oct 31 13:00:50.883794 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:50.883691    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c67de2-5656c49759-8llqd" podUID="d496cda1-2b40-48f1-8d49-a5ecd7f28a2b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 13:00:50.884339 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:50.884311    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-c67de2-7d5dc5649c-62vgx" podUID="a6fa2170-8276-4138-9614-551ddc387fb8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 13:00:50.943149 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:50.943119    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48f68ea6-2cb0-4e83-8d61-db1532be4735-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48f68ea6-2cb0-4e83-8d61-db1532be4735-maas-api-openshift-ca\") pod \"48f68ea6-2cb0-4e83-8d61-db1532be4735\" (UID: \"48f68ea6-2cb0-4e83-8d61-db1532be4735\") "
Oct 31 13:00:50.943349 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:50.943249    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48f68ea6-2cb0-4e83-8d61-db1532be4735-kube-api-access-gqnn7\" (UniqueName: \"kubernetes.io/projected/48f68ea6-2cb0-4e83-8d61-db1532be4735-kube-api-access-gqnn7\") pod \"48f68ea6-2cb0-4e83-8d61-db1532be4735\" (UID: \"48f68ea6-2cb0-4e83-8d61-db1532be4735\") "
Oct 31 13:00:50.943349 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:50.943301    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/48f68ea6-2cb0-4e83-8d61-db1532be4735-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48f68ea6-2cb0-4e83-8d61-db1532be4735-maas-api-xks-ca\") pod \"48f68ea6-2cb0-4e83-8d61-db1532be4735\" (UID: \"48f68ea6-2cb0-4e83-8d61-db1532be4735\") "
Oct 31 13:00:50.943605 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:50.943561    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48f68ea6-2cb0-4e83-8d61-db1532be4735-maas-api-openshift-ca" pod "48f68ea6-2cb0-4e83-8d61-db1532be4735" (UID: "48f68ea6-2cb0-4e83-8d61-db1532be4735"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:00:50.943710 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:50.943681    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48f68ea6-2cb0-4e83-8d61-db1532be4735-maas-api-openshift-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:50.945720 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:50.945689    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48f68ea6-2cb0-4e83-8d61-db1532be4735-maas-api-xks-ca" pod "48f68ea6-2cb0-4e83-8d61-db1532be4735" (UID: "48f68ea6-2cb0-4e83-8d61-db1532be4735"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:00:50.945820 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:50.945692    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48f68ea6-2cb0-4e83-8d61-db1532be4735-kube-api-access-gqnn7" pod "48f68ea6-2cb0-4e83-8d61-db1532be4735" (UID: "48f68ea6-2cb0-4e83-8d61-db1532be4735"). InnerVolumeSpecName "kube-api-access-gqnn7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:00:51.044799 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:51.044769    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqnn7\" (UniqueName: \"kubernetes.io/projected/48f68ea6-2cb0-4e83-8d61-db1532be4735-kube-api-access-gqnn7\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:51.044799 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:51.044798    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48f68ea6-2cb0-4e83-8d61-db1532be4735-maas-api-xks-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:51.627061 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:51.627024    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef807d2f-h9wvz" event={"ID":"48f68ea6-2cb0-4e83-8d61-db1532be4735","Type":"ContainerDied","Data":"f31635c7aa3bf37738d4a2751b83fcbb80d9720e2547c6e9c0dc60a881c3186d"}
Oct 31 13:00:51.627061 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:51.627049    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef807d2f-h9wvz"
Oct 31 13:00:51.627346 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:51.627080    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f31635c7aa3bf37738d4a2751b83fcbb80d9720e2547c6e9c0dc60a881c3186d"
Oct 31 13:00:51.907738 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:51.907648    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w6-67d410-6c74c4dc47-k6m8r" podUID="f85a5783-97ad-4058-bcdc-0981cea23afd" containerName="payload-processing" containerID="cri-o://4e2808ace3cd5f2e086bd0fae56e55008f333403e24023588a1e7d5238b9c297" gracePeriod=25
Oct 31 13:00:51.924266 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:51.924224    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-67d410-5c9cd96795-tvtnh" podUID="6772255c-c420-435e-8075-f52344d30bdb" containerName="payload-pre-processing" containerID="cri-o://40634ad90f586d271990f309270efe61e24804a62b4c02e119264e9dca9287e2" gracePeriod=25
Oct 31 13:00:52.014453 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:52.014409    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-56grg"]
Oct 31 13:00:52.014714 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:52.014685    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-56grg" podUID="f43cdb3b-9bd6-46a7-a14c-feec07c2c755" containerName="istio-proxy" containerID="cri-o://5dbbe131ebca981cfcb804c34551921810cfbf21be15aa15636d6888f9170867" gracePeriod=30
Oct 31 13:00:53.657565 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:53.653598    5060 generic.go:356] "Generic (PLEG): container finished" podID="a6fa2170-8276-4138-9614-551ddc387fb8" containerID="e96b8eab5f9c9c0a8f6acb59152966c8dc7b7f7eda603b139dc8644ab53ab73e" exitCode=0
Oct 31 13:00:53.657565 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:53.653788    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-c67de2-7d5dc5649c-62vgx" event={"ID":"a6fa2170-8276-4138-9614-551ddc387fb8","Type":"ContainerDied","Data":"e96b8eab5f9c9c0a8f6acb59152966c8dc7b7f7eda603b139dc8644ab53ab73e"}
Oct 31 13:00:53.657565 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:53.656787    5060 generic.go:356] "Generic (PLEG): container finished" podID="d496cda1-2b40-48f1-8d49-a5ecd7f28a2b" containerID="1e7016a5833af9a9d8fd6407b4e62e48484ca6d04e5d21270de1aa7ff8736173" exitCode=0
Oct 31 13:00:53.657565 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:53.656826    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c67de2-5656c49759-8llqd" event={"ID":"d496cda1-2b40-48f1-8d49-a5ecd7f28a2b","Type":"ContainerDied","Data":"1e7016a5833af9a9d8fd6407b4e62e48484ca6d04e5d21270de1aa7ff8736173"}
Oct 31 13:00:53.746890 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:53.746863    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-c67de2-7d5dc5649c-62vgx"
Oct 31 13:00:53.775069 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:53.775039    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6fa2170-8276-4138-9614-551ddc387fb8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6fa2170-8276-4138-9614-551ddc387fb8-plugins-config-volume\") pod \"a6fa2170-8276-4138-9614-551ddc387fb8\" (UID: \"a6fa2170-8276-4138-9614-551ddc387fb8\") "
Oct 31 13:00:53.775200 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:53.775100    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6fa2170-8276-4138-9614-551ddc387fb8-kube-api-access-hkwhk\" (UniqueName: \"kubernetes.io/projected/a6fa2170-8276-4138-9614-551ddc387fb8-kube-api-access-hkwhk\") pod \"a6fa2170-8276-4138-9614-551ddc387fb8\" (UID: \"a6fa2170-8276-4138-9614-551ddc387fb8\") "
Oct 31 13:00:53.775442 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:53.775403    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6fa2170-8276-4138-9614-551ddc387fb8-plugins-config-volume" pod "a6fa2170-8276-4138-9614-551ddc387fb8" (UID: "a6fa2170-8276-4138-9614-551ddc387fb8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:00:53.775600 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:53.775582    5060 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6fa2170-8276-4138-9614-551ddc387fb8-plugins-config-volume\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:53.777913 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:53.777885    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6fa2170-8276-4138-9614-551ddc387fb8-kube-api-access-hkwhk" pod "a6fa2170-8276-4138-9614-551ddc387fb8" (UID: "a6fa2170-8276-4138-9614-551ddc387fb8"). InnerVolumeSpecName "kube-api-access-hkwhk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:00:53.780546 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:53.780470    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c67de2-5656c49759-8llqd"
Oct 31 13:00:53.876112 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:53.876077    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d496cda1-2b40-48f1-8d49-a5ecd7f28a2b-kube-api-access-fmng4\" (UniqueName: \"kubernetes.io/projected/d496cda1-2b40-48f1-8d49-a5ecd7f28a2b-kube-api-access-fmng4\") pod \"d496cda1-2b40-48f1-8d49-a5ecd7f28a2b\" (UID: \"d496cda1-2b40-48f1-8d49-a5ecd7f28a2b\") "
Oct 31 13:00:53.876301 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:53.876146    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d496cda1-2b40-48f1-8d49-a5ecd7f28a2b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d496cda1-2b40-48f1-8d49-a5ecd7f28a2b-plugins-config-volume\") pod \"d496cda1-2b40-48f1-8d49-a5ecd7f28a2b\" (UID: \"d496cda1-2b40-48f1-8d49-a5ecd7f28a2b\") "
Oct 31 13:00:53.876477 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:53.876452    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hkwhk\" (UniqueName: \"kubernetes.io/projected/a6fa2170-8276-4138-9614-551ddc387fb8-kube-api-access-hkwhk\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:53.876678 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:53.876649    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d496cda1-2b40-48f1-8d49-a5ecd7f28a2b-plugins-config-volume" pod "d496cda1-2b40-48f1-8d49-a5ecd7f28a2b" (UID: "d496cda1-2b40-48f1-8d49-a5ecd7f28a2b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:00:53.878373 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:53.878349    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d496cda1-2b40-48f1-8d49-a5ecd7f28a2b-kube-api-access-fmng4" pod "d496cda1-2b40-48f1-8d49-a5ecd7f28a2b" (UID: "d496cda1-2b40-48f1-8d49-a5ecd7f28a2b"). InnerVolumeSpecName "kube-api-access-fmng4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:00:53.977551 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:53.977446    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fmng4\" (UniqueName: \"kubernetes.io/projected/d496cda1-2b40-48f1-8d49-a5ecd7f28a2b-kube-api-access-fmng4\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:53.977551 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:53.977513    5060 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d496cda1-2b40-48f1-8d49-a5ecd7f28a2b-plugins-config-volume\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:54.667111 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:54.667083    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-c67de2-7d5dc5649c-62vgx"
Oct 31 13:00:54.667111 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:54.667099    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-c67de2-7d5dc5649c-62vgx" event={"ID":"a6fa2170-8276-4138-9614-551ddc387fb8","Type":"ContainerDied","Data":"89986bb03af6ee0561ab06c858069e984f01d692ec198a9eb25494a4abdfaa77"}
Oct 31 13:00:54.667651 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:54.667183    5060 scope.go:122] "RemoveContainer" containerID="e96b8eab5f9c9c0a8f6acb59152966c8dc7b7f7eda603b139dc8644ab53ab73e"
Oct 31 13:00:54.668954 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:54.668932    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c67de2-5656c49759-8llqd"
Oct 31 13:00:54.668954 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:54.668933    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c67de2-5656c49759-8llqd" event={"ID":"d496cda1-2b40-48f1-8d49-a5ecd7f28a2b","Type":"ContainerDied","Data":"d4e4ae99f052ec65738dee2ca2efcdada2a49db5aaca6aec9f4ba1fd65d78c04"}
Oct 31 13:00:54.681009 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:54.680904    5060 scope.go:122] "RemoveContainer" containerID="1e7016a5833af9a9d8fd6407b4e62e48484ca6d04e5d21270de1aa7ff8736173"
Oct 31 13:00:54.695096 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:54.695067    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-c67de2-7d5dc5649c-62vgx"]
Oct 31 13:00:54.706119 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:54.706090    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-c67de2-7d5dc5649c-62vgx"]
Oct 31 13:00:54.716102 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:54.716071    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-c67de2-5656c49759-8llqd"]
Oct 31 13:00:54.727868 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:54.727802    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-c67de2-5656c49759-8llqd"]
Oct 31 13:00:55.047629 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:55.047588    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6fa2170-8276-4138-9614-551ddc387fb8" path="/var/lib/kubelet/pods/a6fa2170-8276-4138-9614-551ddc387fb8/volumes"
Oct 31 13:00:55.047988 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:55.047971    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d496cda1-2b40-48f1-8d49-a5ecd7f28a2b" path="/var/lib/kubelet/pods/d496cda1-2b40-48f1-8d49-a5ecd7f28a2b/volumes"
Oct 31 13:00:57.298403 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:57.298345    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-56grg"
Oct 31 13:00:57.310907 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:57.310871    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f43cdb3b-9bd6-46a7-a14c-feec07c2c755-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f43cdb3b-9bd6-46a7-a14c-feec07c2c755-workload-certs\") pod \"f43cdb3b-9bd6-46a7-a14c-feec07c2c755\" (UID: \"f43cdb3b-9bd6-46a7-a14c-feec07c2c755\") "
Oct 31 13:00:57.311038 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:57.310947    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f43cdb3b-9bd6-46a7-a14c-feec07c2c755-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f43cdb3b-9bd6-46a7-a14c-feec07c2c755-istio-data\") pod \"f43cdb3b-9bd6-46a7-a14c-feec07c2c755\" (UID: \"f43cdb3b-9bd6-46a7-a14c-feec07c2c755\") "
Oct 31 13:00:57.311096 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:57.311081    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f43cdb3b-9bd6-46a7-a14c-feec07c2c755-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f43cdb3b-9bd6-46a7-a14c-feec07c2c755-istio-podinfo\") pod \"f43cdb3b-9bd6-46a7-a14c-feec07c2c755\" (UID: \"f43cdb3b-9bd6-46a7-a14c-feec07c2c755\") "
Oct 31 13:00:57.311158 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:57.311116    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f43cdb3b-9bd6-46a7-a14c-feec07c2c755-workload-certs" pod "f43cdb3b-9bd6-46a7-a14c-feec07c2c755" (UID: "f43cdb3b-9bd6-46a7-a14c-feec07c2c755"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:00:57.311158 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:57.311122    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f43cdb3b-9bd6-46a7-a14c-feec07c2c755-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f43cdb3b-9bd6-46a7-a14c-feec07c2c755-credential-socket\") pod \"f43cdb3b-9bd6-46a7-a14c-feec07c2c755\" (UID: \"f43cdb3b-9bd6-46a7-a14c-feec07c2c755\") "
Oct 31 13:00:57.311249 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:57.311195    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f43cdb3b-9bd6-46a7-a14c-feec07c2c755-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f43cdb3b-9bd6-46a7-a14c-feec07c2c755-istio-envoy\") pod \"f43cdb3b-9bd6-46a7-a14c-feec07c2c755\" (UID: \"f43cdb3b-9bd6-46a7-a14c-feec07c2c755\") "
Oct 31 13:00:57.311249 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:57.311216    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f43cdb3b-9bd6-46a7-a14c-feec07c2c755-istio-data" pod "f43cdb3b-9bd6-46a7-a14c-feec07c2c755" (UID: "f43cdb3b-9bd6-46a7-a14c-feec07c2c755"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:00:57.311330 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:57.311248    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f43cdb3b-9bd6-46a7-a14c-feec07c2c755-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f43cdb3b-9bd6-46a7-a14c-feec07c2c755-workload-socket\") pod \"f43cdb3b-9bd6-46a7-a14c-feec07c2c755\" (UID: \"f43cdb3b-9bd6-46a7-a14c-feec07c2c755\") "
Oct 31 13:00:57.311330 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:57.311317    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f43cdb3b-9bd6-46a7-a14c-feec07c2c755-istio-token\" (UniqueName: \"kubernetes.io/projected/f43cdb3b-9bd6-46a7-a14c-feec07c2c755-istio-token\") pod \"f43cdb3b-9bd6-46a7-a14c-feec07c2c755\" (UID: \"f43cdb3b-9bd6-46a7-a14c-feec07c2c755\") "
Oct 31 13:00:57.311448 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:57.311329    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f43cdb3b-9bd6-46a7-a14c-feec07c2c755-credential-socket" pod "f43cdb3b-9bd6-46a7-a14c-feec07c2c755" (UID: "f43cdb3b-9bd6-46a7-a14c-feec07c2c755"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:00:57.311448 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:57.311349    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f43cdb3b-9bd6-46a7-a14c-feec07c2c755-kube-api-access-f7ksj\" (UniqueName: \"kubernetes.io/projected/f43cdb3b-9bd6-46a7-a14c-feec07c2c755-kube-api-access-f7ksj\") pod \"f43cdb3b-9bd6-46a7-a14c-feec07c2c755\" (UID: \"f43cdb3b-9bd6-46a7-a14c-feec07c2c755\") "
Oct 31 13:00:57.311448 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:57.311375    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f43cdb3b-9bd6-46a7-a14c-feec07c2c755-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f43cdb3b-9bd6-46a7-a14c-feec07c2c755-istiod-ca-cert\") pod \"f43cdb3b-9bd6-46a7-a14c-feec07c2c755\" (UID: \"f43cdb3b-9bd6-46a7-a14c-feec07c2c755\") "
Oct 31 13:00:57.311631 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:57.311553    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f43cdb3b-9bd6-46a7-a14c-feec07c2c755-workload-socket" pod "f43cdb3b-9bd6-46a7-a14c-feec07c2c755" (UID: "f43cdb3b-9bd6-46a7-a14c-feec07c2c755"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:00:57.311792 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:57.311772    5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f43cdb3b-9bd6-46a7-a14c-feec07c2c755-credential-socket\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:57.311862 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:57.311800    5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f43cdb3b-9bd6-46a7-a14c-feec07c2c755-workload-socket\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:57.311862 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:57.311816    5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f43cdb3b-9bd6-46a7-a14c-feec07c2c755-workload-certs\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:57.311862 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:57.311829    5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f43cdb3b-9bd6-46a7-a14c-feec07c2c755-istio-data\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:57.312089 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:57.311934    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f43cdb3b-9bd6-46a7-a14c-feec07c2c755-istiod-ca-cert" pod "f43cdb3b-9bd6-46a7-a14c-feec07c2c755" (UID: "f43cdb3b-9bd6-46a7-a14c-feec07c2c755"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:00:57.314049 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:57.314022    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f43cdb3b-9bd6-46a7-a14c-feec07c2c755-istio-podinfo" pod "f43cdb3b-9bd6-46a7-a14c-feec07c2c755" (UID: "f43cdb3b-9bd6-46a7-a14c-feec07c2c755"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 13:00:57.314353 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:57.314317    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f43cdb3b-9bd6-46a7-a14c-feec07c2c755-istio-token" pod "f43cdb3b-9bd6-46a7-a14c-feec07c2c755" (UID: "f43cdb3b-9bd6-46a7-a14c-feec07c2c755"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:00:57.314432 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:57.314351    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f43cdb3b-9bd6-46a7-a14c-feec07c2c755-kube-api-access-f7ksj" pod "f43cdb3b-9bd6-46a7-a14c-feec07c2c755" (UID: "f43cdb3b-9bd6-46a7-a14c-feec07c2c755"). InnerVolumeSpecName "kube-api-access-f7ksj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:00:57.314529 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:57.314467    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f43cdb3b-9bd6-46a7-a14c-feec07c2c755-istio-envoy" pod "f43cdb3b-9bd6-46a7-a14c-feec07c2c755" (UID: "f43cdb3b-9bd6-46a7-a14c-feec07c2c755"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:00:57.412551 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:57.412513    5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f43cdb3b-9bd6-46a7-a14c-feec07c2c755-istio-podinfo\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:57.412551 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:57.412546    5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f43cdb3b-9bd6-46a7-a14c-feec07c2c755-istio-envoy\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:57.412551 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:57.412557    5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f43cdb3b-9bd6-46a7-a14c-feec07c2c755-istio-token\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:57.412795 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:57.412568    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7ksj\" (UniqueName: \"kubernetes.io/projected/f43cdb3b-9bd6-46a7-a14c-feec07c2c755-kube-api-access-f7ksj\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:57.412795 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:57.412580    5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f43cdb3b-9bd6-46a7-a14c-feec07c2c755-istiod-ca-cert\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:57.696977 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:57.696887    5060 generic.go:356] "Generic (PLEG): container finished" podID="f43cdb3b-9bd6-46a7-a14c-feec07c2c755" containerID="5dbbe131ebca981cfcb804c34551921810cfbf21be15aa15636d6888f9170867" exitCode=0
Oct 31 13:00:57.696977 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:57.696945    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-56grg"
Oct 31 13:00:57.697170 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:57.696982    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-56grg" event={"ID":"f43cdb3b-9bd6-46a7-a14c-feec07c2c755","Type":"ContainerDied","Data":"5dbbe131ebca981cfcb804c34551921810cfbf21be15aa15636d6888f9170867"}
Oct 31 13:00:57.697170 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:57.697054    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-56grg" event={"ID":"f43cdb3b-9bd6-46a7-a14c-feec07c2c755","Type":"ContainerDied","Data":"9cb68c72dafb6d570b2e653e09a2044fa290142ffbf15863c2addcc364af45e0"}
Oct 31 13:00:57.697170 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:57.697084    5060 scope.go:122] "RemoveContainer" containerID="5dbbe131ebca981cfcb804c34551921810cfbf21be15aa15636d6888f9170867"
Oct 31 13:00:57.708956 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:57.708938    5060 scope.go:122] "RemoveContainer" containerID="5dbbe131ebca981cfcb804c34551921810cfbf21be15aa15636d6888f9170867"
Oct 31 13:00:57.709239 ip-10-0-130-190 kubenswrapper[5060]: E1031 13:00:57.709219    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5dbbe131ebca981cfcb804c34551921810cfbf21be15aa15636d6888f9170867\": container with ID starting with 5dbbe131ebca981cfcb804c34551921810cfbf21be15aa15636d6888f9170867 not found: ID does not exist" containerID="5dbbe131ebca981cfcb804c34551921810cfbf21be15aa15636d6888f9170867"
Oct 31 13:00:57.709289 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:57.709247    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5dbbe131ebca981cfcb804c34551921810cfbf21be15aa15636d6888f9170867"} err="failed to get container status \"5dbbe131ebca981cfcb804c34551921810cfbf21be15aa15636d6888f9170867\": rpc error: code = NotFound desc = could not find container \"5dbbe131ebca981cfcb804c34551921810cfbf21be15aa15636d6888f9170867\": container with ID starting with 5dbbe131ebca981cfcb804c34551921810cfbf21be15aa15636d6888f9170867 not found: ID does not exist"
Oct 31 13:00:57.722899 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:57.722876    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-56grg"]
Oct 31 13:00:57.728090 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:57.728045    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-56grg"]
Oct 31 13:00:59.047103 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:59.047067    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f43cdb3b-9bd6-46a7-a14c-feec07c2c755" path="/var/lib/kubelet/pods/f43cdb3b-9bd6-46a7-a14c-feec07c2c755/volumes"
Oct 31 13:00:59.870068 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:59.870026    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-67d410-5c9cd96795-tvtnh" podUID="6772255c-c420-435e-8075-f52344d30bdb" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 13:00:59.870250 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:00:59.870031    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-67d410-6c74c4dc47-k6m8r" podUID="f85a5783-97ad-4058-bcdc-0981cea23afd" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 13:00:59.921783 ip-10-0-130-190 kubenswrapper[5060]: E1031 13:00:59.921740    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7phb" containerName="extractor"
Oct 31 13:01:02.607595 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:02.607555    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-67d410-2npv4"]
Oct 31 13:01:02.610191 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:02.610165    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-67d410-2npv4"]
Oct 31 13:01:03.049285 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:03.049247    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e13e1c42-cae4-4f43-badd-24293a62300d" path="/var/lib/kubelet/pods/e13e1c42-cae4-4f43-badd-24293a62300d/volumes"
Oct 31 13:01:09.452780 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:09.452742    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-rr5tg"]
Oct 31 13:01:09.453192 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:09.453015    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-rr5tg" podUID="8873fe2c-839e-4d23-83d1-1fe3a451b5cd" containerName="istio-proxy" containerID="cri-o://e0c1db5b4653ba85849774353da9a1a5d2c1fd751be6dca22e10e7926e304778" gracePeriod=30
Oct 31 13:01:09.870668 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:09.870625    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-67d410-6c74c4dc47-k6m8r" podUID="f85a5783-97ad-4058-bcdc-0981cea23afd" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 13:01:09.870856 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:09.870632    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-67d410-5c9cd96795-tvtnh" podUID="6772255c-c420-435e-8075-f52344d30bdb" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 13:01:10.615189 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:10.615150    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-665v2"]
Oct 31 13:01:10.615635 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:10.615347    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-665v2" podUID="982df9b0-91aa-407d-a838-85d26b8fd39f" containerName="discovery" containerID="cri-o://893e20820bd1cd47f78958ec0fd43f8a3c42451ebd6fe386b909888acbfb000c" gracePeriod=30
Oct 31 13:01:10.725097 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:10.725061    5060 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-665v2 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.171:8080/ready\": dial tcp 10.132.0.171:8080: connect: connection refused" start-of-body=
Oct 31 13:01:10.725237 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:10.725127    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-665v2" podUID="982df9b0-91aa-407d-a838-85d26b8fd39f" containerName="discovery" probeResult="failure" output="Get \"http://10.132.0.171:8080/ready\": dial tcp 10.132.0.171:8080: connect: connection refused"
Oct 31 13:01:10.809585 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:10.809370    5060 generic.go:356] "Generic (PLEG): container finished" podID="982df9b0-91aa-407d-a838-85d26b8fd39f" containerID="893e20820bd1cd47f78958ec0fd43f8a3c42451ebd6fe386b909888acbfb000c" exitCode=0
Oct 31 13:01:10.809585 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:10.809443    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-665v2" event={"ID":"982df9b0-91aa-407d-a838-85d26b8fd39f","Type":"ContainerDied","Data":"893e20820bd1cd47f78958ec0fd43f8a3c42451ebd6fe386b909888acbfb000c"}
Oct 31 13:01:10.885281 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:10.885253    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-665v2"
Oct 31 13:01:10.959384 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:10.959338    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/982df9b0-91aa-407d-a838-85d26b8fd39f-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/982df9b0-91aa-407d-a838-85d26b8fd39f-istio-kubeconfig\") pod \"982df9b0-91aa-407d-a838-85d26b8fd39f\" (UID: \"982df9b0-91aa-407d-a838-85d26b8fd39f\") "
Oct 31 13:01:10.959632 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:10.959419    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/982df9b0-91aa-407d-a838-85d26b8fd39f-kube-api-access-dmzjq\" (UniqueName: \"kubernetes.io/projected/982df9b0-91aa-407d-a838-85d26b8fd39f-kube-api-access-dmzjq\") pod \"982df9b0-91aa-407d-a838-85d26b8fd39f\" (UID: \"982df9b0-91aa-407d-a838-85d26b8fd39f\") "
Oct 31 13:01:10.959632 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:10.959464    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/982df9b0-91aa-407d-a838-85d26b8fd39f-istio-token\" (UniqueName: \"kubernetes.io/projected/982df9b0-91aa-407d-a838-85d26b8fd39f-istio-token\") pod \"982df9b0-91aa-407d-a838-85d26b8fd39f\" (UID: \"982df9b0-91aa-407d-a838-85d26b8fd39f\") "
Oct 31 13:01:10.959632 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:10.959521    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/982df9b0-91aa-407d-a838-85d26b8fd39f-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/982df9b0-91aa-407d-a838-85d26b8fd39f-istio-csr-ca-configmap\") pod \"982df9b0-91aa-407d-a838-85d26b8fd39f\" (UID: \"982df9b0-91aa-407d-a838-85d26b8fd39f\") "
Oct 31 13:01:10.959632 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:10.959549    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/982df9b0-91aa-407d-a838-85d26b8fd39f-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/982df9b0-91aa-407d-a838-85d26b8fd39f-local-certs\") pod \"982df9b0-91aa-407d-a838-85d26b8fd39f\" (UID: \"982df9b0-91aa-407d-a838-85d26b8fd39f\") "
Oct 31 13:01:10.959632 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:10.959584    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/982df9b0-91aa-407d-a838-85d26b8fd39f-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/982df9b0-91aa-407d-a838-85d26b8fd39f-istio-csr-dns-cert\") pod \"982df9b0-91aa-407d-a838-85d26b8fd39f\" (UID: \"982df9b0-91aa-407d-a838-85d26b8fd39f\") "
Oct 31 13:01:10.960050 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:10.959651    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/982df9b0-91aa-407d-a838-85d26b8fd39f-cacerts\" (UniqueName: \"kubernetes.io/secret/982df9b0-91aa-407d-a838-85d26b8fd39f-cacerts\") pod \"982df9b0-91aa-407d-a838-85d26b8fd39f\" (UID: \"982df9b0-91aa-407d-a838-85d26b8fd39f\") "
Oct 31 13:01:10.960237 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:10.960188    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/982df9b0-91aa-407d-a838-85d26b8fd39f-istio-csr-ca-configmap" pod "982df9b0-91aa-407d-a838-85d26b8fd39f" (UID: "982df9b0-91aa-407d-a838-85d26b8fd39f"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:01:10.962276 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:10.962235    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/982df9b0-91aa-407d-a838-85d26b8fd39f-istio-token" pod "982df9b0-91aa-407d-a838-85d26b8fd39f" (UID: "982df9b0-91aa-407d-a838-85d26b8fd39f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:01:10.962505 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:10.962460    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/982df9b0-91aa-407d-a838-85d26b8fd39f-kube-api-access-dmzjq" pod "982df9b0-91aa-407d-a838-85d26b8fd39f" (UID: "982df9b0-91aa-407d-a838-85d26b8fd39f"). InnerVolumeSpecName "kube-api-access-dmzjq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:01:10.962596 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:10.962541    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/982df9b0-91aa-407d-a838-85d26b8fd39f-cacerts" pod "982df9b0-91aa-407d-a838-85d26b8fd39f" (UID: "982df9b0-91aa-407d-a838-85d26b8fd39f"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:01:10.962706 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:10.962685    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/982df9b0-91aa-407d-a838-85d26b8fd39f-local-certs" pod "982df9b0-91aa-407d-a838-85d26b8fd39f" (UID: "982df9b0-91aa-407d-a838-85d26b8fd39f"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:01:10.962863 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:10.962844    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/982df9b0-91aa-407d-a838-85d26b8fd39f-istio-kubeconfig" pod "982df9b0-91aa-407d-a838-85d26b8fd39f" (UID: "982df9b0-91aa-407d-a838-85d26b8fd39f"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:01:10.963906 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:10.963877    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/982df9b0-91aa-407d-a838-85d26b8fd39f-istio-csr-dns-cert" pod "982df9b0-91aa-407d-a838-85d26b8fd39f" (UID: "982df9b0-91aa-407d-a838-85d26b8fd39f"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:01:11.061045 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:11.061016    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dmzjq\" (UniqueName: \"kubernetes.io/projected/982df9b0-91aa-407d-a838-85d26b8fd39f-kube-api-access-dmzjq\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:01:11.061045 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:11.061044    5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/982df9b0-91aa-407d-a838-85d26b8fd39f-istio-token\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:01:11.061273 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:11.061054    5060 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/982df9b0-91aa-407d-a838-85d26b8fd39f-istio-csr-ca-configmap\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:01:11.061273 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:11.061065    5060 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/982df9b0-91aa-407d-a838-85d26b8fd39f-local-certs\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:01:11.061273 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:11.061073    5060 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/982df9b0-91aa-407d-a838-85d26b8fd39f-istio-csr-dns-cert\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:01:11.061273 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:11.061082    5060 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/982df9b0-91aa-407d-a838-85d26b8fd39f-cacerts\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:01:11.061273 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:11.061090    5060 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/982df9b0-91aa-407d-a838-85d26b8fd39f-istio-kubeconfig\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:01:11.817825 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:11.817793    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-665v2"
Oct 31 13:01:11.820297 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:11.817820    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-665v2" event={"ID":"982df9b0-91aa-407d-a838-85d26b8fd39f","Type":"ContainerDied","Data":"978c81e2c75664f4231650d452d36cb12ceec8ab8309e31fb2f5d5869b331bfb"}
Oct 31 13:01:11.820297 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:11.817907    5060 scope.go:122] "RemoveContainer" containerID="893e20820bd1cd47f78958ec0fd43f8a3c42451ebd6fe386b909888acbfb000c"
Oct 31 13:01:11.838136 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:11.838108    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-665v2"]
Oct 31 13:01:11.842601 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:11.842572    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-665v2"]
Oct 31 13:01:12.194108 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:12.194081    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-67d410-6c74c4dc47-k6m8r"
Oct 31 13:01:12.222046 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:12.222020    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-67d410-5c9cd96795-tvtnh"
Oct 31 13:01:12.274519 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:12.274458    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f85a5783-97ad-4058-bcdc-0981cea23afd-kube-api-access-5l9kj\" (UniqueName: \"kubernetes.io/projected/f85a5783-97ad-4058-bcdc-0981cea23afd-kube-api-access-5l9kj\") pod \"f85a5783-97ad-4058-bcdc-0981cea23afd\" (UID: \"f85a5783-97ad-4058-bcdc-0981cea23afd\") "
Oct 31 13:01:12.274702 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:12.274576    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6772255c-c420-435e-8075-f52344d30bdb-kube-api-access-t4swg\" (UniqueName: \"kubernetes.io/projected/6772255c-c420-435e-8075-f52344d30bdb-kube-api-access-t4swg\") pod \"6772255c-c420-435e-8075-f52344d30bdb\" (UID: \"6772255c-c420-435e-8075-f52344d30bdb\") "
Oct 31 13:01:12.274702 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:12.274669    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f85a5783-97ad-4058-bcdc-0981cea23afd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f85a5783-97ad-4058-bcdc-0981cea23afd-plugins-config-volume\") pod \"f85a5783-97ad-4058-bcdc-0981cea23afd\" (UID: \"f85a5783-97ad-4058-bcdc-0981cea23afd\") "
Oct 31 13:01:12.275321 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:12.275288    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f85a5783-97ad-4058-bcdc-0981cea23afd-plugins-config-volume" pod "f85a5783-97ad-4058-bcdc-0981cea23afd" (UID: "f85a5783-97ad-4058-bcdc-0981cea23afd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:01:12.276836 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:12.276809    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6772255c-c420-435e-8075-f52344d30bdb-kube-api-access-t4swg" pod "6772255c-c420-435e-8075-f52344d30bdb" (UID: "6772255c-c420-435e-8075-f52344d30bdb"). InnerVolumeSpecName "kube-api-access-t4swg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:01:12.277076 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:12.277059    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f85a5783-97ad-4058-bcdc-0981cea23afd-kube-api-access-5l9kj" pod "f85a5783-97ad-4058-bcdc-0981cea23afd" (UID: "f85a5783-97ad-4058-bcdc-0981cea23afd"). InnerVolumeSpecName "kube-api-access-5l9kj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:01:12.376072 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:12.375985    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6772255c-c420-435e-8075-f52344d30bdb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6772255c-c420-435e-8075-f52344d30bdb-plugins-config-volume\") pod \"6772255c-c420-435e-8075-f52344d30bdb\" (UID: \"6772255c-c420-435e-8075-f52344d30bdb\") "
Oct 31 13:01:12.376337 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:12.376313    5060 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f85a5783-97ad-4058-bcdc-0981cea23afd-plugins-config-volume\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:01:12.376412 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:12.376344    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5l9kj\" (UniqueName: \"kubernetes.io/projected/f85a5783-97ad-4058-bcdc-0981cea23afd-kube-api-access-5l9kj\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:01:12.376412 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:12.376359    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4swg\" (UniqueName: \"kubernetes.io/projected/6772255c-c420-435e-8075-f52344d30bdb-kube-api-access-t4swg\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:01:12.376412 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:12.376394    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6772255c-c420-435e-8075-f52344d30bdb-plugins-config-volume" pod "6772255c-c420-435e-8075-f52344d30bdb" (UID: "6772255c-c420-435e-8075-f52344d30bdb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:01:12.477317 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:12.477279    5060 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6772255c-c420-435e-8075-f52344d30bdb-plugins-config-volume\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:01:12.826580 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:12.826543    5060 generic.go:356] "Generic (PLEG): container finished" podID="f85a5783-97ad-4058-bcdc-0981cea23afd" containerID="4e2808ace3cd5f2e086bd0fae56e55008f333403e24023588a1e7d5238b9c297" exitCode=0
Oct 31 13:01:12.827071 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:12.826595    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-67d410-6c74c4dc47-k6m8r"
Oct 31 13:01:12.827071 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:12.826630    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-67d410-6c74c4dc47-k6m8r" event={"ID":"f85a5783-97ad-4058-bcdc-0981cea23afd","Type":"ContainerDied","Data":"4e2808ace3cd5f2e086bd0fae56e55008f333403e24023588a1e7d5238b9c297"}
Oct 31 13:01:12.827071 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:12.826686    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-67d410-6c74c4dc47-k6m8r" event={"ID":"f85a5783-97ad-4058-bcdc-0981cea23afd","Type":"ContainerDied","Data":"1519ac1298dcc41c7d47fde810ef9ef87c09a8d685ebf6cff36a40a611b0f056"}
Oct 31 13:01:12.827071 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:12.826716    5060 scope.go:122] "RemoveContainer" containerID="4e2808ace3cd5f2e086bd0fae56e55008f333403e24023588a1e7d5238b9c297"
Oct 31 13:01:12.828171 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:12.828147    5060 generic.go:356] "Generic (PLEG): container finished" podID="6772255c-c420-435e-8075-f52344d30bdb" containerID="40634ad90f586d271990f309270efe61e24804a62b4c02e119264e9dca9287e2" exitCode=0
Oct 31 13:01:12.828289 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:12.828189    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-67d410-5c9cd96795-tvtnh"
Oct 31 13:01:12.828289 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:12.828235    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-67d410-5c9cd96795-tvtnh" event={"ID":"6772255c-c420-435e-8075-f52344d30bdb","Type":"ContainerDied","Data":"40634ad90f586d271990f309270efe61e24804a62b4c02e119264e9dca9287e2"}
Oct 31 13:01:12.828289 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:12.828285    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-67d410-5c9cd96795-tvtnh" event={"ID":"6772255c-c420-435e-8075-f52344d30bdb","Type":"ContainerDied","Data":"a2e547320249af02a43494d9d9904dbe1587e0bc756591b2e0bdc5b5263b5072"}
Oct 31 13:01:12.838861 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:12.838841    5060 scope.go:122] "RemoveContainer" containerID="4e2808ace3cd5f2e086bd0fae56e55008f333403e24023588a1e7d5238b9c297"
Oct 31 13:01:12.839136 ip-10-0-130-190 kubenswrapper[5060]: E1031 13:01:12.839114    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e2808ace3cd5f2e086bd0fae56e55008f333403e24023588a1e7d5238b9c297\": container with ID starting with 4e2808ace3cd5f2e086bd0fae56e55008f333403e24023588a1e7d5238b9c297 not found: ID does not exist" containerID="4e2808ace3cd5f2e086bd0fae56e55008f333403e24023588a1e7d5238b9c297"
Oct 31 13:01:12.839187 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:12.839145    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e2808ace3cd5f2e086bd0fae56e55008f333403e24023588a1e7d5238b9c297"} err="failed to get container status \"4e2808ace3cd5f2e086bd0fae56e55008f333403e24023588a1e7d5238b9c297\": rpc error: code = NotFound desc = could not find container \"4e2808ace3cd5f2e086bd0fae56e55008f333403e24023588a1e7d5238b9c297\": container with ID starting with 4e2808ace3cd5f2e086bd0fae56e55008f333403e24023588a1e7d5238b9c297 not found: ID does not exist"
Oct 31 13:01:12.839187 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:12.839163    5060 scope.go:122] "RemoveContainer" containerID="40634ad90f586d271990f309270efe61e24804a62b4c02e119264e9dca9287e2"
Oct 31 13:01:12.849540 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:12.849520    5060 scope.go:122] "RemoveContainer" containerID="40634ad90f586d271990f309270efe61e24804a62b4c02e119264e9dca9287e2"
Oct 31 13:01:12.849832 ip-10-0-130-190 kubenswrapper[5060]: E1031 13:01:12.849805    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40634ad90f586d271990f309270efe61e24804a62b4c02e119264e9dca9287e2\": container with ID starting with 40634ad90f586d271990f309270efe61e24804a62b4c02e119264e9dca9287e2 not found: ID does not exist" containerID="40634ad90f586d271990f309270efe61e24804a62b4c02e119264e9dca9287e2"
Oct 31 13:01:12.849888 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:12.849840    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40634ad90f586d271990f309270efe61e24804a62b4c02e119264e9dca9287e2"} err="failed to get container status \"40634ad90f586d271990f309270efe61e24804a62b4c02e119264e9dca9287e2\": rpc error: code = NotFound desc = could not find container \"40634ad90f586d271990f309270efe61e24804a62b4c02e119264e9dca9287e2\": container with ID starting with 40634ad90f586d271990f309270efe61e24804a62b4c02e119264e9dca9287e2 not found: ID does not exist"
Oct 31 13:01:12.854002 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:12.853979    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-67d410-5c9cd96795-tvtnh"]
Oct 31 13:01:12.858159 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:12.858133    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-67d410-5c9cd96795-tvtnh"]
Oct 31 13:01:12.868705 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:12.868683    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-67d410-6c74c4dc47-k6m8r"]
Oct 31 13:01:12.879821 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:12.879796    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-67d410-6c74c4dc47-k6m8r"]
Oct 31 13:01:13.047892 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:13.047845    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6772255c-c420-435e-8075-f52344d30bdb" path="/var/lib/kubelet/pods/6772255c-c420-435e-8075-f52344d30bdb/volumes"
Oct 31 13:01:13.048627 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:13.048602    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="982df9b0-91aa-407d-a838-85d26b8fd39f" path="/var/lib/kubelet/pods/982df9b0-91aa-407d-a838-85d26b8fd39f/volumes"
Oct 31 13:01:13.049229 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:13.049206    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f85a5783-97ad-4058-bcdc-0981cea23afd" path="/var/lib/kubelet/pods/f85a5783-97ad-4058-bcdc-0981cea23afd/volumes"
Oct 31 13:01:14.040664 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:14.040623    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-64d9577675-87ftl"]
Oct 31 13:01:14.041328 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:14.041309    5060 cpu_manager.go:414] "containerMap: removing container" podUID="6772255c-c420-435e-8075-f52344d30bdb" containerName="payload-pre-processing"
Oct 31 13:01:14.041374 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:14.041331    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6772255c-c420-435e-8075-f52344d30bdb" containerName="payload-pre-processing"
Oct 31 13:01:14.041374 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:14.041342    5060 cpu_manager.go:414] "containerMap: removing container" podUID="f85a5783-97ad-4058-bcdc-0981cea23afd" containerName="payload-processing"
Oct 31 13:01:14.041374 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:14.041347    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f85a5783-97ad-4058-bcdc-0981cea23afd" containerName="payload-processing"
Oct 31 13:01:14.041374 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:14.041355    5060 cpu_manager.go:414] "containerMap: removing container" podUID="982df9b0-91aa-407d-a838-85d26b8fd39f" containerName="discovery"
Oct 31 13:01:14.041374 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:14.041360    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="982df9b0-91aa-407d-a838-85d26b8fd39f" containerName="discovery"
Oct 31 13:01:14.041374 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:14.041371    5060 cpu_manager.go:414] "containerMap: removing container" podUID="48f68ea6-2cb0-4e83-8d61-db1532be4735" containerName="revoke-keys"
Oct 31 13:01:14.041374 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:14.041375    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48f68ea6-2cb0-4e83-8d61-db1532be4735" containerName="revoke-keys"
Oct 31 13:01:14.041609 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:14.041391    5060 cpu_manager.go:414] "containerMap: removing container" podUID="a6fa2170-8276-4138-9614-551ddc387fb8" containerName="payload-processing"
Oct 31 13:01:14.041609 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:14.041397    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6fa2170-8276-4138-9614-551ddc387fb8" containerName="payload-processing"
Oct 31 13:01:14.041609 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:14.041406    5060 cpu_manager.go:414] "containerMap: removing container" podUID="f43cdb3b-9bd6-46a7-a14c-feec07c2c755" containerName="istio-proxy"
Oct 31 13:01:14.041609 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:14.041410    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f43cdb3b-9bd6-46a7-a14c-feec07c2c755" containerName="istio-proxy"
Oct 31 13:01:14.041609 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:14.041417    5060 cpu_manager.go:414] "containerMap: removing container" podUID="d496cda1-2b40-48f1-8d49-a5ecd7f28a2b" containerName="payload-pre-processing"
Oct 31 13:01:14.041609 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:14.041423    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d496cda1-2b40-48f1-8d49-a5ecd7f28a2b" containerName="payload-pre-processing"
Oct 31 13:01:14.041609 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:14.041505    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6fa2170-8276-4138-9614-551ddc387fb8" containerName="payload-processing"
Oct 31 13:01:14.041609 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:14.041518    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="f43cdb3b-9bd6-46a7-a14c-feec07c2c755" containerName="istio-proxy"
Oct 31 13:01:14.041609 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:14.041527    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="d496cda1-2b40-48f1-8d49-a5ecd7f28a2b" containerName="payload-pre-processing"
Oct 31 13:01:14.041609 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:14.041537    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="6772255c-c420-435e-8075-f52344d30bdb" containerName="payload-pre-processing"
Oct 31 13:01:14.041609 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:14.041544    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="f85a5783-97ad-4058-bcdc-0981cea23afd" containerName="payload-processing"
Oct 31 13:01:14.041609 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:14.041549    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="982df9b0-91aa-407d-a838-85d26b8fd39f" containerName="discovery"
Oct 31 13:01:14.041609 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:14.041554    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="48f68ea6-2cb0-4e83-8d61-db1532be4735" containerName="revoke-keys"
Oct 31 13:01:14.046088 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:14.046072    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-64d9577675-87ftl"
Oct 31 13:01:14.053936 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:14.053909    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-t8k9s\""
Oct 31 13:01:14.053936 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:14.053913    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Oct 31 13:01:14.054131 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:14.053993    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Oct 31 13:01:14.057148 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:14.057117    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-64d9577675-87ftl"]
Oct 31 13:01:14.063042 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:14.059897    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Oct 31 13:01:14.197517 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:14.195134    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txfnw\" (UniqueName: \"kubernetes.io/projected/6f51322f-0910-43aa-a5d9-e34ba479e813-kube-api-access-txfnw\") pod \"maas-controller-64d9577675-87ftl\" (UID: \"6f51322f-0910-43aa-a5d9-e34ba479e813\") " pod="opendatahub/maas-controller-64d9577675-87ftl"
Oct 31 13:01:14.197517 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:14.195202    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f51322f-0910-43aa-a5d9-e34ba479e813-metrics-certs\") pod \"maas-controller-64d9577675-87ftl\" (UID: \"6f51322f-0910-43aa-a5d9-e34ba479e813\") " pod="opendatahub/maas-controller-64d9577675-87ftl"
Oct 31 13:01:14.197517 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:14.195253    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6f51322f-0910-43aa-a5d9-e34ba479e813-webhook-cert\") pod \"maas-controller-64d9577675-87ftl\" (UID: \"6f51322f-0910-43aa-a5d9-e34ba479e813\") " pod="opendatahub/maas-controller-64d9577675-87ftl"
Oct 31 13:01:14.301573 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:14.301315    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txfnw\" (UniqueName: \"kubernetes.io/projected/6f51322f-0910-43aa-a5d9-e34ba479e813-kube-api-access-txfnw\") pod \"maas-controller-64d9577675-87ftl\" (UID: \"6f51322f-0910-43aa-a5d9-e34ba479e813\") " pod="opendatahub/maas-controller-64d9577675-87ftl"
Oct 31 13:01:14.301573 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:14.301381    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f51322f-0910-43aa-a5d9-e34ba479e813-metrics-certs\") pod \"maas-controller-64d9577675-87ftl\" (UID: \"6f51322f-0910-43aa-a5d9-e34ba479e813\") " pod="opendatahub/maas-controller-64d9577675-87ftl"
Oct 31 13:01:14.301573 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:14.301422    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6f51322f-0910-43aa-a5d9-e34ba479e813-webhook-cert\") pod \"maas-controller-64d9577675-87ftl\" (UID: \"6f51322f-0910-43aa-a5d9-e34ba479e813\") " pod="opendatahub/maas-controller-64d9577675-87ftl"
Oct 31 13:01:14.327143 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:14.326983    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6f51322f-0910-43aa-a5d9-e34ba479e813-webhook-cert\") pod \"maas-controller-64d9577675-87ftl\" (UID: \"6f51322f-0910-43aa-a5d9-e34ba479e813\") " pod="opendatahub/maas-controller-64d9577675-87ftl"
Oct 31 13:01:14.328067 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:14.328022    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-txfnw\" (UniqueName: \"kubernetes.io/projected/6f51322f-0910-43aa-a5d9-e34ba479e813-kube-api-access-txfnw\") pod \"maas-controller-64d9577675-87ftl\" (UID: \"6f51322f-0910-43aa-a5d9-e34ba479e813\") " pod="opendatahub/maas-controller-64d9577675-87ftl"
Oct 31 13:01:14.328067 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:14.328022    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f51322f-0910-43aa-a5d9-e34ba479e813-metrics-certs\") pod \"maas-controller-64d9577675-87ftl\" (UID: \"6f51322f-0910-43aa-a5d9-e34ba479e813\") " pod="opendatahub/maas-controller-64d9577675-87ftl"
Oct 31 13:01:14.368593 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:14.368255    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-64d9577675-87ftl"
Oct 31 13:01:14.735521 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:14.731607    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-64d9577675-87ftl"]
Oct 31 13:01:14.743207 ip-10-0-130-190 kubenswrapper[5060]: W1031 13:01:14.743173    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f51322f_0910_43aa_a5d9_e34ba479e813.slice/crio-79cf78835da2488673ea01df873e29bae574a027d1117b16555606e36221e3e1 WatchSource:0}: Error finding container 79cf78835da2488673ea01df873e29bae574a027d1117b16555606e36221e3e1: Status 404 returned error can't find the container with id 79cf78835da2488673ea01df873e29bae574a027d1117b16555606e36221e3e1
Oct 31 13:01:14.744515 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:14.744477    5060 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 13:01:14.748143 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:14.748123    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-rr5tg"
Oct 31 13:01:14.851592 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:14.851472    5060 generic.go:356] "Generic (PLEG): container finished" podID="8873fe2c-839e-4d23-83d1-1fe3a451b5cd" containerID="e0c1db5b4653ba85849774353da9a1a5d2c1fd751be6dca22e10e7926e304778" exitCode=0
Oct 31 13:01:14.851592 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:14.851551    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-rr5tg"
Oct 31 13:01:14.851592 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:14.851557    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-rr5tg" event={"ID":"8873fe2c-839e-4d23-83d1-1fe3a451b5cd","Type":"ContainerDied","Data":"e0c1db5b4653ba85849774353da9a1a5d2c1fd751be6dca22e10e7926e304778"}
Oct 31 13:01:14.851880 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:14.851627    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-rr5tg" event={"ID":"8873fe2c-839e-4d23-83d1-1fe3a451b5cd","Type":"ContainerDied","Data":"3475b16fe741a8d8cadfea0dbcb62e45d397853074e63dd362b8710f5bfa6b00"}
Oct 31 13:01:14.851880 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:14.851659    5060 scope.go:122] "RemoveContainer" containerID="e0c1db5b4653ba85849774353da9a1a5d2c1fd751be6dca22e10e7926e304778"
Oct 31 13:01:14.853054 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:14.853026    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-64d9577675-87ftl" event={"ID":"6f51322f-0910-43aa-a5d9-e34ba479e813","Type":"ContainerStarted","Data":"79cf78835da2488673ea01df873e29bae574a027d1117b16555606e36221e3e1"}
Oct 31 13:01:14.896836 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:14.896123    5060 scope.go:122] "RemoveContainer" containerID="e0c1db5b4653ba85849774353da9a1a5d2c1fd751be6dca22e10e7926e304778"
Oct 31 13:01:14.896836 ip-10-0-130-190 kubenswrapper[5060]: E1031 13:01:14.896660    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0c1db5b4653ba85849774353da9a1a5d2c1fd751be6dca22e10e7926e304778\": container with ID starting with e0c1db5b4653ba85849774353da9a1a5d2c1fd751be6dca22e10e7926e304778 not found: ID does not exist" containerID="e0c1db5b4653ba85849774353da9a1a5d2c1fd751be6dca22e10e7926e304778"
Oct 31 13:01:14.896836 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:14.896702    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0c1db5b4653ba85849774353da9a1a5d2c1fd751be6dca22e10e7926e304778"} err="failed to get container status \"e0c1db5b4653ba85849774353da9a1a5d2c1fd751be6dca22e10e7926e304778\": rpc error: code = NotFound desc = could not find container \"e0c1db5b4653ba85849774353da9a1a5d2c1fd751be6dca22e10e7926e304778\": container with ID starting with e0c1db5b4653ba85849774353da9a1a5d2c1fd751be6dca22e10e7926e304778 not found: ID does not exist"
Oct 31 13:01:14.932365 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:14.932333    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8873fe2c-839e-4d23-83d1-1fe3a451b5cd-kube-api-access-v8jcz\" (UniqueName: \"kubernetes.io/projected/8873fe2c-839e-4d23-83d1-1fe3a451b5cd-kube-api-access-v8jcz\") pod \"8873fe2c-839e-4d23-83d1-1fe3a451b5cd\" (UID: \"8873fe2c-839e-4d23-83d1-1fe3a451b5cd\") "
Oct 31 13:01:14.932574 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:14.932370    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8873fe2c-839e-4d23-83d1-1fe3a451b5cd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8873fe2c-839e-4d23-83d1-1fe3a451b5cd-istiod-ca-cert\") pod \"8873fe2c-839e-4d23-83d1-1fe3a451b5cd\" (UID: \"8873fe2c-839e-4d23-83d1-1fe3a451b5cd\") "
Oct 31 13:01:14.932574 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:14.932396    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8873fe2c-839e-4d23-83d1-1fe3a451b5cd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8873fe2c-839e-4d23-83d1-1fe3a451b5cd-credential-socket\") pod \"8873fe2c-839e-4d23-83d1-1fe3a451b5cd\" (UID: \"8873fe2c-839e-4d23-83d1-1fe3a451b5cd\") "
Oct 31 13:01:14.932574 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:14.932414    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8873fe2c-839e-4d23-83d1-1fe3a451b5cd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8873fe2c-839e-4d23-83d1-1fe3a451b5cd-workload-certs\") pod \"8873fe2c-839e-4d23-83d1-1fe3a451b5cd\" (UID: \"8873fe2c-839e-4d23-83d1-1fe3a451b5cd\") "
Oct 31 13:01:14.932574 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:14.932502    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8873fe2c-839e-4d23-83d1-1fe3a451b5cd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8873fe2c-839e-4d23-83d1-1fe3a451b5cd-istio-data\") pod \"8873fe2c-839e-4d23-83d1-1fe3a451b5cd\" (UID: \"8873fe2c-839e-4d23-83d1-1fe3a451b5cd\") "
Oct 31 13:01:14.932574 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:14.932559    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8873fe2c-839e-4d23-83d1-1fe3a451b5cd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8873fe2c-839e-4d23-83d1-1fe3a451b5cd-istio-podinfo\") pod \"8873fe2c-839e-4d23-83d1-1fe3a451b5cd\" (UID: \"8873fe2c-839e-4d23-83d1-1fe3a451b5cd\") "
Oct 31 13:01:14.932875 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:14.932593    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8873fe2c-839e-4d23-83d1-1fe3a451b5cd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8873fe2c-839e-4d23-83d1-1fe3a451b5cd-istio-envoy\") pod \"8873fe2c-839e-4d23-83d1-1fe3a451b5cd\" (UID: \"8873fe2c-839e-4d23-83d1-1fe3a451b5cd\") "
Oct 31 13:01:14.932875 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:14.932677    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8873fe2c-839e-4d23-83d1-1fe3a451b5cd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8873fe2c-839e-4d23-83d1-1fe3a451b5cd-workload-socket\") pod \"8873fe2c-839e-4d23-83d1-1fe3a451b5cd\" (UID: \"8873fe2c-839e-4d23-83d1-1fe3a451b5cd\") "
Oct 31 13:01:14.932875 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:14.932724    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8873fe2c-839e-4d23-83d1-1fe3a451b5cd-istio-token\" (UniqueName: \"kubernetes.io/projected/8873fe2c-839e-4d23-83d1-1fe3a451b5cd-istio-token\") pod \"8873fe2c-839e-4d23-83d1-1fe3a451b5cd\" (UID: \"8873fe2c-839e-4d23-83d1-1fe3a451b5cd\") "
Oct 31 13:01:14.932875 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:14.932765    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8873fe2c-839e-4d23-83d1-1fe3a451b5cd-credential-socket" pod "8873fe2c-839e-4d23-83d1-1fe3a451b5cd" (UID: "8873fe2c-839e-4d23-83d1-1fe3a451b5cd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:01:14.933103 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:14.932875    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8873fe2c-839e-4d23-83d1-1fe3a451b5cd-istio-data" pod "8873fe2c-839e-4d23-83d1-1fe3a451b5cd" (UID: "8873fe2c-839e-4d23-83d1-1fe3a451b5cd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:01:14.933103 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:14.932906    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8873fe2c-839e-4d23-83d1-1fe3a451b5cd-istiod-ca-cert" pod "8873fe2c-839e-4d23-83d1-1fe3a451b5cd" (UID: "8873fe2c-839e-4d23-83d1-1fe3a451b5cd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:01:14.933103 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:14.932989    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8873fe2c-839e-4d23-83d1-1fe3a451b5cd-workload-socket" pod "8873fe2c-839e-4d23-83d1-1fe3a451b5cd" (UID: "8873fe2c-839e-4d23-83d1-1fe3a451b5cd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:01:14.933284 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:14.933129    5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8873fe2c-839e-4d23-83d1-1fe3a451b5cd-istio-data\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:01:14.933284 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:14.933145    5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8873fe2c-839e-4d23-83d1-1fe3a451b5cd-workload-socket\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:01:14.933284 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:14.933157    5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8873fe2c-839e-4d23-83d1-1fe3a451b5cd-istiod-ca-cert\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:01:14.933284 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:14.933166    5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8873fe2c-839e-4d23-83d1-1fe3a451b5cd-credential-socket\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:01:14.933284 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:14.933159    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8873fe2c-839e-4d23-83d1-1fe3a451b5cd-workload-certs" pod "8873fe2c-839e-4d23-83d1-1fe3a451b5cd" (UID: "8873fe2c-839e-4d23-83d1-1fe3a451b5cd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:01:14.934869 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:14.934839    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8873fe2c-839e-4d23-83d1-1fe3a451b5cd-istio-envoy" pod "8873fe2c-839e-4d23-83d1-1fe3a451b5cd" (UID: "8873fe2c-839e-4d23-83d1-1fe3a451b5cd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:01:14.934985 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:14.934904    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8873fe2c-839e-4d23-83d1-1fe3a451b5cd-istio-podinfo" pod "8873fe2c-839e-4d23-83d1-1fe3a451b5cd" (UID: "8873fe2c-839e-4d23-83d1-1fe3a451b5cd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 13:01:14.934985 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:14.934922    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8873fe2c-839e-4d23-83d1-1fe3a451b5cd-istio-token" pod "8873fe2c-839e-4d23-83d1-1fe3a451b5cd" (UID: "8873fe2c-839e-4d23-83d1-1fe3a451b5cd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:01:14.934985 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:14.934912    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8873fe2c-839e-4d23-83d1-1fe3a451b5cd-kube-api-access-v8jcz" pod "8873fe2c-839e-4d23-83d1-1fe3a451b5cd" (UID: "8873fe2c-839e-4d23-83d1-1fe3a451b5cd"). InnerVolumeSpecName "kube-api-access-v8jcz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:01:15.033795 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:15.033752    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8jcz\" (UniqueName: \"kubernetes.io/projected/8873fe2c-839e-4d23-83d1-1fe3a451b5cd-kube-api-access-v8jcz\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:01:15.033795 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:15.033789    5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8873fe2c-839e-4d23-83d1-1fe3a451b5cd-workload-certs\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:01:15.033795 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:15.033799    5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8873fe2c-839e-4d23-83d1-1fe3a451b5cd-istio-podinfo\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:01:15.033795 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:15.033808    5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8873fe2c-839e-4d23-83d1-1fe3a451b5cd-istio-envoy\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:01:15.034076 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:15.033817    5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8873fe2c-839e-4d23-83d1-1fe3a451b5cd-istio-token\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:01:15.172224 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:15.172135    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-rr5tg"]
Oct 31 13:01:15.177932 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:15.177903    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-rr5tg"]
Oct 31 13:01:17.047975 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:17.047948    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8873fe2c-839e-4d23-83d1-1fe3a451b5cd" path="/var/lib/kubelet/pods/8873fe2c-839e-4d23-83d1-1fe3a451b5cd/volumes"
Oct 31 13:01:17.878864 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:17.878827    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-64d9577675-87ftl" event={"ID":"6f51322f-0910-43aa-a5d9-e34ba479e813","Type":"ContainerStarted","Data":"da0b54cf2e2e9fc62286e3cc33f60fa87eed48c20670179bd3697521e068fc0a"}
Oct 31 13:01:17.879078 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:17.878945    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-64d9577675-87ftl"
Oct 31 13:01:17.902371 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:17.902320    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-64d9577675-87ftl" podStartSLOduration=1.6536670089999999 podStartE2EDuration="3.902307751s" totalImagesPullingTime="2.248640742s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:01:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:01:17.898721324 +0000 UTC m=+2041.001789039" watchObservedRunningTime="2026-10-31 13:01:17.902307751 +0000 UTC m=+2041.005375464"
Oct 31 13:01:26.481902 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:26.481862    5060 scope.go:122] "RemoveContainer" containerID="7c7ae659a71f10768cffcbb8ddebdb945b3a38814ceb9134d99f0a663d784c92"
Oct 31 13:01:26.491697 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:26.491674    5060 scope.go:122] "RemoveContainer" containerID="3779cdba2e95eaba88ec5a09f2dcf6f2bfafa3160cf3f673b512cbb32005ef08"
Oct 31 13:01:26.500284 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:26.500255    5060 scope.go:122] "RemoveContainer" containerID="76055c9723c7a18e8a7e5da4e772dcfdb52e9d700b7affd448a76e0913e2c078"
Oct 31 13:01:26.508839 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:26.508820    5060 scope.go:122] "RemoveContainer" containerID="24c14e673ccf5d91faeafd6d8df3d67096da5b59c3817940bf602dfa94c7af6f"
Oct 31 13:01:26.516936 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:26.516910    5060 scope.go:122] "RemoveContainer" containerID="bef6629e9804a28bc44e435352e7dd68d4480f04713f5aed4f01142413cd271e"
Oct 31 13:01:26.525297 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:26.525273    5060 scope.go:122] "RemoveContainer" containerID="25a4c056e2d959b3e485921e82bc2a0f70a23fd8e3658d33626268655eaa0ed6"
Oct 31 13:01:28.888191 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:28.888159    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-64d9577675-87ftl"
Oct 31 13:01:29.920715 ip-10-0-130-190 kubenswrapper[5060]: E1031 13:01:29.920676    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7phb" containerName="extractor"
Oct 31 13:01:34.298290 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:34.298244    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-t6spd"]
Oct 31 13:01:34.298801 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:34.298782    5060 cpu_manager.go:414] "containerMap: removing container" podUID="8873fe2c-839e-4d23-83d1-1fe3a451b5cd" containerName="istio-proxy"
Oct 31 13:01:34.298854 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:34.298803    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8873fe2c-839e-4d23-83d1-1fe3a451b5cd" containerName="istio-proxy"
Oct 31 13:01:34.298892 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:34.298879    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="8873fe2c-839e-4d23-83d1-1fe3a451b5cd" containerName="istio-proxy"
Oct 31 13:01:34.304365 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:34.304342    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-t6spd"
Oct 31 13:01:34.311751 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:34.311722    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-t6spd"]
Oct 31 13:01:34.312719 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:34.312695    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 13:01:34.312719 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:34.312712    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7wsmb\""
Oct 31 13:01:34.312937 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:34.312731    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 13:01:34.312937 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:34.312840    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 13:01:34.415843 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:34.415803    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6rqf\" (UniqueName: \"kubernetes.io/projected/d5c01d42-6758-451e-9937-2f5bf4b97264-kube-api-access-x6rqf\") pod \"maas-api-revoke-sub-85237e45-t6spd\" (UID: \"d5c01d42-6758-451e-9937-2f5bf4b97264\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-t6spd"
Oct 31 13:01:34.416000 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:34.415863    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5c01d42-6758-451e-9937-2f5bf4b97264-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-t6spd\" (UID: \"d5c01d42-6758-451e-9937-2f5bf4b97264\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-t6spd"
Oct 31 13:01:34.416000 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:34.415969    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5c01d42-6758-451e-9937-2f5bf4b97264-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-t6spd\" (UID: \"d5c01d42-6758-451e-9937-2f5bf4b97264\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-t6spd"
Oct 31 13:01:34.517289 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:34.517241    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5c01d42-6758-451e-9937-2f5bf4b97264-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-t6spd\" (UID: \"d5c01d42-6758-451e-9937-2f5bf4b97264\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-t6spd"
Oct 31 13:01:34.517455 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:34.517355    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5c01d42-6758-451e-9937-2f5bf4b97264-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-t6spd\" (UID: \"d5c01d42-6758-451e-9937-2f5bf4b97264\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-t6spd"
Oct 31 13:01:34.517455 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:34.517403    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6rqf\" (UniqueName: \"kubernetes.io/projected/d5c01d42-6758-451e-9937-2f5bf4b97264-kube-api-access-x6rqf\") pod \"maas-api-revoke-sub-85237e45-t6spd\" (UID: \"d5c01d42-6758-451e-9937-2f5bf4b97264\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-t6spd"
Oct 31 13:01:34.517935 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:34.517910    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5c01d42-6758-451e-9937-2f5bf4b97264-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-t6spd\" (UID: \"d5c01d42-6758-451e-9937-2f5bf4b97264\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-t6spd"
Oct 31 13:01:34.519822 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:34.519797    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5c01d42-6758-451e-9937-2f5bf4b97264-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-t6spd\" (UID: \"d5c01d42-6758-451e-9937-2f5bf4b97264\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-t6spd"
Oct 31 13:01:34.530466 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:34.530441    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6rqf\" (UniqueName: \"kubernetes.io/projected/d5c01d42-6758-451e-9937-2f5bf4b97264-kube-api-access-x6rqf\") pod \"maas-api-revoke-sub-85237e45-t6spd\" (UID: \"d5c01d42-6758-451e-9937-2f5bf4b97264\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-t6spd"
Oct 31 13:01:34.620602 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:34.620522    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-t6spd"
Oct 31 13:01:34.905576 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:34.905545    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-t6spd"]
Oct 31 13:01:34.907308 ip-10-0-130-190 kubenswrapper[5060]: W1031 13:01:34.907280    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5c01d42_6758_451e_9937_2f5bf4b97264.slice/crio-377f109b8faf573fe34be4e0f032b45a37555f67059726adee293731200e9296 WatchSource:0}: Error finding container 377f109b8faf573fe34be4e0f032b45a37555f67059726adee293731200e9296: Status 404 returned error can't find the container with id 377f109b8faf573fe34be4e0f032b45a37555f67059726adee293731200e9296
Oct 31 13:01:35.003339 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:35.003304    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-t6spd" event={"ID":"d5c01d42-6758-451e-9937-2f5bf4b97264","Type":"ContainerStarted","Data":"377f109b8faf573fe34be4e0f032b45a37555f67059726adee293731200e9296"}
Oct 31 13:01:36.011312 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:36.011278    5060 generic.go:356] "Generic (PLEG): container finished" podID="d5c01d42-6758-451e-9937-2f5bf4b97264" containerID="2dba03e2aa46a9afb3149277531c5664585b2bd6764c97ce2da43ebe7b1d634f" exitCode=0
Oct 31 13:01:36.011828 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:36.011370    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-t6spd" event={"ID":"d5c01d42-6758-451e-9937-2f5bf4b97264","Type":"ContainerDied","Data":"2dba03e2aa46a9afb3149277531c5664585b2bd6764c97ce2da43ebe7b1d634f"}
Oct 31 13:01:37.159873 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:37.159844    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-t6spd"
Oct 31 13:01:37.245802 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:37.245768    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d5c01d42-6758-451e-9937-2f5bf4b97264-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5c01d42-6758-451e-9937-2f5bf4b97264-maas-api-xks-ca\") pod \"d5c01d42-6758-451e-9937-2f5bf4b97264\" (UID: \"d5c01d42-6758-451e-9937-2f5bf4b97264\") "
Oct 31 13:01:37.245971 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:37.245852    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5c01d42-6758-451e-9937-2f5bf4b97264-kube-api-access-x6rqf\" (UniqueName: \"kubernetes.io/projected/d5c01d42-6758-451e-9937-2f5bf4b97264-kube-api-access-x6rqf\") pod \"d5c01d42-6758-451e-9937-2f5bf4b97264\" (UID: \"d5c01d42-6758-451e-9937-2f5bf4b97264\") "
Oct 31 13:01:37.245971 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:37.245898    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5c01d42-6758-451e-9937-2f5bf4b97264-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5c01d42-6758-451e-9937-2f5bf4b97264-maas-api-openshift-ca\") pod \"d5c01d42-6758-451e-9937-2f5bf4b97264\" (UID: \"d5c01d42-6758-451e-9937-2f5bf4b97264\") "
Oct 31 13:01:37.246266 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:37.246237    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5c01d42-6758-451e-9937-2f5bf4b97264-maas-api-openshift-ca" pod "d5c01d42-6758-451e-9937-2f5bf4b97264" (UID: "d5c01d42-6758-451e-9937-2f5bf4b97264"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:01:37.247899 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:37.247867    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5c01d42-6758-451e-9937-2f5bf4b97264-maas-api-xks-ca" pod "d5c01d42-6758-451e-9937-2f5bf4b97264" (UID: "d5c01d42-6758-451e-9937-2f5bf4b97264"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:01:37.248001 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:37.247986    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5c01d42-6758-451e-9937-2f5bf4b97264-kube-api-access-x6rqf" pod "d5c01d42-6758-451e-9937-2f5bf4b97264" (UID: "d5c01d42-6758-451e-9937-2f5bf4b97264"). InnerVolumeSpecName "kube-api-access-x6rqf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:01:37.347566 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:37.347535    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5c01d42-6758-451e-9937-2f5bf4b97264-maas-api-openshift-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:01:37.347566 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:37.347562    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5c01d42-6758-451e-9937-2f5bf4b97264-maas-api-xks-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:01:37.347566 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:37.347573    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x6rqf\" (UniqueName: \"kubernetes.io/projected/d5c01d42-6758-451e-9937-2f5bf4b97264-kube-api-access-x6rqf\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:01:38.027972 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:38.027935    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-t6spd" event={"ID":"d5c01d42-6758-451e-9937-2f5bf4b97264","Type":"ContainerDied","Data":"377f109b8faf573fe34be4e0f032b45a37555f67059726adee293731200e9296"}
Oct 31 13:01:38.028197 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:38.028024    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="377f109b8faf573fe34be4e0f032b45a37555f67059726adee293731200e9296"
Oct 31 13:01:38.028197 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:38.028118    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-t6spd"
Oct 31 13:01:45.945517 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:45.945468    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-64d9577675-87ftl"]
Oct 31 13:01:45.945951 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:45.945677    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-64d9577675-87ftl" podUID="6f51322f-0910-43aa-a5d9-e34ba479e813" containerName="manager" containerID="cri-o://da0b54cf2e2e9fc62286e3cc33f60fa87eed48c20670179bd3697521e068fc0a" gracePeriod=30
Oct 31 13:01:46.080130 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:46.080092    5060 generic.go:356] "Generic (PLEG): container finished" podID="6f51322f-0910-43aa-a5d9-e34ba479e813" containerID="da0b54cf2e2e9fc62286e3cc33f60fa87eed48c20670179bd3697521e068fc0a" exitCode=0
Oct 31 13:01:46.080271 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:46.080137    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-64d9577675-87ftl" event={"ID":"6f51322f-0910-43aa-a5d9-e34ba479e813","Type":"ContainerDied","Data":"da0b54cf2e2e9fc62286e3cc33f60fa87eed48c20670179bd3697521e068fc0a"}
Oct 31 13:01:46.197620 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:46.197555    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-64d9577675-87ftl"
Oct 31 13:01:46.330597 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:46.330561    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6f51322f-0910-43aa-a5d9-e34ba479e813-webhook-cert\" (UniqueName: \"kubernetes.io/secret/6f51322f-0910-43aa-a5d9-e34ba479e813-webhook-cert\") pod \"6f51322f-0910-43aa-a5d9-e34ba479e813\" (UID: \"6f51322f-0910-43aa-a5d9-e34ba479e813\") "
Oct 31 13:01:46.330809 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:46.330770    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f51322f-0910-43aa-a5d9-e34ba479e813-kube-api-access-txfnw\" (UniqueName: \"kubernetes.io/projected/6f51322f-0910-43aa-a5d9-e34ba479e813-kube-api-access-txfnw\") pod \"6f51322f-0910-43aa-a5d9-e34ba479e813\" (UID: \"6f51322f-0910-43aa-a5d9-e34ba479e813\") "
Oct 31 13:01:46.330905 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:46.330884    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6f51322f-0910-43aa-a5d9-e34ba479e813-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f51322f-0910-43aa-a5d9-e34ba479e813-metrics-certs\") pod \"6f51322f-0910-43aa-a5d9-e34ba479e813\" (UID: \"6f51322f-0910-43aa-a5d9-e34ba479e813\") "
Oct 31 13:01:46.332907 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:46.332878    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f51322f-0910-43aa-a5d9-e34ba479e813-kube-api-access-txfnw" pod "6f51322f-0910-43aa-a5d9-e34ba479e813" (UID: "6f51322f-0910-43aa-a5d9-e34ba479e813"). InnerVolumeSpecName "kube-api-access-txfnw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:01:46.333031 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:46.332903    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f51322f-0910-43aa-a5d9-e34ba479e813-webhook-cert" pod "6f51322f-0910-43aa-a5d9-e34ba479e813" (UID: "6f51322f-0910-43aa-a5d9-e34ba479e813"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:01:46.333660 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:46.333634    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f51322f-0910-43aa-a5d9-e34ba479e813-metrics-certs" pod "6f51322f-0910-43aa-a5d9-e34ba479e813" (UID: "6f51322f-0910-43aa-a5d9-e34ba479e813"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:01:46.431901 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:46.431866    5060 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6f51322f-0910-43aa-a5d9-e34ba479e813-webhook-cert\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:01:46.431901 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:46.431897    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-txfnw\" (UniqueName: \"kubernetes.io/projected/6f51322f-0910-43aa-a5d9-e34ba479e813-kube-api-access-txfnw\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:01:46.432103 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:46.431909    5060 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f51322f-0910-43aa-a5d9-e34ba479e813-metrics-certs\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:01:47.088340 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:47.088300    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-64d9577675-87ftl" event={"ID":"6f51322f-0910-43aa-a5d9-e34ba479e813","Type":"ContainerDied","Data":"79cf78835da2488673ea01df873e29bae574a027d1117b16555606e36221e3e1"}
Oct 31 13:01:47.088741 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:47.088360    5060 scope.go:122] "RemoveContainer" containerID="da0b54cf2e2e9fc62286e3cc33f60fa87eed48c20670179bd3697521e068fc0a"
Oct 31 13:01:47.088741 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:47.088317    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-64d9577675-87ftl"
Oct 31 13:01:47.118285 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:47.118253    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-64d9577675-87ftl"]
Oct 31 13:01:47.124161 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:47.124135    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-64d9577675-87ftl"]
Oct 31 13:01:47.131343 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:47.131317    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-rzth9"]
Oct 31 13:01:47.131765 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:47.131728    5060 cpu_manager.go:414] "containerMap: removing container" podUID="6f51322f-0910-43aa-a5d9-e34ba479e813" containerName="manager"
Oct 31 13:01:47.131812 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:47.131767    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f51322f-0910-43aa-a5d9-e34ba479e813" containerName="manager"
Oct 31 13:01:47.131812 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:47.131783    5060 cpu_manager.go:414] "containerMap: removing container" podUID="d5c01d42-6758-451e-9937-2f5bf4b97264" containerName="revoke-keys"
Oct 31 13:01:47.131812 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:47.131788    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5c01d42-6758-451e-9937-2f5bf4b97264" containerName="revoke-keys"
Oct 31 13:01:47.131934 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:47.131868    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f51322f-0910-43aa-a5d9-e34ba479e813" containerName="manager"
Oct 31 13:01:47.131934 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:47.131878    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5c01d42-6758-451e-9937-2f5bf4b97264" containerName="revoke-keys"
Oct 31 13:01:47.136071 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:47.136055    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-rzth9"
Oct 31 13:01:47.151439 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:47.151415    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-rzth9"]
Oct 31 13:01:47.185553 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:47.185518    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-mwc4r"]
Oct 31 13:01:47.189266 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:47.189248    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-mwc4r"
Oct 31 13:01:47.202610 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:47.202471    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-mwc4r"]
Oct 31 13:01:47.239610 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:47.239580    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d2738c9-9abf-47cf-a6d8-b83789055353-istio-envoy\") pod \"maas-default-gateway-openshift-default-645cf464d7-rzth9\" (UID: \"1d2738c9-9abf-47cf-a6d8-b83789055353\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-rzth9"
Oct 31 13:01:47.239752 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:47.239614    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vc9hv\" (UniqueName: \"kubernetes.io/projected/1d2738c9-9abf-47cf-a6d8-b83789055353-kube-api-access-vc9hv\") pod \"maas-default-gateway-openshift-default-645cf464d7-rzth9\" (UID: \"1d2738c9-9abf-47cf-a6d8-b83789055353\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-rzth9"
Oct 31 13:01:47.239752 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:47.239644    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d2738c9-9abf-47cf-a6d8-b83789055353-workload-socket\") pod \"maas-default-gateway-openshift-default-645cf464d7-rzth9\" (UID: \"1d2738c9-9abf-47cf-a6d8-b83789055353\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-rzth9"
Oct 31 13:01:47.239752 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:47.239707    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d2738c9-9abf-47cf-a6d8-b83789055353-istio-data\") pod \"maas-default-gateway-openshift-default-645cf464d7-rzth9\" (UID: \"1d2738c9-9abf-47cf-a6d8-b83789055353\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-rzth9"
Oct 31 13:01:47.239863 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:47.239815    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d2738c9-9abf-47cf-a6d8-b83789055353-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-645cf464d7-rzth9\" (UID: \"1d2738c9-9abf-47cf-a6d8-b83789055353\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-rzth9"
Oct 31 13:01:47.239863 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:47.239842    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d2738c9-9abf-47cf-a6d8-b83789055353-credential-socket\") pod \"maas-default-gateway-openshift-default-645cf464d7-rzth9\" (UID: \"1d2738c9-9abf-47cf-a6d8-b83789055353\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-rzth9"
Oct 31 13:01:47.239935 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:47.239922    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d2738c9-9abf-47cf-a6d8-b83789055353-istio-podinfo\") pod \"maas-default-gateway-openshift-default-645cf464d7-rzth9\" (UID: \"1d2738c9-9abf-47cf-a6d8-b83789055353\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-rzth9"
Oct 31 13:01:47.239975 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:47.239952    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d2738c9-9abf-47cf-a6d8-b83789055353-workload-certs\") pod \"maas-default-gateway-openshift-default-645cf464d7-rzth9\" (UID: \"1d2738c9-9abf-47cf-a6d8-b83789055353\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-rzth9"
Oct 31 13:01:47.240012 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:47.239977    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d2738c9-9abf-47cf-a6d8-b83789055353-istio-token\") pod \"maas-default-gateway-openshift-default-645cf464d7-rzth9\" (UID: \"1d2738c9-9abf-47cf-a6d8-b83789055353\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-rzth9"
Oct 31 13:01:47.340696 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:47.340623    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d2738c9-9abf-47cf-a6d8-b83789055353-workload-socket\") pod \"maas-default-gateway-openshift-default-645cf464d7-rzth9\" (UID: \"1d2738c9-9abf-47cf-a6d8-b83789055353\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-rzth9"
Oct 31 13:01:47.340696 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:47.340654    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d2738c9-9abf-47cf-a6d8-b83789055353-istio-data\") pod \"maas-default-gateway-openshift-default-645cf464d7-rzth9\" (UID: \"1d2738c9-9abf-47cf-a6d8-b83789055353\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-rzth9"
Oct 31 13:01:47.340696 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:47.340678    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51fa30eb-9214-4402-b911-28d4a2ff4a5e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-645cf464d7-mwc4r\" (UID: \"51fa30eb-9214-4402-b911-28d4a2ff4a5e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-mwc4r"
Oct 31 13:01:47.340931 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:47.340700    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51fa30eb-9214-4402-b911-28d4a2ff4a5e-credential-socket\") pod \"maas-default-gateway-openshift-default-645cf464d7-mwc4r\" (UID: \"51fa30eb-9214-4402-b911-28d4a2ff4a5e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-mwc4r"
Oct 31 13:01:47.340931 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:47.340775    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d2738c9-9abf-47cf-a6d8-b83789055353-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-645cf464d7-rzth9\" (UID: \"1d2738c9-9abf-47cf-a6d8-b83789055353\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-rzth9"
Oct 31 13:01:47.340931 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:47.340804    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d2738c9-9abf-47cf-a6d8-b83789055353-credential-socket\") pod \"maas-default-gateway-openshift-default-645cf464d7-rzth9\" (UID: \"1d2738c9-9abf-47cf-a6d8-b83789055353\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-rzth9"
Oct 31 13:01:47.340931 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:47.340838    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-879k4\" (UniqueName: \"kubernetes.io/projected/51fa30eb-9214-4402-b911-28d4a2ff4a5e-kube-api-access-879k4\") pod \"maas-default-gateway-openshift-default-645cf464d7-mwc4r\" (UID: \"51fa30eb-9214-4402-b911-28d4a2ff4a5e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-mwc4r"
Oct 31 13:01:47.340931 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:47.340855    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51fa30eb-9214-4402-b911-28d4a2ff4a5e-workload-socket\") pod \"maas-default-gateway-openshift-default-645cf464d7-mwc4r\" (UID: \"51fa30eb-9214-4402-b911-28d4a2ff4a5e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-mwc4r"
Oct 31 13:01:47.340931 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:47.340877    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51fa30eb-9214-4402-b911-28d4a2ff4a5e-istio-envoy\") pod \"maas-default-gateway-openshift-default-645cf464d7-mwc4r\" (UID: \"51fa30eb-9214-4402-b911-28d4a2ff4a5e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-mwc4r"
Oct 31 13:01:47.341206 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:47.341006    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d2738c9-9abf-47cf-a6d8-b83789055353-istio-podinfo\") pod \"maas-default-gateway-openshift-default-645cf464d7-rzth9\" (UID: \"1d2738c9-9abf-47cf-a6d8-b83789055353\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-rzth9"
Oct 31 13:01:47.341206 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:47.341039    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d2738c9-9abf-47cf-a6d8-b83789055353-workload-socket\") pod \"maas-default-gateway-openshift-default-645cf464d7-rzth9\" (UID: \"1d2738c9-9abf-47cf-a6d8-b83789055353\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-rzth9"
Oct 31 13:01:47.341206 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:47.341055    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d2738c9-9abf-47cf-a6d8-b83789055353-workload-certs\") pod \"maas-default-gateway-openshift-default-645cf464d7-rzth9\" (UID: \"1d2738c9-9abf-47cf-a6d8-b83789055353\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-rzth9"
Oct 31 13:01:47.341206 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:47.341087    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d2738c9-9abf-47cf-a6d8-b83789055353-istio-token\") pod \"maas-default-gateway-openshift-default-645cf464d7-rzth9\" (UID: \"1d2738c9-9abf-47cf-a6d8-b83789055353\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-rzth9"
Oct 31 13:01:47.341206 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:47.341095    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d2738c9-9abf-47cf-a6d8-b83789055353-istio-data\") pod \"maas-default-gateway-openshift-default-645cf464d7-rzth9\" (UID: \"1d2738c9-9abf-47cf-a6d8-b83789055353\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-rzth9"
Oct 31 13:01:47.341206 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:47.341132    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51fa30eb-9214-4402-b911-28d4a2ff4a5e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-645cf464d7-mwc4r\" (UID: \"51fa30eb-9214-4402-b911-28d4a2ff4a5e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-mwc4r"
Oct 31 13:01:47.341206 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:47.341166    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d2738c9-9abf-47cf-a6d8-b83789055353-credential-socket\") pod \"maas-default-gateway-openshift-default-645cf464d7-rzth9\" (UID: \"1d2738c9-9abf-47cf-a6d8-b83789055353\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-rzth9"
Oct 31 13:01:47.341206 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:47.341176    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51fa30eb-9214-4402-b911-28d4a2ff4a5e-istio-token\") pod \"maas-default-gateway-openshift-default-645cf464d7-mwc4r\" (UID: \"51fa30eb-9214-4402-b911-28d4a2ff4a5e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-mwc4r"
Oct 31 13:01:47.341206 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:47.341207    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51fa30eb-9214-4402-b911-28d4a2ff4a5e-workload-certs\") pod \"maas-default-gateway-openshift-default-645cf464d7-mwc4r\" (UID: \"51fa30eb-9214-4402-b911-28d4a2ff4a5e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-mwc4r"
Oct 31 13:01:47.341732 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:47.341293    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d2738c9-9abf-47cf-a6d8-b83789055353-istio-envoy\") pod \"maas-default-gateway-openshift-default-645cf464d7-rzth9\" (UID: \"1d2738c9-9abf-47cf-a6d8-b83789055353\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-rzth9"
Oct 31 13:01:47.341732 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:47.341350    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d2738c9-9abf-47cf-a6d8-b83789055353-workload-certs\") pod \"maas-default-gateway-openshift-default-645cf464d7-rzth9\" (UID: \"1d2738c9-9abf-47cf-a6d8-b83789055353\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-rzth9"
Oct 31 13:01:47.341732 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:47.341373    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vc9hv\" (UniqueName: \"kubernetes.io/projected/1d2738c9-9abf-47cf-a6d8-b83789055353-kube-api-access-vc9hv\") pod \"maas-default-gateway-openshift-default-645cf464d7-rzth9\" (UID: \"1d2738c9-9abf-47cf-a6d8-b83789055353\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-rzth9"
Oct 31 13:01:47.341732 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:47.341406    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51fa30eb-9214-4402-b911-28d4a2ff4a5e-istio-data\") pod \"maas-default-gateway-openshift-default-645cf464d7-mwc4r\" (UID: \"51fa30eb-9214-4402-b911-28d4a2ff4a5e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-mwc4r"
Oct 31 13:01:47.341732 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:47.341445    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d2738c9-9abf-47cf-a6d8-b83789055353-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-645cf464d7-rzth9\" (UID: \"1d2738c9-9abf-47cf-a6d8-b83789055353\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-rzth9"
Oct 31 13:01:47.343446 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:47.343421    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d2738c9-9abf-47cf-a6d8-b83789055353-istio-envoy\") pod \"maas-default-gateway-openshift-default-645cf464d7-rzth9\" (UID: \"1d2738c9-9abf-47cf-a6d8-b83789055353\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-rzth9"
Oct 31 13:01:47.343646 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:47.343626    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d2738c9-9abf-47cf-a6d8-b83789055353-istio-podinfo\") pod \"maas-default-gateway-openshift-default-645cf464d7-rzth9\" (UID: \"1d2738c9-9abf-47cf-a6d8-b83789055353\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-rzth9"
Oct 31 13:01:47.354085 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:47.354062    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vc9hv\" (UniqueName: \"kubernetes.io/projected/1d2738c9-9abf-47cf-a6d8-b83789055353-kube-api-access-vc9hv\") pod \"maas-default-gateway-openshift-default-645cf464d7-rzth9\" (UID: \"1d2738c9-9abf-47cf-a6d8-b83789055353\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-rzth9"
Oct 31 13:01:47.354721 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:47.354697    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d2738c9-9abf-47cf-a6d8-b83789055353-istio-token\") pod \"maas-default-gateway-openshift-default-645cf464d7-rzth9\" (UID: \"1d2738c9-9abf-47cf-a6d8-b83789055353\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-rzth9"
Oct 31 13:01:47.442508 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:47.442455    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51fa30eb-9214-4402-b911-28d4a2ff4a5e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-645cf464d7-mwc4r\" (UID: \"51fa30eb-9214-4402-b911-28d4a2ff4a5e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-mwc4r"
Oct 31 13:01:47.442673 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:47.442519    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51fa30eb-9214-4402-b911-28d4a2ff4a5e-istio-token\") pod \"maas-default-gateway-openshift-default-645cf464d7-mwc4r\" (UID: \"51fa30eb-9214-4402-b911-28d4a2ff4a5e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-mwc4r"
Oct 31 13:01:47.442673 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:47.442536    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51fa30eb-9214-4402-b911-28d4a2ff4a5e-workload-certs\") pod \"maas-default-gateway-openshift-default-645cf464d7-mwc4r\" (UID: \"51fa30eb-9214-4402-b911-28d4a2ff4a5e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-mwc4r"
Oct 31 13:01:47.442673 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:47.442561    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51fa30eb-9214-4402-b911-28d4a2ff4a5e-istio-data\") pod \"maas-default-gateway-openshift-default-645cf464d7-mwc4r\" (UID: \"51fa30eb-9214-4402-b911-28d4a2ff4a5e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-mwc4r"
Oct 31 13:01:47.442673 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:47.442587    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51fa30eb-9214-4402-b911-28d4a2ff4a5e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-645cf464d7-mwc4r\" (UID: \"51fa30eb-9214-4402-b911-28d4a2ff4a5e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-mwc4r"
Oct 31 13:01:47.442673 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:47.442607    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51fa30eb-9214-4402-b911-28d4a2ff4a5e-credential-socket\") pod \"maas-default-gateway-openshift-default-645cf464d7-mwc4r\" (UID: \"51fa30eb-9214-4402-b911-28d4a2ff4a5e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-mwc4r"
Oct 31 13:01:47.442673 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:47.442646    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-879k4\" (UniqueName: \"kubernetes.io/projected/51fa30eb-9214-4402-b911-28d4a2ff4a5e-kube-api-access-879k4\") pod \"maas-default-gateway-openshift-default-645cf464d7-mwc4r\" (UID: \"51fa30eb-9214-4402-b911-28d4a2ff4a5e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-mwc4r"
Oct 31 13:01:47.442673 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:47.442672    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51fa30eb-9214-4402-b911-28d4a2ff4a5e-workload-socket\") pod \"maas-default-gateway-openshift-default-645cf464d7-mwc4r\" (UID: \"51fa30eb-9214-4402-b911-28d4a2ff4a5e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-mwc4r"
Oct 31 13:01:47.443126 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:47.442704    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51fa30eb-9214-4402-b911-28d4a2ff4a5e-istio-envoy\") pod \"maas-default-gateway-openshift-default-645cf464d7-mwc4r\" (UID: \"51fa30eb-9214-4402-b911-28d4a2ff4a5e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-mwc4r"
Oct 31 13:01:47.443126 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:47.443085    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51fa30eb-9214-4402-b911-28d4a2ff4a5e-workload-certs\") pod \"maas-default-gateway-openshift-default-645cf464d7-mwc4r\" (UID: \"51fa30eb-9214-4402-b911-28d4a2ff4a5e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-mwc4r"
Oct 31 13:01:47.443249 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:47.443165    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51fa30eb-9214-4402-b911-28d4a2ff4a5e-workload-socket\") pod \"maas-default-gateway-openshift-default-645cf464d7-mwc4r\" (UID: \"51fa30eb-9214-4402-b911-28d4a2ff4a5e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-mwc4r"
Oct 31 13:01:47.443249 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:47.443200    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51fa30eb-9214-4402-b911-28d4a2ff4a5e-credential-socket\") pod \"maas-default-gateway-openshift-default-645cf464d7-mwc4r\" (UID: \"51fa30eb-9214-4402-b911-28d4a2ff4a5e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-mwc4r"
Oct 31 13:01:47.443368 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:47.443313    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51fa30eb-9214-4402-b911-28d4a2ff4a5e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-645cf464d7-mwc4r\" (UID: \"51fa30eb-9214-4402-b911-28d4a2ff4a5e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-mwc4r"
Oct 31 13:01:47.443368 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:47.443333    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51fa30eb-9214-4402-b911-28d4a2ff4a5e-istio-data\") pod \"maas-default-gateway-openshift-default-645cf464d7-mwc4r\" (UID: \"51fa30eb-9214-4402-b911-28d4a2ff4a5e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-mwc4r"
Oct 31 13:01:47.444985 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:47.444956    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51fa30eb-9214-4402-b911-28d4a2ff4a5e-istio-envoy\") pod \"maas-default-gateway-openshift-default-645cf464d7-mwc4r\" (UID: \"51fa30eb-9214-4402-b911-28d4a2ff4a5e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-mwc4r"
Oct 31 13:01:47.445066 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:47.445048    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51fa30eb-9214-4402-b911-28d4a2ff4a5e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-645cf464d7-mwc4r\" (UID: \"51fa30eb-9214-4402-b911-28d4a2ff4a5e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-mwc4r"
Oct 31 13:01:47.449383 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:47.449361    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-rzth9"
Oct 31 13:01:47.455668 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:47.455632    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51fa30eb-9214-4402-b911-28d4a2ff4a5e-istio-token\") pod \"maas-default-gateway-openshift-default-645cf464d7-mwc4r\" (UID: \"51fa30eb-9214-4402-b911-28d4a2ff4a5e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-mwc4r"
Oct 31 13:01:47.455819 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:47.455798    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-879k4\" (UniqueName: \"kubernetes.io/projected/51fa30eb-9214-4402-b911-28d4a2ff4a5e-kube-api-access-879k4\") pod \"maas-default-gateway-openshift-default-645cf464d7-mwc4r\" (UID: \"51fa30eb-9214-4402-b911-28d4a2ff4a5e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-mwc4r"
Oct 31 13:01:47.505806 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:47.505771    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-mwc4r"
Oct 31 13:01:47.739649 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:47.739624    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-rzth9"]
Oct 31 13:01:47.741308 ip-10-0-130-190 kubenswrapper[5060]: W1031 13:01:47.741281    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d2738c9_9abf_47cf_a6d8_b83789055353.slice/crio-48bc43a6c7f89071137b84fbfd3b091404c128bd673523ec034f3ac2c270f78f WatchSource:0}: Error finding container 48bc43a6c7f89071137b84fbfd3b091404c128bd673523ec034f3ac2c270f78f: Status 404 returned error can't find the container with id 48bc43a6c7f89071137b84fbfd3b091404c128bd673523ec034f3ac2c270f78f
Oct 31 13:01:47.743477 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:47.743447    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 13:01:47.743561 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:47.743525    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 13:01:47.743561 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:47.743554    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 13:01:47.801222 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:47.801197    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-mwc4r"]
Oct 31 13:01:47.802066 ip-10-0-130-190 kubenswrapper[5060]: W1031 13:01:47.802029    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51fa30eb_9214_4402_b911_28d4a2ff4a5e.slice/crio-27a3db6a7a7acb18a7304adf16853782ed5cbe1d5ebbe0ce046015436353330b WatchSource:0}: Error finding container 27a3db6a7a7acb18a7304adf16853782ed5cbe1d5ebbe0ce046015436353330b: Status 404 returned error can't find the container with id 27a3db6a7a7acb18a7304adf16853782ed5cbe1d5ebbe0ce046015436353330b
Oct 31 13:01:47.804347 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:47.804318    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 13:01:47.804405 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:47.804383    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 13:01:47.804446 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:47.804428    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 13:01:48.108759 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:48.108725    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-rzth9" event={"ID":"1d2738c9-9abf-47cf-a6d8-b83789055353","Type":"ContainerStarted","Data":"01e1593435b063230a9785cc57a5a923bbc4b7d7890eaa3717d1007af0dfd3d4"}
Oct 31 13:01:48.109360 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:48.108788    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-rzth9" event={"ID":"1d2738c9-9abf-47cf-a6d8-b83789055353","Type":"ContainerStarted","Data":"48bc43a6c7f89071137b84fbfd3b091404c128bd673523ec034f3ac2c270f78f"}
Oct 31 13:01:48.111721 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:48.111688    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-mwc4r" event={"ID":"51fa30eb-9214-4402-b911-28d4a2ff4a5e","Type":"ContainerStarted","Data":"c95c870062154a06732139cb16db411b265ed5a8e2a79cbd29c47612af49ed38"}
Oct 31 13:01:48.111869 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:48.111743    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-mwc4r" event={"ID":"51fa30eb-9214-4402-b911-28d4a2ff4a5e","Type":"ContainerStarted","Data":"27a3db6a7a7acb18a7304adf16853782ed5cbe1d5ebbe0ce046015436353330b"}
Oct 31 13:01:48.148759 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:48.148702    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-rzth9" podStartSLOduration=1.148685633 podStartE2EDuration="1.148685633s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:01:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:01:48.132607568 +0000 UTC m=+2071.235675271" watchObservedRunningTime="2026-10-31 13:01:48.148685633 +0000 UTC m=+2071.251753344"
Oct 31 13:01:48.167215 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:48.167148    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-mwc4r" podStartSLOduration=1.167130531 podStartE2EDuration="1.167130531s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:01:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:01:48.162295414 +0000 UTC m=+2071.265363139" watchObservedRunningTime="2026-10-31 13:01:48.167130531 +0000 UTC m=+2071.270198244"
Oct 31 13:01:48.279512 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:48.278999    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-64d9577675-jdt6k"]
Oct 31 13:01:48.292034 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:48.291991    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-64d9577675-jdt6k"
Oct 31 13:01:48.300864 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:48.298571    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-64d9577675-jdt6k"]
Oct 31 13:01:48.300864 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:48.298586    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Oct 31 13:01:48.300864 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:48.298790    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Oct 31 13:01:48.300864 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:48.298832    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-t8k9s\""
Oct 31 13:01:48.353830 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:48.353802    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Oct 31 13:01:48.450721 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:48.450631    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-rzth9"
Oct 31 13:01:48.451956 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:48.451931    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28ntg\" (UniqueName: \"kubernetes.io/projected/66ebd441-f618-4dab-9bdd-d119f4111152-kube-api-access-28ntg\") pod \"maas-controller-64d9577675-jdt6k\" (UID: \"66ebd441-f618-4dab-9bdd-d119f4111152\") " pod="opendatahub/maas-controller-64d9577675-jdt6k"
Oct 31 13:01:48.452137 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:48.451963    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/66ebd441-f618-4dab-9bdd-d119f4111152-webhook-cert\") pod \"maas-controller-64d9577675-jdt6k\" (UID: \"66ebd441-f618-4dab-9bdd-d119f4111152\") " pod="opendatahub/maas-controller-64d9577675-jdt6k"
Oct 31 13:01:48.452137 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:48.451989    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66ebd441-f618-4dab-9bdd-d119f4111152-metrics-certs\") pod \"maas-controller-64d9577675-jdt6k\" (UID: \"66ebd441-f618-4dab-9bdd-d119f4111152\") " pod="opendatahub/maas-controller-64d9577675-jdt6k"
Oct 31 13:01:48.452558 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:48.452535    5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-645cf464d7-rzth9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.13:15021/healthz/ready\": dial tcp 10.132.1.13:15021: connect: connection refused" start-of-body=
Oct 31 13:01:48.452652 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:48.452605    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-rzth9" podUID="1d2738c9-9abf-47cf-a6d8-b83789055353" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.13:15021/healthz/ready\": dial tcp 10.132.1.13:15021: connect: connection refused"
Oct 31 13:01:48.506960 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:48.506928    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-mwc4r"
Oct 31 13:01:48.508962 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:48.508936    5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-645cf464d7-mwc4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.14:15021/healthz/ready\": dial tcp 10.132.1.14:15021: connect: connection refused" start-of-body=
Oct 31 13:01:48.509143 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:48.509124    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-mwc4r" podUID="51fa30eb-9214-4402-b911-28d4a2ff4a5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.14:15021/healthz/ready\": dial tcp 10.132.1.14:15021: connect: connection refused"
Oct 31 13:01:48.552839 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:48.552807    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28ntg\" (UniqueName: \"kubernetes.io/projected/66ebd441-f618-4dab-9bdd-d119f4111152-kube-api-access-28ntg\") pod \"maas-controller-64d9577675-jdt6k\" (UID: \"66ebd441-f618-4dab-9bdd-d119f4111152\") " pod="opendatahub/maas-controller-64d9577675-jdt6k"
Oct 31 13:01:48.552839 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:48.552843    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/66ebd441-f618-4dab-9bdd-d119f4111152-webhook-cert\") pod \"maas-controller-64d9577675-jdt6k\" (UID: \"66ebd441-f618-4dab-9bdd-d119f4111152\") " pod="opendatahub/maas-controller-64d9577675-jdt6k"
Oct 31 13:01:48.553043 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:48.552867    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66ebd441-f618-4dab-9bdd-d119f4111152-metrics-certs\") pod \"maas-controller-64d9577675-jdt6k\" (UID: \"66ebd441-f618-4dab-9bdd-d119f4111152\") " pod="opendatahub/maas-controller-64d9577675-jdt6k"
Oct 31 13:01:48.577511 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:48.577252    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66ebd441-f618-4dab-9bdd-d119f4111152-metrics-certs\") pod \"maas-controller-64d9577675-jdt6k\" (UID: \"66ebd441-f618-4dab-9bdd-d119f4111152\") " pod="opendatahub/maas-controller-64d9577675-jdt6k"
Oct 31 13:01:48.597516 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:48.593438    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28ntg\" (UniqueName: \"kubernetes.io/projected/66ebd441-f618-4dab-9bdd-d119f4111152-kube-api-access-28ntg\") pod \"maas-controller-64d9577675-jdt6k\" (UID: \"66ebd441-f618-4dab-9bdd-d119f4111152\") " pod="opendatahub/maas-controller-64d9577675-jdt6k"
Oct 31 13:01:48.610417 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:48.610375    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/66ebd441-f618-4dab-9bdd-d119f4111152-webhook-cert\") pod \"maas-controller-64d9577675-jdt6k\" (UID: \"66ebd441-f618-4dab-9bdd-d119f4111152\") " pod="opendatahub/maas-controller-64d9577675-jdt6k"
Oct 31 13:01:48.625903 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:48.625865    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-64d9577675-jdt6k"
Oct 31 13:01:49.006820 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:49.006779    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-64d9577675-jdt6k"]
Oct 31 13:01:49.007099 ip-10-0-130-190 kubenswrapper[5060]: W1031 13:01:49.007053    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66ebd441_f618_4dab_9bdd_d119f4111152.slice/crio-1855b50f3c9fc84f30394676db2038d0163244496b070547e1fb33034d2efbec WatchSource:0}: Error finding container 1855b50f3c9fc84f30394676db2038d0163244496b070547e1fb33034d2efbec: Status 404 returned error can't find the container with id 1855b50f3c9fc84f30394676db2038d0163244496b070547e1fb33034d2efbec
Oct 31 13:01:49.048159 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:49.048131    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f51322f-0910-43aa-a5d9-e34ba479e813" path="/var/lib/kubelet/pods/6f51322f-0910-43aa-a5d9-e34ba479e813/volumes"
Oct 31 13:01:49.119449 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:49.119405    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-64d9577675-jdt6k" event={"ID":"66ebd441-f618-4dab-9bdd-d119f4111152","Type":"ContainerStarted","Data":"1855b50f3c9fc84f30394676db2038d0163244496b070547e1fb33034d2efbec"}
Oct 31 13:01:49.450025 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:49.449907    5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-645cf464d7-rzth9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.13:15021/healthz/ready\": dial tcp 10.132.1.13:15021: connect: connection refused" start-of-body=
Oct 31 13:01:49.450025 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:49.449986    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-rzth9" podUID="1d2738c9-9abf-47cf-a6d8-b83789055353" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.13:15021/healthz/ready\": dial tcp 10.132.1.13:15021: connect: connection refused"
Oct 31 13:01:49.508311 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:49.508272    5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-645cf464d7-mwc4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.14:15021/healthz/ready\": dial tcp 10.132.1.14:15021: connect: connection refused" start-of-body=
Oct 31 13:01:49.508615 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:49.508585    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-mwc4r" podUID="51fa30eb-9214-4402-b911-28d4a2ff4a5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.14:15021/healthz/ready\": dial tcp 10.132.1.14:15021: connect: connection refused"
Oct 31 13:01:50.128459 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:50.128425    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-64d9577675-jdt6k" event={"ID":"66ebd441-f618-4dab-9bdd-d119f4111152","Type":"ContainerStarted","Data":"18f8acddc79ba3d5ebbf7fc9fcad55624fdd64f9612f171a46d9287c6915d22e"}
Oct 31 13:01:50.128938 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:50.128879    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-64d9577675-jdt6k"
Oct 31 13:01:50.153345 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:50.153297    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-64d9577675-jdt6k" podStartSLOduration=1.712932532 podStartE2EDuration="2.153284665s" totalImagesPullingTime="440.352133ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:01:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:01:50.15191288 +0000 UTC m=+2073.254980596" watchObservedRunningTime="2026-10-31 13:01:50.153284665 +0000 UTC m=+2073.256352377"
Oct 31 13:01:50.450862 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:50.450787    5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-645cf464d7-rzth9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.13:15021/healthz/ready\": dial tcp 10.132.1.13:15021: connect: connection refused" start-of-body=
Oct 31 13:01:50.451008 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:50.450873    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-rzth9" podUID="1d2738c9-9abf-47cf-a6d8-b83789055353" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.13:15021/healthz/ready\": dial tcp 10.132.1.13:15021: connect: connection refused"
Oct 31 13:01:50.507516 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:50.507189    5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-645cf464d7-mwc4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.14:15021/healthz/ready\": dial tcp 10.132.1.14:15021: connect: connection refused" start-of-body=
Oct 31 13:01:50.507516 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:50.507251    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-mwc4r" podUID="51fa30eb-9214-4402-b911-28d4a2ff4a5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.14:15021/healthz/ready\": dial tcp 10.132.1.14:15021: connect: connection refused"
Oct 31 13:01:51.450556 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:51.450516    5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-645cf464d7-rzth9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.13:15021/healthz/ready\": dial tcp 10.132.1.13:15021: connect: connection refused" start-of-body=
Oct 31 13:01:51.450963 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:51.450581    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-rzth9" podUID="1d2738c9-9abf-47cf-a6d8-b83789055353" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.13:15021/healthz/ready\": dial tcp 10.132.1.13:15021: connect: connection refused"
Oct 31 13:01:51.506850 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:51.506796    5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-645cf464d7-mwc4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.14:15021/healthz/ready\": dial tcp 10.132.1.14:15021: connect: connection refused" start-of-body=
Oct 31 13:01:51.507037 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:51.506859    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-mwc4r" podUID="51fa30eb-9214-4402-b911-28d4a2ff4a5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.14:15021/healthz/ready\": dial tcp 10.132.1.14:15021: connect: connection refused"
Oct 31 13:01:52.450158 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:52.450121    5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-645cf464d7-rzth9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.13:15021/healthz/ready\": dial tcp 10.132.1.13:15021: connect: connection refused" start-of-body=
Oct 31 13:01:52.450344 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:52.450167    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-rzth9" podUID="1d2738c9-9abf-47cf-a6d8-b83789055353" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.13:15021/healthz/ready\": dial tcp 10.132.1.13:15021: connect: connection refused"
Oct 31 13:01:52.507422 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:52.507384    5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-645cf464d7-mwc4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.14:15021/healthz/ready\": dial tcp 10.132.1.14:15021: connect: connection refused" start-of-body=
Oct 31 13:01:52.507857 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:52.507436    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-mwc4r" podUID="51fa30eb-9214-4402-b911-28d4a2ff4a5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.14:15021/healthz/ready\": dial tcp 10.132.1.14:15021: connect: connection refused"
Oct 31 13:01:53.450267 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:53.450231    5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-645cf464d7-rzth9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.13:15021/healthz/ready\": dial tcp 10.132.1.13:15021: connect: connection refused" start-of-body=
Oct 31 13:01:53.450454 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:53.450297    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-rzth9" podUID="1d2738c9-9abf-47cf-a6d8-b83789055353" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.13:15021/healthz/ready\": dial tcp 10.132.1.13:15021: connect: connection refused"
Oct 31 13:01:53.507250 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:53.507211    5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-645cf464d7-mwc4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.14:15021/healthz/ready\": dial tcp 10.132.1.14:15021: connect: connection refused" start-of-body=
Oct 31 13:01:53.507461 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:53.507269    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-mwc4r" podUID="51fa30eb-9214-4402-b911-28d4a2ff4a5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.14:15021/healthz/ready\": dial tcp 10.132.1.14:15021: connect: connection refused"
Oct 31 13:01:54.450444 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:54.450404    5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-645cf464d7-rzth9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.13:15021/healthz/ready\": dial tcp 10.132.1.13:15021: connect: connection refused" start-of-body=
Oct 31 13:01:54.450656 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:54.450462    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-rzth9" podUID="1d2738c9-9abf-47cf-a6d8-b83789055353" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.13:15021/healthz/ready\": dial tcp 10.132.1.13:15021: connect: connection refused"
Oct 31 13:01:54.506893 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:54.506856    5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-645cf464d7-mwc4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.14:15021/healthz/ready\": dial tcp 10.132.1.14:15021: connect: connection refused" start-of-body=
Oct 31 13:01:54.507056 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:54.506917    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-mwc4r" podUID="51fa30eb-9214-4402-b911-28d4a2ff4a5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.14:15021/healthz/ready\": dial tcp 10.132.1.14:15021: connect: connection refused"
Oct 31 13:01:55.025548 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:55.025500    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05c99f42-tvtf4"]
Oct 31 13:01:55.028982 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:55.028955    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05c99f42-tvtf4"]
Oct 31 13:01:55.048479 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:55.048450    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25f69674-db4a-4ce5-a441-d51b61b91d2f" path="/var/lib/kubelet/pods/25f69674-db4a-4ce5-a441-d51b61b91d2f/volumes"
Oct 31 13:01:56.158211 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:56.158173    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-rzth9"
Oct 31 13:01:56.158733 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:56.158632    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-rzth9"
Oct 31 13:01:56.159550 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:56.159529    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-rzth9"
Oct 31 13:01:56.203036 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:56.203001    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-mwc4r"
Oct 31 13:01:56.203344 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:56.203313    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-mwc4r"
Oct 31 13:01:56.204136 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:01:56.204117    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-mwc4r"
Oct 31 13:01:59.920860 ip-10-0-130-190 kubenswrapper[5060]: E1031 13:01:59.920822    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7phb" containerName="extractor"
Oct 31 13:02:02.142356 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:02:02.142324    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-64d9577675-jdt6k"
Oct 31 13:02:17.903840 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:02:17.903806    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pcvvm_f032909d-bc39-4c9e-99b7-f3d800842119/console-operator/2.log"
Oct 31 13:02:17.910007 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:02:17.909982    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pcvvm_f032909d-bc39-4c9e-99b7-f3d800842119/console-operator/2.log"
Oct 31 13:02:26.619595 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:02:26.619557    5060 scope.go:122] "RemoveContainer" containerID="6a3d83b482969b68a66f6be84990624ebc3f7d126b78b096e032372d8922a38f"
Oct 31 13:02:26.630338 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:02:26.630315    5060 scope.go:122] "RemoveContainer" containerID="0c73620ff7fd6c14c6f1cc8f744c0dd5935bd0c8fa36f7e8893ecf82a64731a9"
Oct 31 13:02:26.639416 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:02:26.639396    5060 scope.go:122] "RemoveContainer" containerID="f3312ffbe5d00b5a088382db79f0b7a35cab76989dea919ce7a3555a8878a7a8"
Oct 31 13:02:28.983213 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:02:28.983133    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-dq8tm"]
Oct 31 13:02:28.988851 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:02:28.988826    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-dq8tm"
Oct 31 13:02:28.993878 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:02:28.993854    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 13:02:28.994067 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:02:28.994039    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7wsmb\""
Oct 31 13:02:28.994152 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:02:28.994073    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 13:02:28.994152 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:02:28.994092    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 13:02:28.996697 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:02:28.996672    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-dq8tm"]
Oct 31 13:02:29.090741 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:02:29.090707    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9116105-64ca-4d5b-8620-a5f60a504e5c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-dq8tm\" (UID: \"e9116105-64ca-4d5b-8620-a5f60a504e5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-dq8tm"
Oct 31 13:02:29.090916 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:02:29.090755    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9116105-64ca-4d5b-8620-a5f60a504e5c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-dq8tm\" (UID: \"e9116105-64ca-4d5b-8620-a5f60a504e5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-dq8tm"
Oct 31 13:02:29.090916 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:02:29.090857    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t67kq\" (UniqueName: \"kubernetes.io/projected/e9116105-64ca-4d5b-8620-a5f60a504e5c-kube-api-access-t67kq\") pod \"maas-api-revoke-sub-c1e271a9-dq8tm\" (UID: \"e9116105-64ca-4d5b-8620-a5f60a504e5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-dq8tm"
Oct 31 13:02:29.192129 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:02:29.192094    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9116105-64ca-4d5b-8620-a5f60a504e5c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-dq8tm\" (UID: \"e9116105-64ca-4d5b-8620-a5f60a504e5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-dq8tm"
Oct 31 13:02:29.192304 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:02:29.192141    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9116105-64ca-4d5b-8620-a5f60a504e5c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-dq8tm\" (UID: \"e9116105-64ca-4d5b-8620-a5f60a504e5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-dq8tm"
Oct 31 13:02:29.192304 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:02:29.192176    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t67kq\" (UniqueName: \"kubernetes.io/projected/e9116105-64ca-4d5b-8620-a5f60a504e5c-kube-api-access-t67kq\") pod \"maas-api-revoke-sub-c1e271a9-dq8tm\" (UID: \"e9116105-64ca-4d5b-8620-a5f60a504e5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-dq8tm"
Oct 31 13:02:29.192878 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:02:29.192856    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9116105-64ca-4d5b-8620-a5f60a504e5c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-dq8tm\" (UID: \"e9116105-64ca-4d5b-8620-a5f60a504e5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-dq8tm"
Oct 31 13:02:29.194449 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:02:29.194427    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9116105-64ca-4d5b-8620-a5f60a504e5c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-dq8tm\" (UID: \"e9116105-64ca-4d5b-8620-a5f60a504e5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-dq8tm"
Oct 31 13:02:29.204983 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:02:29.204956    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t67kq\" (UniqueName: \"kubernetes.io/projected/e9116105-64ca-4d5b-8620-a5f60a504e5c-kube-api-access-t67kq\") pod \"maas-api-revoke-sub-c1e271a9-dq8tm\" (UID: \"e9116105-64ca-4d5b-8620-a5f60a504e5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-dq8tm"
Oct 31 13:02:29.304731 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:02:29.304691    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-dq8tm"
Oct 31 13:02:29.606568 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:02:29.599224    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-dq8tm"]
Oct 31 13:02:29.608822 ip-10-0-130-190 kubenswrapper[5060]: W1031 13:02:29.608774    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9116105_64ca_4d5b_8620_a5f60a504e5c.slice/crio-422d10d68ab389ad934ade4a1112d3c7ef827c31587868de610c6e18a27cd8e9 WatchSource:0}: Error finding container 422d10d68ab389ad934ade4a1112d3c7ef827c31587868de610c6e18a27cd8e9: Status 404 returned error can't find the container with id 422d10d68ab389ad934ade4a1112d3c7ef827c31587868de610c6e18a27cd8e9
Oct 31 13:02:29.920277 ip-10-0-130-190 kubenswrapper[5060]: E1031 13:02:29.920238    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7phb" containerName="extractor"
Oct 31 13:02:30.446970 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:02:30.446933    5060 generic.go:356] "Generic (PLEG): container finished" podID="e9116105-64ca-4d5b-8620-a5f60a504e5c" containerID="6d049363621f24156515c9980e6a206c21951a3bf5998b93cbc8686ed24f28f4" exitCode=0
Oct 31 13:02:30.447388 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:02:30.447018    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-dq8tm" event={"ID":"e9116105-64ca-4d5b-8620-a5f60a504e5c","Type":"ContainerDied","Data":"6d049363621f24156515c9980e6a206c21951a3bf5998b93cbc8686ed24f28f4"}
Oct 31 13:02:30.447388 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:02:30.447086    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-dq8tm" event={"ID":"e9116105-64ca-4d5b-8620-a5f60a504e5c","Type":"ContainerStarted","Data":"422d10d68ab389ad934ade4a1112d3c7ef827c31587868de610c6e18a27cd8e9"}
Oct 31 13:02:31.598677 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:02:31.598650    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-dq8tm"
Oct 31 13:02:31.717040 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:02:31.716915    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e9116105-64ca-4d5b-8620-a5f60a504e5c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9116105-64ca-4d5b-8620-a5f60a504e5c-maas-api-xks-ca\") pod \"e9116105-64ca-4d5b-8620-a5f60a504e5c\" (UID: \"e9116105-64ca-4d5b-8620-a5f60a504e5c\") "
Oct 31 13:02:31.717240 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:02:31.717047    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9116105-64ca-4d5b-8620-a5f60a504e5c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9116105-64ca-4d5b-8620-a5f60a504e5c-maas-api-openshift-ca\") pod \"e9116105-64ca-4d5b-8620-a5f60a504e5c\" (UID: \"e9116105-64ca-4d5b-8620-a5f60a504e5c\") "
Oct 31 13:02:31.717240 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:02:31.717108    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9116105-64ca-4d5b-8620-a5f60a504e5c-kube-api-access-t67kq\" (UniqueName: \"kubernetes.io/projected/e9116105-64ca-4d5b-8620-a5f60a504e5c-kube-api-access-t67kq\") pod \"e9116105-64ca-4d5b-8620-a5f60a504e5c\" (UID: \"e9116105-64ca-4d5b-8620-a5f60a504e5c\") "
Oct 31 13:02:31.717412 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:02:31.717388    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9116105-64ca-4d5b-8620-a5f60a504e5c-maas-api-openshift-ca" pod "e9116105-64ca-4d5b-8620-a5f60a504e5c" (UID: "e9116105-64ca-4d5b-8620-a5f60a504e5c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:02:31.719318 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:02:31.719285    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9116105-64ca-4d5b-8620-a5f60a504e5c-maas-api-xks-ca" pod "e9116105-64ca-4d5b-8620-a5f60a504e5c" (UID: "e9116105-64ca-4d5b-8620-a5f60a504e5c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:02:31.721659 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:02:31.721630    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9116105-64ca-4d5b-8620-a5f60a504e5c-kube-api-access-t67kq" pod "e9116105-64ca-4d5b-8620-a5f60a504e5c" (UID: "e9116105-64ca-4d5b-8620-a5f60a504e5c"). InnerVolumeSpecName "kube-api-access-t67kq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:02:31.818604 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:02:31.818570    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9116105-64ca-4d5b-8620-a5f60a504e5c-maas-api-xks-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:02:31.818604 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:02:31.818601    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9116105-64ca-4d5b-8620-a5f60a504e5c-maas-api-openshift-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:02:31.818604 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:02:31.818612    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t67kq\" (UniqueName: \"kubernetes.io/projected/e9116105-64ca-4d5b-8620-a5f60a504e5c-kube-api-access-t67kq\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:02:32.462683 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:02:32.462645    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-dq8tm" event={"ID":"e9116105-64ca-4d5b-8620-a5f60a504e5c","Type":"ContainerDied","Data":"422d10d68ab389ad934ade4a1112d3c7ef827c31587868de610c6e18a27cd8e9"}
Oct 31 13:02:32.462683 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:02:32.462671    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-dq8tm"
Oct 31 13:02:32.463091 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:02:32.462696    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="422d10d68ab389ad934ade4a1112d3c7ef827c31587868de610c6e18a27cd8e9"
Oct 31 13:02:42.201515 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:02:42.200651    5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-645cf464d7-rzth9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.13:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 13:02:42.201515 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:02:42.200758    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-rzth9" podUID="1d2738c9-9abf-47cf-a6d8-b83789055353" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.13:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:02:42.245669 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:02:42.245619    5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-645cf464d7-mwc4r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.14:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 13:02:42.245848 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:02:42.245696    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-mwc4r" podUID="51fa30eb-9214-4402-b911-28d4a2ff4a5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.14:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:02:47.791655 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:02:47.791620    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-csff5"]
Oct 31 13:02:47.792163 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:02:47.792091    5060 cpu_manager.go:414] "containerMap: removing container" podUID="e9116105-64ca-4d5b-8620-a5f60a504e5c" containerName="revoke-keys"
Oct 31 13:02:47.792163 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:02:47.792102    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9116105-64ca-4d5b-8620-a5f60a504e5c" containerName="revoke-keys"
Oct 31 13:02:47.792315 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:02:47.792175    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9116105-64ca-4d5b-8620-a5f60a504e5c" containerName="revoke-keys"
Oct 31 13:02:47.802822 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:02:47.800981    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-csff5"
Oct 31 13:02:47.806679 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:02:47.806648    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 13:02:47.807317 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:02:47.807291    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 13:02:47.807456 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:02:47.807295    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 13:02:47.807583 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:02:47.807483    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7wsmb\""
Oct 31 13:02:47.808896 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:02:47.808872    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-csff5"]
Oct 31 13:02:47.843008 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:02:47.842977    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4m86s\" (UniqueName: \"kubernetes.io/projected/114f1bed-041e-48f6-9bc9-12d816bdc5ad-kube-api-access-4m86s\") pod \"maas-api-revoke-sub-2e1f5b5f-csff5\" (UID: \"114f1bed-041e-48f6-9bc9-12d816bdc5ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-csff5"
Oct 31 13:02:47.843174 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:02:47.843122    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/114f1bed-041e-48f6-9bc9-12d816bdc5ad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-csff5\" (UID: \"114f1bed-041e-48f6-9bc9-12d816bdc5ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-csff5"
Oct 31 13:02:47.843229 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:02:47.843213    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/114f1bed-041e-48f6-9bc9-12d816bdc5ad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-csff5\" (UID: \"114f1bed-041e-48f6-9bc9-12d816bdc5ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-csff5"
Oct 31 13:02:47.943936 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:02:47.943898    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4m86s\" (UniqueName: \"kubernetes.io/projected/114f1bed-041e-48f6-9bc9-12d816bdc5ad-kube-api-access-4m86s\") pod \"maas-api-revoke-sub-2e1f5b5f-csff5\" (UID: \"114f1bed-041e-48f6-9bc9-12d816bdc5ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-csff5"
Oct 31 13:02:47.944123 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:02:47.943982    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/114f1bed-041e-48f6-9bc9-12d816bdc5ad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-csff5\" (UID: \"114f1bed-041e-48f6-9bc9-12d816bdc5ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-csff5"
Oct 31 13:02:47.944123 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:02:47.944010    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/114f1bed-041e-48f6-9bc9-12d816bdc5ad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-csff5\" (UID: \"114f1bed-041e-48f6-9bc9-12d816bdc5ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-csff5"
Oct 31 13:02:47.944723 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:02:47.944694    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/114f1bed-041e-48f6-9bc9-12d816bdc5ad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-csff5\" (UID: \"114f1bed-041e-48f6-9bc9-12d816bdc5ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-csff5"
Oct 31 13:02:47.946435 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:02:47.946409    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/114f1bed-041e-48f6-9bc9-12d816bdc5ad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-csff5\" (UID: \"114f1bed-041e-48f6-9bc9-12d816bdc5ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-csff5"
Oct 31 13:02:47.956024 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:02:47.956000    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4m86s\" (UniqueName: \"kubernetes.io/projected/114f1bed-041e-48f6-9bc9-12d816bdc5ad-kube-api-access-4m86s\") pod \"maas-api-revoke-sub-2e1f5b5f-csff5\" (UID: \"114f1bed-041e-48f6-9bc9-12d816bdc5ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-csff5"
Oct 31 13:02:48.122022 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:02:48.121939    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-csff5"
Oct 31 13:02:48.400776 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:02:48.400748    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-csff5"]
Oct 31 13:02:48.402020 ip-10-0-130-190 kubenswrapper[5060]: W1031 13:02:48.401986    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod114f1bed_041e_48f6_9bc9_12d816bdc5ad.slice/crio-388c672d22a5f9c938a594b72cb6b729248a4ee243463aea6f13213413e4a747 WatchSource:0}: Error finding container 388c672d22a5f9c938a594b72cb6b729248a4ee243463aea6f13213413e4a747: Status 404 returned error can't find the container with id 388c672d22a5f9c938a594b72cb6b729248a4ee243463aea6f13213413e4a747
Oct 31 13:02:48.647432 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:02:48.647391    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-csff5" event={"ID":"114f1bed-041e-48f6-9bc9-12d816bdc5ad","Type":"ContainerStarted","Data":"388c672d22a5f9c938a594b72cb6b729248a4ee243463aea6f13213413e4a747"}
Oct 31 13:02:49.655168 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:02:49.655135    5060 generic.go:356] "Generic (PLEG): container finished" podID="114f1bed-041e-48f6-9bc9-12d816bdc5ad" containerID="bd95b88425e537f69341837457b26afdfd57004325881bd311ec1db16c3bb116" exitCode=0
Oct 31 13:02:49.655590 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:02:49.655224    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-csff5" event={"ID":"114f1bed-041e-48f6-9bc9-12d816bdc5ad","Type":"ContainerDied","Data":"bd95b88425e537f69341837457b26afdfd57004325881bd311ec1db16c3bb116"}
Oct 31 13:02:50.807669 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:02:50.807646    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-csff5"
Oct 31 13:02:50.874233 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:02:50.874198    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/114f1bed-041e-48f6-9bc9-12d816bdc5ad-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/114f1bed-041e-48f6-9bc9-12d816bdc5ad-maas-api-openshift-ca\") pod \"114f1bed-041e-48f6-9bc9-12d816bdc5ad\" (UID: \"114f1bed-041e-48f6-9bc9-12d816bdc5ad\") "
Oct 31 13:02:50.874428 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:02:50.874260    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/114f1bed-041e-48f6-9bc9-12d816bdc5ad-kube-api-access-4m86s\" (UniqueName: \"kubernetes.io/projected/114f1bed-041e-48f6-9bc9-12d816bdc5ad-kube-api-access-4m86s\") pod \"114f1bed-041e-48f6-9bc9-12d816bdc5ad\" (UID: \"114f1bed-041e-48f6-9bc9-12d816bdc5ad\") "
Oct 31 13:02:50.874428 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:02:50.874290    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/114f1bed-041e-48f6-9bc9-12d816bdc5ad-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/114f1bed-041e-48f6-9bc9-12d816bdc5ad-maas-api-xks-ca\") pod \"114f1bed-041e-48f6-9bc9-12d816bdc5ad\" (UID: \"114f1bed-041e-48f6-9bc9-12d816bdc5ad\") "
Oct 31 13:02:50.874603 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:02:50.874577    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/114f1bed-041e-48f6-9bc9-12d816bdc5ad-maas-api-openshift-ca" pod "114f1bed-041e-48f6-9bc9-12d816bdc5ad" (UID: "114f1bed-041e-48f6-9bc9-12d816bdc5ad"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:02:50.876475 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:02:50.876449    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/114f1bed-041e-48f6-9bc9-12d816bdc5ad-kube-api-access-4m86s" pod "114f1bed-041e-48f6-9bc9-12d816bdc5ad" (UID: "114f1bed-041e-48f6-9bc9-12d816bdc5ad"). InnerVolumeSpecName "kube-api-access-4m86s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:02:50.876593 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:02:50.876449    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/114f1bed-041e-48f6-9bc9-12d816bdc5ad-maas-api-xks-ca" pod "114f1bed-041e-48f6-9bc9-12d816bdc5ad" (UID: "114f1bed-041e-48f6-9bc9-12d816bdc5ad"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:02:50.975181 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:02:50.975095    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/114f1bed-041e-48f6-9bc9-12d816bdc5ad-maas-api-openshift-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:02:50.975181 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:02:50.975127    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4m86s\" (UniqueName: \"kubernetes.io/projected/114f1bed-041e-48f6-9bc9-12d816bdc5ad-kube-api-access-4m86s\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:02:50.975181 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:02:50.975138    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/114f1bed-041e-48f6-9bc9-12d816bdc5ad-maas-api-xks-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:02:51.670262 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:02:51.670226    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-csff5"
Oct 31 13:02:51.670417 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:02:51.670225    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-csff5" event={"ID":"114f1bed-041e-48f6-9bc9-12d816bdc5ad","Type":"ContainerDied","Data":"388c672d22a5f9c938a594b72cb6b729248a4ee243463aea6f13213413e4a747"}
Oct 31 13:02:51.670417 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:02:51.670355    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="388c672d22a5f9c938a594b72cb6b729248a4ee243463aea6f13213413e4a747"
Oct 31 13:02:58.728810 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:02:58.728771    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-lk95k"]
Oct 31 13:02:58.730511 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:02:58.730467    5060 cpu_manager.go:414] "containerMap: removing container" podUID="114f1bed-041e-48f6-9bc9-12d816bdc5ad" containerName="revoke-keys"
Oct 31 13:02:58.730677 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:02:58.730663    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="114f1bed-041e-48f6-9bc9-12d816bdc5ad" containerName="revoke-keys"
Oct 31 13:02:58.730986 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:02:58.730971    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="114f1bed-041e-48f6-9bc9-12d816bdc5ad" containerName="revoke-keys"
Oct 31 13:02:58.737559 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:02:58.736601    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-lk95k"
Oct 31 13:02:58.741313 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:02:58.741255    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 13:02:58.741905 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:02:58.741883    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 13:02:58.742012 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:02:58.741999    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7wsmb\""
Oct 31 13:02:58.742088 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:02:58.742040    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 13:02:58.743018 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:02:58.742986    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-lk95k"]
Oct 31 13:02:58.852139 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:02:58.852105    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ba9cdab-bb2b-4aa0-9cef-180597f674a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-lk95k\" (UID: \"6ba9cdab-bb2b-4aa0-9cef-180597f674a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-lk95k"
Oct 31 13:02:58.852314 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:02:58.852163    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ba9cdab-bb2b-4aa0-9cef-180597f674a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-lk95k\" (UID: \"6ba9cdab-bb2b-4aa0-9cef-180597f674a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-lk95k"
Oct 31 13:02:58.852314 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:02:58.852283    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fz7p7\" (UniqueName: \"kubernetes.io/projected/6ba9cdab-bb2b-4aa0-9cef-180597f674a2-kube-api-access-fz7p7\") pod \"maas-api-revoke-sub-ec843018-lk95k\" (UID: \"6ba9cdab-bb2b-4aa0-9cef-180597f674a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-lk95k"
Oct 31 13:02:58.953140 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:02:58.953102    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ba9cdab-bb2b-4aa0-9cef-180597f674a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-lk95k\" (UID: \"6ba9cdab-bb2b-4aa0-9cef-180597f674a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-lk95k"
Oct 31 13:02:58.953325 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:02:58.953165    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ba9cdab-bb2b-4aa0-9cef-180597f674a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-lk95k\" (UID: \"6ba9cdab-bb2b-4aa0-9cef-180597f674a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-lk95k"
Oct 31 13:02:58.953325 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:02:58.953213    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fz7p7\" (UniqueName: \"kubernetes.io/projected/6ba9cdab-bb2b-4aa0-9cef-180597f674a2-kube-api-access-fz7p7\") pod \"maas-api-revoke-sub-ec843018-lk95k\" (UID: \"6ba9cdab-bb2b-4aa0-9cef-180597f674a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-lk95k"
Oct 31 13:02:58.953836 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:02:58.953812    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ba9cdab-bb2b-4aa0-9cef-180597f674a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-lk95k\" (UID: \"6ba9cdab-bb2b-4aa0-9cef-180597f674a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-lk95k"
Oct 31 13:02:58.955693 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:02:58.955669    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ba9cdab-bb2b-4aa0-9cef-180597f674a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-lk95k\" (UID: \"6ba9cdab-bb2b-4aa0-9cef-180597f674a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-lk95k"
Oct 31 13:02:58.966127 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:02:58.966099    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fz7p7\" (UniqueName: \"kubernetes.io/projected/6ba9cdab-bb2b-4aa0-9cef-180597f674a2-kube-api-access-fz7p7\") pod \"maas-api-revoke-sub-ec843018-lk95k\" (UID: \"6ba9cdab-bb2b-4aa0-9cef-180597f674a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-lk95k"
Oct 31 13:02:59.054485 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:02:59.054448    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-lk95k"
Oct 31 13:02:59.376031 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:02:59.376004    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-lk95k"]
Oct 31 13:02:59.378094 ip-10-0-130-190 kubenswrapper[5060]: W1031 13:02:59.378062    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ba9cdab_bb2b_4aa0_9cef_180597f674a2.slice/crio-bad6a363c1c18ef094389b19b572612f7e1d5319cbdb599563d95dc965415691 WatchSource:0}: Error finding container bad6a363c1c18ef094389b19b572612f7e1d5319cbdb599563d95dc965415691: Status 404 returned error can't find the container with id bad6a363c1c18ef094389b19b572612f7e1d5319cbdb599563d95dc965415691
Oct 31 13:02:59.727534 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:02:59.727468    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-lk95k" event={"ID":"6ba9cdab-bb2b-4aa0-9cef-180597f674a2","Type":"ContainerStarted","Data":"147f2c65b917848d42aa5eae80331ec59e3074e1162b8bfb98790a4fca649c4d"}
Oct 31 13:02:59.727694 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:02:59.727556    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-lk95k" event={"ID":"6ba9cdab-bb2b-4aa0-9cef-180597f674a2","Type":"ContainerStarted","Data":"bad6a363c1c18ef094389b19b572612f7e1d5319cbdb599563d95dc965415691"}
Oct 31 13:02:59.750812 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:02:59.750762    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-lk95k" podStartSLOduration=1.750749369 podStartE2EDuration="1.750749369s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:02:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:02:59.74721586 +0000 UTC m=+2142.850283574" watchObservedRunningTime="2026-10-31 13:02:59.750749369 +0000 UTC m=+2142.853817081"
Oct 31 13:02:59.922484 ip-10-0-130-190 kubenswrapper[5060]: E1031 13:02:59.922400    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7phb" containerName="extractor"
Oct 31 13:03:00.738218 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:03:00.738174    5060 generic.go:356] "Generic (PLEG): container finished" podID="6ba9cdab-bb2b-4aa0-9cef-180597f674a2" containerID="147f2c65b917848d42aa5eae80331ec59e3074e1162b8bfb98790a4fca649c4d" exitCode=0
Oct 31 13:03:00.738436 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:03:00.738257    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-lk95k" event={"ID":"6ba9cdab-bb2b-4aa0-9cef-180597f674a2","Type":"ContainerDied","Data":"147f2c65b917848d42aa5eae80331ec59e3074e1162b8bfb98790a4fca649c4d"}
Oct 31 13:03:02.716524 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:03:02.715808    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-lk95k"
Oct 31 13:03:02.786757 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:03:02.773923    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6ba9cdab-bb2b-4aa0-9cef-180597f674a2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ba9cdab-bb2b-4aa0-9cef-180597f674a2-maas-api-xks-ca\") pod \"6ba9cdab-bb2b-4aa0-9cef-180597f674a2\" (UID: \"6ba9cdab-bb2b-4aa0-9cef-180597f674a2\") "
Oct 31 13:03:02.786757 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:03:02.782751    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ba9cdab-bb2b-4aa0-9cef-180597f674a2-maas-api-openshift-ca" pod "6ba9cdab-bb2b-4aa0-9cef-180597f674a2" (UID: "6ba9cdab-bb2b-4aa0-9cef-180597f674a2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:03:02.786757 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:03:02.782836    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ba9cdab-bb2b-4aa0-9cef-180597f674a2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ba9cdab-bb2b-4aa0-9cef-180597f674a2-maas-api-openshift-ca\") pod \"6ba9cdab-bb2b-4aa0-9cef-180597f674a2\" (UID: \"6ba9cdab-bb2b-4aa0-9cef-180597f674a2\") "
Oct 31 13:03:02.786757 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:03:02.782907    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ba9cdab-bb2b-4aa0-9cef-180597f674a2-kube-api-access-fz7p7\" (UniqueName: \"kubernetes.io/projected/6ba9cdab-bb2b-4aa0-9cef-180597f674a2-kube-api-access-fz7p7\") pod \"6ba9cdab-bb2b-4aa0-9cef-180597f674a2\" (UID: \"6ba9cdab-bb2b-4aa0-9cef-180597f674a2\") "
Oct 31 13:03:02.905205 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:03:02.873763    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-lk95k" event={"ID":"6ba9cdab-bb2b-4aa0-9cef-180597f674a2","Type":"ContainerDied","Data":"bad6a363c1c18ef094389b19b572612f7e1d5319cbdb599563d95dc965415691"}
Oct 31 13:03:02.905205 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:03:02.873837    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bad6a363c1c18ef094389b19b572612f7e1d5319cbdb599563d95dc965415691"
Oct 31 13:03:02.905205 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:03:02.873913    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-lk95k"
Oct 31 13:03:02.905205 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:03:02.905071    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ba9cdab-bb2b-4aa0-9cef-180597f674a2-maas-api-openshift-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:03:02.967825 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:03:02.967664    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ba9cdab-bb2b-4aa0-9cef-180597f674a2-kube-api-access-fz7p7" pod "6ba9cdab-bb2b-4aa0-9cef-180597f674a2" (UID: "6ba9cdab-bb2b-4aa0-9cef-180597f674a2"). InnerVolumeSpecName "kube-api-access-fz7p7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:03:02.967825 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:03:02.967790    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ba9cdab-bb2b-4aa0-9cef-180597f674a2-maas-api-xks-ca" pod "6ba9cdab-bb2b-4aa0-9cef-180597f674a2" (UID: "6ba9cdab-bb2b-4aa0-9cef-180597f674a2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:03:03.022363 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:03:03.022291    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ba9cdab-bb2b-4aa0-9cef-180597f674a2-maas-api-xks-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:03:03.022363 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:03:03.022334    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fz7p7\" (UniqueName: \"kubernetes.io/projected/6ba9cdab-bb2b-4aa0-9cef-180597f674a2-kube-api-access-fz7p7\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:03:24.703581 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:03:24.703544    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vzdmm"]
Oct 31 13:03:24.704182 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:03:24.704165    5060 cpu_manager.go:414] "containerMap: removing container" podUID="6ba9cdab-bb2b-4aa0-9cef-180597f674a2" containerName="revoke-keys"
Oct 31 13:03:24.704229 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:03:24.704183    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ba9cdab-bb2b-4aa0-9cef-180597f674a2" containerName="revoke-keys"
Oct 31 13:03:24.704284 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:03:24.704275    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ba9cdab-bb2b-4aa0-9cef-180597f674a2" containerName="revoke-keys"
Oct 31 13:03:24.707593 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:03:24.707561    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vzdmm"
Oct 31 13:03:24.713127 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:03:24.713100    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7wsmb\""
Oct 31 13:03:24.713312 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:03:24.713145    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 13:03:24.713312 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:03:24.713190    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 13:03:24.713462 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:03:24.713260    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 13:03:24.718736 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:03:24.718706    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vzdmm"]
Oct 31 13:03:24.788945 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:03:24.788913    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68253505-aee8-43f7-bd54-aefce9f53e8c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-vzdmm\" (UID: \"68253505-aee8-43f7-bd54-aefce9f53e8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vzdmm"
Oct 31 13:03:24.789114 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:03:24.788954    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68253505-aee8-43f7-bd54-aefce9f53e8c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-vzdmm\" (UID: \"68253505-aee8-43f7-bd54-aefce9f53e8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vzdmm"
Oct 31 13:03:24.789114 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:03:24.789094    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tc57r\" (UniqueName: \"kubernetes.io/projected/68253505-aee8-43f7-bd54-aefce9f53e8c-kube-api-access-tc57r\") pod \"maas-api-revoke-sub-046e1343-vzdmm\" (UID: \"68253505-aee8-43f7-bd54-aefce9f53e8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vzdmm"
Oct 31 13:03:24.889868 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:03:24.889833    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tc57r\" (UniqueName: \"kubernetes.io/projected/68253505-aee8-43f7-bd54-aefce9f53e8c-kube-api-access-tc57r\") pod \"maas-api-revoke-sub-046e1343-vzdmm\" (UID: \"68253505-aee8-43f7-bd54-aefce9f53e8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vzdmm"
Oct 31 13:03:24.890069 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:03:24.889906    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68253505-aee8-43f7-bd54-aefce9f53e8c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-vzdmm\" (UID: \"68253505-aee8-43f7-bd54-aefce9f53e8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vzdmm"
Oct 31 13:03:24.890069 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:03:24.889940    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68253505-aee8-43f7-bd54-aefce9f53e8c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-vzdmm\" (UID: \"68253505-aee8-43f7-bd54-aefce9f53e8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vzdmm"
Oct 31 13:03:24.890599 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:03:24.890579    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68253505-aee8-43f7-bd54-aefce9f53e8c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-vzdmm\" (UID: \"68253505-aee8-43f7-bd54-aefce9f53e8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vzdmm"
Oct 31 13:03:24.892210 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:03:24.892190    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68253505-aee8-43f7-bd54-aefce9f53e8c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-vzdmm\" (UID: \"68253505-aee8-43f7-bd54-aefce9f53e8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vzdmm"
Oct 31 13:03:24.902723 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:03:24.902700    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tc57r\" (UniqueName: \"kubernetes.io/projected/68253505-aee8-43f7-bd54-aefce9f53e8c-kube-api-access-tc57r\") pod \"maas-api-revoke-sub-046e1343-vzdmm\" (UID: \"68253505-aee8-43f7-bd54-aefce9f53e8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vzdmm"
Oct 31 13:03:25.024085 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:03:25.024057    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vzdmm"
Oct 31 13:03:25.304894 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:03:25.304857    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vzdmm"]
Oct 31 13:03:25.305506 ip-10-0-130-190 kubenswrapper[5060]: W1031 13:03:25.305460    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68253505_aee8_43f7_bd54_aefce9f53e8c.slice/crio-2f074b089a183e2ee8bac644626c40e4510a2c8711ae3ad16e69e6c65d3ea022 WatchSource:0}: Error finding container 2f074b089a183e2ee8bac644626c40e4510a2c8711ae3ad16e69e6c65d3ea022: Status 404 returned error can't find the container with id 2f074b089a183e2ee8bac644626c40e4510a2c8711ae3ad16e69e6c65d3ea022
Oct 31 13:03:26.025134 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:03:26.025098    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f64af0e-chm2q"]
Oct 31 13:03:26.030061 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:03:26.030030    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f64af0e-chm2q"]
Oct 31 13:03:26.096832 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:03:26.096800    5060 generic.go:356] "Generic (PLEG): container finished" podID="68253505-aee8-43f7-bd54-aefce9f53e8c" containerID="1ebfd62e4daf98266087abb4a85e97522fd52c0fbdc2e5c458fc3f0912086667" exitCode=0
Oct 31 13:03:26.096985 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:03:26.096879    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vzdmm" event={"ID":"68253505-aee8-43f7-bd54-aefce9f53e8c","Type":"ContainerDied","Data":"1ebfd62e4daf98266087abb4a85e97522fd52c0fbdc2e5c458fc3f0912086667"}
Oct 31 13:03:26.096985 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:03:26.096929    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vzdmm" event={"ID":"68253505-aee8-43f7-bd54-aefce9f53e8c","Type":"ContainerStarted","Data":"2f074b089a183e2ee8bac644626c40e4510a2c8711ae3ad16e69e6c65d3ea022"}
Oct 31 13:03:26.678020 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:03:26.677989    5060 scope.go:122] "RemoveContainer" containerID="303bfa7ccec2421ef1e9d0125106744445fef4d0d89db08937c51343313386c1"
Oct 31 13:03:27.047071 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:03:27.047038    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4cef8546-3c4a-471a-88cb-f31b837b6f70" path="/var/lib/kubelet/pods/4cef8546-3c4a-471a-88cb-f31b837b6f70/volumes"
Oct 31 13:03:27.258451 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:03:27.258427    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vzdmm"
Oct 31 13:03:27.416840 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:03:27.416751    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/68253505-aee8-43f7-bd54-aefce9f53e8c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68253505-aee8-43f7-bd54-aefce9f53e8c-maas-api-xks-ca\") pod \"68253505-aee8-43f7-bd54-aefce9f53e8c\" (UID: \"68253505-aee8-43f7-bd54-aefce9f53e8c\") "
Oct 31 13:03:27.416840 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:03:27.416818    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68253505-aee8-43f7-bd54-aefce9f53e8c-kube-api-access-tc57r\" (UniqueName: \"kubernetes.io/projected/68253505-aee8-43f7-bd54-aefce9f53e8c-kube-api-access-tc57r\") pod \"68253505-aee8-43f7-bd54-aefce9f53e8c\" (UID: \"68253505-aee8-43f7-bd54-aefce9f53e8c\") "
Oct 31 13:03:27.417053 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:03:27.416884    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68253505-aee8-43f7-bd54-aefce9f53e8c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68253505-aee8-43f7-bd54-aefce9f53e8c-maas-api-openshift-ca\") pod \"68253505-aee8-43f7-bd54-aefce9f53e8c\" (UID: \"68253505-aee8-43f7-bd54-aefce9f53e8c\") "
Oct 31 13:03:27.417254 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:03:27.417228    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68253505-aee8-43f7-bd54-aefce9f53e8c-maas-api-openshift-ca" pod "68253505-aee8-43f7-bd54-aefce9f53e8c" (UID: "68253505-aee8-43f7-bd54-aefce9f53e8c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:03:27.418891 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:03:27.418861    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68253505-aee8-43f7-bd54-aefce9f53e8c-maas-api-xks-ca" pod "68253505-aee8-43f7-bd54-aefce9f53e8c" (UID: "68253505-aee8-43f7-bd54-aefce9f53e8c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:03:27.418988 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:03:27.418965    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68253505-aee8-43f7-bd54-aefce9f53e8c-kube-api-access-tc57r" pod "68253505-aee8-43f7-bd54-aefce9f53e8c" (UID: "68253505-aee8-43f7-bd54-aefce9f53e8c"). InnerVolumeSpecName "kube-api-access-tc57r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:03:27.518430 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:03:27.518393    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68253505-aee8-43f7-bd54-aefce9f53e8c-maas-api-xks-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:03:27.518430 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:03:27.518426    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tc57r\" (UniqueName: \"kubernetes.io/projected/68253505-aee8-43f7-bd54-aefce9f53e8c-kube-api-access-tc57r\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:03:27.518430 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:03:27.518436    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68253505-aee8-43f7-bd54-aefce9f53e8c-maas-api-openshift-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:03:28.111980 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:03:28.111947    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vzdmm"
Oct 31 13:03:28.112418 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:03:28.111988    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vzdmm" event={"ID":"68253505-aee8-43f7-bd54-aefce9f53e8c","Type":"ContainerDied","Data":"2f074b089a183e2ee8bac644626c40e4510a2c8711ae3ad16e69e6c65d3ea022"}
Oct 31 13:03:28.112418 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:03:28.112037    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2f074b089a183e2ee8bac644626c40e4510a2c8711ae3ad16e69e6c65d3ea022"
Oct 31 13:03:29.919931 ip-10-0-130-190 kubenswrapper[5060]: E1031 13:03:29.919894    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7phb" containerName="extractor"
Oct 31 13:03:34.918071 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:03:34.918031    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-n5mwc"]
Oct 31 13:03:34.918723 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:03:34.918702    5060 cpu_manager.go:414] "containerMap: removing container" podUID="68253505-aee8-43f7-bd54-aefce9f53e8c" containerName="revoke-keys"
Oct 31 13:03:34.918826 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:03:34.918725    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68253505-aee8-43f7-bd54-aefce9f53e8c" containerName="revoke-keys"
Oct 31 13:03:34.918891 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:03:34.918878    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="68253505-aee8-43f7-bd54-aefce9f53e8c" containerName="revoke-keys"
Oct 31 13:03:34.922579 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:03:34.922550    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-n5mwc"
Oct 31 13:03:34.927472 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:03:34.927442    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 13:03:34.928424 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:03:34.928400    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 13:03:34.928620 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:03:34.928435    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 13:03:34.928755 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:03:34.928466    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7wsmb\""
Oct 31 13:03:34.933375 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:03:34.933348    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-n5mwc"]
Oct 31 13:03:34.982346 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:03:34.982312    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9vww\" (UniqueName: \"kubernetes.io/projected/3ab0af02-0da7-4d8f-9e72-53f16ee23c18-kube-api-access-r9vww\") pod \"maas-api-revoke-sub-d3fba056-n5mwc\" (UID: \"3ab0af02-0da7-4d8f-9e72-53f16ee23c18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-n5mwc"
Oct 31 13:03:34.982539 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:03:34.982362    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ab0af02-0da7-4d8f-9e72-53f16ee23c18-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-n5mwc\" (UID: \"3ab0af02-0da7-4d8f-9e72-53f16ee23c18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-n5mwc"
Oct 31 13:03:34.982539 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:03:34.982455    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ab0af02-0da7-4d8f-9e72-53f16ee23c18-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-n5mwc\" (UID: \"3ab0af02-0da7-4d8f-9e72-53f16ee23c18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-n5mwc"
Oct 31 13:03:35.083029 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:03:35.082987    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ab0af02-0da7-4d8f-9e72-53f16ee23c18-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-n5mwc\" (UID: \"3ab0af02-0da7-4d8f-9e72-53f16ee23c18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-n5mwc"
Oct 31 13:03:35.083223 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:03:35.083101    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9vww\" (UniqueName: \"kubernetes.io/projected/3ab0af02-0da7-4d8f-9e72-53f16ee23c18-kube-api-access-r9vww\") pod \"maas-api-revoke-sub-d3fba056-n5mwc\" (UID: \"3ab0af02-0da7-4d8f-9e72-53f16ee23c18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-n5mwc"
Oct 31 13:03:35.083223 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:03:35.083129    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ab0af02-0da7-4d8f-9e72-53f16ee23c18-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-n5mwc\" (UID: \"3ab0af02-0da7-4d8f-9e72-53f16ee23c18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-n5mwc"
Oct 31 13:03:35.083661 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:03:35.083638    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ab0af02-0da7-4d8f-9e72-53f16ee23c18-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-n5mwc\" (UID: \"3ab0af02-0da7-4d8f-9e72-53f16ee23c18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-n5mwc"
Oct 31 13:03:35.085436 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:03:35.085410    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ab0af02-0da7-4d8f-9e72-53f16ee23c18-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-n5mwc\" (UID: \"3ab0af02-0da7-4d8f-9e72-53f16ee23c18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-n5mwc"
Oct 31 13:03:35.095806 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:03:35.095784    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9vww\" (UniqueName: \"kubernetes.io/projected/3ab0af02-0da7-4d8f-9e72-53f16ee23c18-kube-api-access-r9vww\") pod \"maas-api-revoke-sub-d3fba056-n5mwc\" (UID: \"3ab0af02-0da7-4d8f-9e72-53f16ee23c18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-n5mwc"
Oct 31 13:03:35.242210 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:03:35.242131    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-n5mwc"
Oct 31 13:03:35.525380 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:03:35.525355    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-n5mwc"]
Oct 31 13:03:35.526240 ip-10-0-130-190 kubenswrapper[5060]: W1031 13:03:35.526204    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ab0af02_0da7_4d8f_9e72_53f16ee23c18.slice/crio-210e344d69fda2c4d7ba4dca617d3646070ae747bc246028629cfaac634ff8c5 WatchSource:0}: Error finding container 210e344d69fda2c4d7ba4dca617d3646070ae747bc246028629cfaac634ff8c5: Status 404 returned error can't find the container with id 210e344d69fda2c4d7ba4dca617d3646070ae747bc246028629cfaac634ff8c5
Oct 31 13:03:36.179187 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:03:36.179023    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-n5mwc" event={"ID":"3ab0af02-0da7-4d8f-9e72-53f16ee23c18","Type":"ContainerStarted","Data":"163c2802d750ecc725f38129ddba4e3836b0d86d67d90eb2295b2db14c4e9078"}
Oct 31 13:03:36.179187 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:03:36.179094    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-n5mwc" event={"ID":"3ab0af02-0da7-4d8f-9e72-53f16ee23c18","Type":"ContainerStarted","Data":"210e344d69fda2c4d7ba4dca617d3646070ae747bc246028629cfaac634ff8c5"}
Oct 31 13:03:36.202368 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:03:36.202301    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-n5mwc" podStartSLOduration=2.2022836789999998 podStartE2EDuration="2.202283679s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:03:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:03:36.197387312 +0000 UTC m=+2179.300455025" watchObservedRunningTime="2026-10-31 13:03:36.202283679 +0000 UTC m=+2179.305351388"
Oct 31 13:03:37.201614 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:03:37.200633    5060 generic.go:356] "Generic (PLEG): container finished" podID="3ab0af02-0da7-4d8f-9e72-53f16ee23c18" containerID="163c2802d750ecc725f38129ddba4e3836b0d86d67d90eb2295b2db14c4e9078" exitCode=0
Oct 31 13:03:37.201614 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:03:37.200781    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-n5mwc" event={"ID":"3ab0af02-0da7-4d8f-9e72-53f16ee23c18","Type":"ContainerDied","Data":"163c2802d750ecc725f38129ddba4e3836b0d86d67d90eb2295b2db14c4e9078"}
Oct 31 13:03:38.872541 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:03:38.866673    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-n5mwc"
Oct 31 13:03:39.018388 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:03:39.018314    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ab0af02-0da7-4d8f-9e72-53f16ee23c18-kube-api-access-r9vww\" (UniqueName: \"kubernetes.io/projected/3ab0af02-0da7-4d8f-9e72-53f16ee23c18-kube-api-access-r9vww\") pod \"3ab0af02-0da7-4d8f-9e72-53f16ee23c18\" (UID: \"3ab0af02-0da7-4d8f-9e72-53f16ee23c18\") "
Oct 31 13:03:39.018597 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:03:39.018520    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ab0af02-0da7-4d8f-9e72-53f16ee23c18-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ab0af02-0da7-4d8f-9e72-53f16ee23c18-maas-api-openshift-ca\") pod \"3ab0af02-0da7-4d8f-9e72-53f16ee23c18\" (UID: \"3ab0af02-0da7-4d8f-9e72-53f16ee23c18\") "
Oct 31 13:03:39.018597 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:03:39.018583    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ab0af02-0da7-4d8f-9e72-53f16ee23c18-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ab0af02-0da7-4d8f-9e72-53f16ee23c18-maas-api-xks-ca\") pod \"3ab0af02-0da7-4d8f-9e72-53f16ee23c18\" (UID: \"3ab0af02-0da7-4d8f-9e72-53f16ee23c18\") "
Oct 31 13:03:39.019760 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:03:39.019695    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ab0af02-0da7-4d8f-9e72-53f16ee23c18-maas-api-openshift-ca" pod "3ab0af02-0da7-4d8f-9e72-53f16ee23c18" (UID: "3ab0af02-0da7-4d8f-9e72-53f16ee23c18"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:03:39.023157 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:03:39.022682    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ab0af02-0da7-4d8f-9e72-53f16ee23c18-maas-api-xks-ca" pod "3ab0af02-0da7-4d8f-9e72-53f16ee23c18" (UID: "3ab0af02-0da7-4d8f-9e72-53f16ee23c18"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:03:39.023157 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:03:39.023479    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ab0af02-0da7-4d8f-9e72-53f16ee23c18-kube-api-access-r9vww" pod "3ab0af02-0da7-4d8f-9e72-53f16ee23c18" (UID: "3ab0af02-0da7-4d8f-9e72-53f16ee23c18"). InnerVolumeSpecName "kube-api-access-r9vww". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:03:39.033128 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:03:39.033099    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-acad0e17-qnx5x"]
Oct 31 13:03:39.038566 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:03:39.038526    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-acad0e17-qnx5x"]
Oct 31 13:03:39.051428 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:03:39.051385    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d7613f9-bcf1-4226-9249-21f08991f8e1" path="/var/lib/kubelet/pods/1d7613f9-bcf1-4226-9249-21f08991f8e1/volumes"
Oct 31 13:03:39.119755 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:03:39.119725    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r9vww\" (UniqueName: \"kubernetes.io/projected/3ab0af02-0da7-4d8f-9e72-53f16ee23c18-kube-api-access-r9vww\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:03:39.119755 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:03:39.119753    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ab0af02-0da7-4d8f-9e72-53f16ee23c18-maas-api-openshift-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:03:39.119952 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:03:39.119769    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ab0af02-0da7-4d8f-9e72-53f16ee23c18-maas-api-xks-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:03:39.230700 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:03:39.230672    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-n5mwc"
Oct 31 13:03:39.230909 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:03:39.230708    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-n5mwc" event={"ID":"3ab0af02-0da7-4d8f-9e72-53f16ee23c18","Type":"ContainerDied","Data":"210e344d69fda2c4d7ba4dca617d3646070ae747bc246028629cfaac634ff8c5"}
Oct 31 13:03:39.230909 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:03:39.230763    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="210e344d69fda2c4d7ba4dca617d3646070ae747bc246028629cfaac634ff8c5"
Oct 31 13:03:55.437855 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:03:55.437817    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-lkrg5"]
Oct 31 13:03:55.438384 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:03:55.438367    5060 cpu_manager.go:414] "containerMap: removing container" podUID="3ab0af02-0da7-4d8f-9e72-53f16ee23c18" containerName="revoke-keys"
Oct 31 13:03:55.438440 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:03:55.438385    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ab0af02-0da7-4d8f-9e72-53f16ee23c18" containerName="revoke-keys"
Oct 31 13:03:55.438484 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:03:55.438474    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ab0af02-0da7-4d8f-9e72-53f16ee23c18" containerName="revoke-keys"
Oct 31 13:03:55.441787 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:03:55.441768    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-lkrg5"
Oct 31 13:03:55.446082 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:03:55.446057    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 13:03:55.452273 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:03:55.447426    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 13:03:55.452273 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:03:55.447444    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 13:03:55.452273 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:03:55.452214    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-lkrg5"]
Oct 31 13:03:55.452956 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:03:55.452939    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7wsmb\""
Oct 31 13:03:55.538983 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:03:55.538948    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c283cc5-c508-43bb-a626-f0eedaac259e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-lkrg5\" (UID: \"5c283cc5-c508-43bb-a626-f0eedaac259e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-lkrg5"
Oct 31 13:03:55.539165 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:03:55.539095    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7qw5\" (UniqueName: \"kubernetes.io/projected/5c283cc5-c508-43bb-a626-f0eedaac259e-kube-api-access-n7qw5\") pod \"maas-api-revoke-sub-ae594979-lkrg5\" (UID: \"5c283cc5-c508-43bb-a626-f0eedaac259e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-lkrg5"
Oct 31 13:03:55.539242 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:03:55.539207    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c283cc5-c508-43bb-a626-f0eedaac259e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-lkrg5\" (UID: \"5c283cc5-c508-43bb-a626-f0eedaac259e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-lkrg5"
Oct 31 13:03:55.640092 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:03:55.640044    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7qw5\" (UniqueName: \"kubernetes.io/projected/5c283cc5-c508-43bb-a626-f0eedaac259e-kube-api-access-n7qw5\") pod \"maas-api-revoke-sub-ae594979-lkrg5\" (UID: \"5c283cc5-c508-43bb-a626-f0eedaac259e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-lkrg5"
Oct 31 13:03:55.640412 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:03:55.640115    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c283cc5-c508-43bb-a626-f0eedaac259e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-lkrg5\" (UID: \"5c283cc5-c508-43bb-a626-f0eedaac259e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-lkrg5"
Oct 31 13:03:55.640412 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:03:55.640143    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c283cc5-c508-43bb-a626-f0eedaac259e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-lkrg5\" (UID: \"5c283cc5-c508-43bb-a626-f0eedaac259e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-lkrg5"
Oct 31 13:03:55.640822 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:03:55.640801    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c283cc5-c508-43bb-a626-f0eedaac259e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-lkrg5\" (UID: \"5c283cc5-c508-43bb-a626-f0eedaac259e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-lkrg5"
Oct 31 13:03:55.642591 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:03:55.642567    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c283cc5-c508-43bb-a626-f0eedaac259e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-lkrg5\" (UID: \"5c283cc5-c508-43bb-a626-f0eedaac259e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-lkrg5"
Oct 31 13:03:55.652479 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:03:55.652448    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7qw5\" (UniqueName: \"kubernetes.io/projected/5c283cc5-c508-43bb-a626-f0eedaac259e-kube-api-access-n7qw5\") pod \"maas-api-revoke-sub-ae594979-lkrg5\" (UID: \"5c283cc5-c508-43bb-a626-f0eedaac259e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-lkrg5"
Oct 31 13:03:55.757853 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:03:55.757777    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-lkrg5"
Oct 31 13:03:56.036335 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:03:56.036305    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-lkrg5"]
Oct 31 13:03:56.037379 ip-10-0-130-190 kubenswrapper[5060]: W1031 13:03:56.037354    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c283cc5_c508_43bb_a626_f0eedaac259e.slice/crio-32250f6658830bc79b76d65063ec072c7ab479bfcab8c4ba319da5e060fcdea8 WatchSource:0}: Error finding container 32250f6658830bc79b76d65063ec072c7ab479bfcab8c4ba319da5e060fcdea8: Status 404 returned error can't find the container with id 32250f6658830bc79b76d65063ec072c7ab479bfcab8c4ba319da5e060fcdea8
Oct 31 13:03:56.448560 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:03:56.448516    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-lkrg5" event={"ID":"5c283cc5-c508-43bb-a626-f0eedaac259e","Type":"ContainerStarted","Data":"2ecafe6bc658820b6b0848fc74b124c014b28dff1a301b36f5e534ee5604f48c"}
Oct 31 13:03:56.448944 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:03:56.448581    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-lkrg5" event={"ID":"5c283cc5-c508-43bb-a626-f0eedaac259e","Type":"ContainerStarted","Data":"32250f6658830bc79b76d65063ec072c7ab479bfcab8c4ba319da5e060fcdea8"}
Oct 31 13:03:56.469326 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:03:56.469274    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-lkrg5" podStartSLOduration=1.469261351 podStartE2EDuration="1.469261351s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:03:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:03:56.46639449 +0000 UTC m=+2199.569462203" watchObservedRunningTime="2026-10-31 13:03:56.469261351 +0000 UTC m=+2199.572329063"
Oct 31 13:03:57.457676 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:03:57.457643    5060 generic.go:356] "Generic (PLEG): container finished" podID="5c283cc5-c508-43bb-a626-f0eedaac259e" containerID="2ecafe6bc658820b6b0848fc74b124c014b28dff1a301b36f5e534ee5604f48c" exitCode=0
Oct 31 13:03:57.458097 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:03:57.457726    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-lkrg5" event={"ID":"5c283cc5-c508-43bb-a626-f0eedaac259e","Type":"ContainerDied","Data":"2ecafe6bc658820b6b0848fc74b124c014b28dff1a301b36f5e534ee5604f48c"}
Oct 31 13:03:58.601398 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:03:58.601368    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-lkrg5"
Oct 31 13:03:58.773763 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:03:58.773732    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c283cc5-c508-43bb-a626-f0eedaac259e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c283cc5-c508-43bb-a626-f0eedaac259e-maas-api-openshift-ca\") pod \"5c283cc5-c508-43bb-a626-f0eedaac259e\" (UID: \"5c283cc5-c508-43bb-a626-f0eedaac259e\") "
Oct 31 13:03:58.773925 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:03:58.773806    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c283cc5-c508-43bb-a626-f0eedaac259e-kube-api-access-n7qw5\" (UniqueName: \"kubernetes.io/projected/5c283cc5-c508-43bb-a626-f0eedaac259e-kube-api-access-n7qw5\") pod \"5c283cc5-c508-43bb-a626-f0eedaac259e\" (UID: \"5c283cc5-c508-43bb-a626-f0eedaac259e\") "
Oct 31 13:03:58.773925 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:03:58.773920    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5c283cc5-c508-43bb-a626-f0eedaac259e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c283cc5-c508-43bb-a626-f0eedaac259e-maas-api-xks-ca\") pod \"5c283cc5-c508-43bb-a626-f0eedaac259e\" (UID: \"5c283cc5-c508-43bb-a626-f0eedaac259e\") "
Oct 31 13:03:58.774184 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:03:58.774156    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c283cc5-c508-43bb-a626-f0eedaac259e-maas-api-openshift-ca" pod "5c283cc5-c508-43bb-a626-f0eedaac259e" (UID: "5c283cc5-c508-43bb-a626-f0eedaac259e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:03:58.775941 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:03:58.775912    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c283cc5-c508-43bb-a626-f0eedaac259e-maas-api-xks-ca" pod "5c283cc5-c508-43bb-a626-f0eedaac259e" (UID: "5c283cc5-c508-43bb-a626-f0eedaac259e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:03:58.776071 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:03:58.776056    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c283cc5-c508-43bb-a626-f0eedaac259e-kube-api-access-n7qw5" pod "5c283cc5-c508-43bb-a626-f0eedaac259e" (UID: "5c283cc5-c508-43bb-a626-f0eedaac259e"). InnerVolumeSpecName "kube-api-access-n7qw5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:03:58.875535 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:03:58.875474    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c283cc5-c508-43bb-a626-f0eedaac259e-maas-api-xks-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:03:58.875535 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:03:58.875530    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c283cc5-c508-43bb-a626-f0eedaac259e-maas-api-openshift-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:03:58.875535 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:03:58.875541    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n7qw5\" (UniqueName: \"kubernetes.io/projected/5c283cc5-c508-43bb-a626-f0eedaac259e-kube-api-access-n7qw5\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:03:59.471931 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:03:59.471852    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-lkrg5"
Oct 31 13:03:59.471931 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:03:59.471880    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-lkrg5" event={"ID":"5c283cc5-c508-43bb-a626-f0eedaac259e","Type":"ContainerDied","Data":"32250f6658830bc79b76d65063ec072c7ab479bfcab8c4ba319da5e060fcdea8"}
Oct 31 13:03:59.472139 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:03:59.471953    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="32250f6658830bc79b76d65063ec072c7ab479bfcab8c4ba319da5e060fcdea8"
Oct 31 13:03:59.920889 ip-10-0-130-190 kubenswrapper[5060]: E1031 13:03:59.920847    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7phb" containerName="extractor"
Oct 31 13:04:13.552652 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:04:13.552615    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-g48rp"]
Oct 31 13:04:13.553245 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:04:13.553225    5060 cpu_manager.go:414] "containerMap: removing container" podUID="5c283cc5-c508-43bb-a626-f0eedaac259e" containerName="revoke-keys"
Oct 31 13:04:13.553245 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:04:13.553246    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c283cc5-c508-43bb-a626-f0eedaac259e" containerName="revoke-keys"
Oct 31 13:04:13.553375 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:04:13.553338    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c283cc5-c508-43bb-a626-f0eedaac259e" containerName="revoke-keys"
Oct 31 13:04:13.556690 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:04:13.556673    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-g48rp"
Oct 31 13:04:13.563458 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:04:13.563435    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 13:04:13.563651 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:04:13.563605    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 13:04:13.563651 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:04:13.563604    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 13:04:13.563823 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:04:13.563651    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7wsmb\""
Oct 31 13:04:13.568126 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:04:13.568087    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-g48rp"]
Oct 31 13:04:13.653964 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:04:13.653931    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a33e935-ef9b-4ded-a616-f0996f626f20-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-g48rp\" (UID: \"7a33e935-ef9b-4ded-a616-f0996f626f20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-g48rp"
Oct 31 13:04:13.654144 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:04:13.653970    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xt74b\" (UniqueName: \"kubernetes.io/projected/7a33e935-ef9b-4ded-a616-f0996f626f20-kube-api-access-xt74b\") pod \"maas-api-revoke-sub-a4aef7d7-g48rp\" (UID: \"7a33e935-ef9b-4ded-a616-f0996f626f20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-g48rp"
Oct 31 13:04:13.654144 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:04:13.654048    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a33e935-ef9b-4ded-a616-f0996f626f20-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-g48rp\" (UID: \"7a33e935-ef9b-4ded-a616-f0996f626f20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-g48rp"
Oct 31 13:04:13.754754 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:04:13.754711    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a33e935-ef9b-4ded-a616-f0996f626f20-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-g48rp\" (UID: \"7a33e935-ef9b-4ded-a616-f0996f626f20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-g48rp"
Oct 31 13:04:13.754754 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:04:13.754758    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xt74b\" (UniqueName: \"kubernetes.io/projected/7a33e935-ef9b-4ded-a616-f0996f626f20-kube-api-access-xt74b\") pod \"maas-api-revoke-sub-a4aef7d7-g48rp\" (UID: \"7a33e935-ef9b-4ded-a616-f0996f626f20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-g48rp"
Oct 31 13:04:13.754994 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:04:13.754789    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a33e935-ef9b-4ded-a616-f0996f626f20-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-g48rp\" (UID: \"7a33e935-ef9b-4ded-a616-f0996f626f20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-g48rp"
Oct 31 13:04:13.755389 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:04:13.755369    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a33e935-ef9b-4ded-a616-f0996f626f20-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-g48rp\" (UID: \"7a33e935-ef9b-4ded-a616-f0996f626f20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-g48rp"
Oct 31 13:04:13.757206 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:04:13.757182    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a33e935-ef9b-4ded-a616-f0996f626f20-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-g48rp\" (UID: \"7a33e935-ef9b-4ded-a616-f0996f626f20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-g48rp"
Oct 31 13:04:13.767460 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:04:13.767427    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xt74b\" (UniqueName: \"kubernetes.io/projected/7a33e935-ef9b-4ded-a616-f0996f626f20-kube-api-access-xt74b\") pod \"maas-api-revoke-sub-a4aef7d7-g48rp\" (UID: \"7a33e935-ef9b-4ded-a616-f0996f626f20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-g48rp"
Oct 31 13:04:13.872981 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:04:13.872896    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-g48rp"
Oct 31 13:04:14.151284 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:04:14.151243    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-g48rp"]
Oct 31 13:04:14.152596 ip-10-0-130-190 kubenswrapper[5060]: W1031 13:04:14.152566    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a33e935_ef9b_4ded_a616_f0996f626f20.slice/crio-0b34097f99acd8324a288f3518c8f184bf33cf5b25674eb41d9d03310f69a521 WatchSource:0}: Error finding container 0b34097f99acd8324a288f3518c8f184bf33cf5b25674eb41d9d03310f69a521: Status 404 returned error can't find the container with id 0b34097f99acd8324a288f3518c8f184bf33cf5b25674eb41d9d03310f69a521
Oct 31 13:04:14.635328 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:04:14.635292    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-g48rp" event={"ID":"7a33e935-ef9b-4ded-a616-f0996f626f20","Type":"ContainerStarted","Data":"be5ce9257ee010965feefd2475b7e08e127f6bf232441960051f640d9cc232dd"}
Oct 31 13:04:14.635777 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:04:14.635348    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-g48rp" event={"ID":"7a33e935-ef9b-4ded-a616-f0996f626f20","Type":"ContainerStarted","Data":"0b34097f99acd8324a288f3518c8f184bf33cf5b25674eb41d9d03310f69a521"}
Oct 31 13:04:14.653767 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:04:14.653713    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-g48rp" podStartSLOduration=1.653701619 podStartE2EDuration="1.653701619s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:04:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:04:14.653096842 +0000 UTC m=+2217.756164553" watchObservedRunningTime="2026-10-31 13:04:14.653701619 +0000 UTC m=+2217.756769330"
Oct 31 13:04:15.643535 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:04:15.643479    5060 generic.go:356] "Generic (PLEG): container finished" podID="7a33e935-ef9b-4ded-a616-f0996f626f20" containerID="be5ce9257ee010965feefd2475b7e08e127f6bf232441960051f640d9cc232dd" exitCode=0
Oct 31 13:04:15.643938 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:04:15.643525    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-g48rp" event={"ID":"7a33e935-ef9b-4ded-a616-f0996f626f20","Type":"ContainerDied","Data":"be5ce9257ee010965feefd2475b7e08e127f6bf232441960051f640d9cc232dd"}
Oct 31 13:04:16.803678 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:04:16.803648    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-g48rp"
Oct 31 13:04:16.883472 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:04:16.883438    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a33e935-ef9b-4ded-a616-f0996f626f20-kube-api-access-xt74b\" (UniqueName: \"kubernetes.io/projected/7a33e935-ef9b-4ded-a616-f0996f626f20-kube-api-access-xt74b\") pod \"7a33e935-ef9b-4ded-a616-f0996f626f20\" (UID: \"7a33e935-ef9b-4ded-a616-f0996f626f20\") "
Oct 31 13:04:16.883652 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:04:16.883547    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a33e935-ef9b-4ded-a616-f0996f626f20-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a33e935-ef9b-4ded-a616-f0996f626f20-maas-api-openshift-ca\") pod \"7a33e935-ef9b-4ded-a616-f0996f626f20\" (UID: \"7a33e935-ef9b-4ded-a616-f0996f626f20\") "
Oct 31 13:04:16.883652 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:04:16.883596    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a33e935-ef9b-4ded-a616-f0996f626f20-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a33e935-ef9b-4ded-a616-f0996f626f20-maas-api-xks-ca\") pod \"7a33e935-ef9b-4ded-a616-f0996f626f20\" (UID: \"7a33e935-ef9b-4ded-a616-f0996f626f20\") "
Oct 31 13:04:16.883905 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:04:16.883872    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a33e935-ef9b-4ded-a616-f0996f626f20-maas-api-openshift-ca" pod "7a33e935-ef9b-4ded-a616-f0996f626f20" (UID: "7a33e935-ef9b-4ded-a616-f0996f626f20"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:04:16.885580 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:04:16.885555    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a33e935-ef9b-4ded-a616-f0996f626f20-kube-api-access-xt74b" pod "7a33e935-ef9b-4ded-a616-f0996f626f20" (UID: "7a33e935-ef9b-4ded-a616-f0996f626f20"). InnerVolumeSpecName "kube-api-access-xt74b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:04:16.885580 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:04:16.885565    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a33e935-ef9b-4ded-a616-f0996f626f20-maas-api-xks-ca" pod "7a33e935-ef9b-4ded-a616-f0996f626f20" (UID: "7a33e935-ef9b-4ded-a616-f0996f626f20"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:04:16.984421 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:04:16.984349    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a33e935-ef9b-4ded-a616-f0996f626f20-maas-api-openshift-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:04:16.984421 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:04:16.984380    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a33e935-ef9b-4ded-a616-f0996f626f20-maas-api-xks-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:04:16.984421 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:04:16.984391    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xt74b\" (UniqueName: \"kubernetes.io/projected/7a33e935-ef9b-4ded-a616-f0996f626f20-kube-api-access-xt74b\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:04:17.658399 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:04:17.658357    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-g48rp" event={"ID":"7a33e935-ef9b-4ded-a616-f0996f626f20","Type":"ContainerDied","Data":"0b34097f99acd8324a288f3518c8f184bf33cf5b25674eb41d9d03310f69a521"}
Oct 31 13:04:17.658399 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:04:17.658379    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-g48rp"
Oct 31 13:04:17.658646 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:04:17.658429    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0b34097f99acd8324a288f3518c8f184bf33cf5b25674eb41d9d03310f69a521"
Oct 31 13:04:26.704021 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:04:26.703987    5060 scope.go:122] "RemoveContainer" containerID="a110a99c491380e7629140ccd2fca4d8106c276a5fd4bd342af3443e55b0c9e3"
Oct 31 13:04:29.947301 ip-10-0-130-190 kubenswrapper[5060]: E1031 13:04:29.947259    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7phb" containerName="extractor"
Oct 31 13:04:32.343223 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:04:32.343188    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-g8np6"]
Oct 31 13:04:32.343685 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:04:32.343668    5060 cpu_manager.go:414] "containerMap: removing container" podUID="7a33e935-ef9b-4ded-a616-f0996f626f20" containerName="revoke-keys"
Oct 31 13:04:32.343736 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:04:32.343686    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a33e935-ef9b-4ded-a616-f0996f626f20" containerName="revoke-keys"
Oct 31 13:04:32.343805 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:04:32.343796    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a33e935-ef9b-4ded-a616-f0996f626f20" containerName="revoke-keys"
Oct 31 13:04:32.348564 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:04:32.348542    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-g8np6"
Oct 31 13:04:32.359423 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:04:32.359395    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-g8np6"]
Oct 31 13:04:32.360838 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:04:32.360803    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 13:04:32.360838 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:04:32.360809    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7wsmb\""
Oct 31 13:04:32.394693 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:04:32.394651    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 13:04:32.394693 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:04:32.394651    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 13:04:32.523212 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:04:32.523176    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27d9938b-04e6-4792-aaca-78da1f022533-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-g8np6\" (UID: \"27d9938b-04e6-4792-aaca-78da1f022533\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-g8np6"
Oct 31 13:04:32.523394 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:04:32.523315    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6chp\" (UniqueName: \"kubernetes.io/projected/27d9938b-04e6-4792-aaca-78da1f022533-kube-api-access-d6chp\") pod \"maas-api-revoke-sub-79805ca9-g8np6\" (UID: \"27d9938b-04e6-4792-aaca-78da1f022533\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-g8np6"
Oct 31 13:04:32.523394 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:04:32.523368    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27d9938b-04e6-4792-aaca-78da1f022533-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-g8np6\" (UID: \"27d9938b-04e6-4792-aaca-78da1f022533\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-g8np6"
Oct 31 13:04:32.624525 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:04:32.624388    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6chp\" (UniqueName: \"kubernetes.io/projected/27d9938b-04e6-4792-aaca-78da1f022533-kube-api-access-d6chp\") pod \"maas-api-revoke-sub-79805ca9-g8np6\" (UID: \"27d9938b-04e6-4792-aaca-78da1f022533\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-g8np6"
Oct 31 13:04:32.624525 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:04:32.624469    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27d9938b-04e6-4792-aaca-78da1f022533-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-g8np6\" (UID: \"27d9938b-04e6-4792-aaca-78da1f022533\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-g8np6"
Oct 31 13:04:32.624753 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:04:32.624560    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27d9938b-04e6-4792-aaca-78da1f022533-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-g8np6\" (UID: \"27d9938b-04e6-4792-aaca-78da1f022533\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-g8np6"
Oct 31 13:04:32.625234 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:04:32.625210    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27d9938b-04e6-4792-aaca-78da1f022533-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-g8np6\" (UID: \"27d9938b-04e6-4792-aaca-78da1f022533\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-g8np6"
Oct 31 13:04:32.626916 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:04:32.626886    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27d9938b-04e6-4792-aaca-78da1f022533-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-g8np6\" (UID: \"27d9938b-04e6-4792-aaca-78da1f022533\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-g8np6"
Oct 31 13:04:32.637035 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:04:32.637005    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6chp\" (UniqueName: \"kubernetes.io/projected/27d9938b-04e6-4792-aaca-78da1f022533-kube-api-access-d6chp\") pod \"maas-api-revoke-sub-79805ca9-g8np6\" (UID: \"27d9938b-04e6-4792-aaca-78da1f022533\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-g8np6"
Oct 31 13:04:32.668891 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:04:32.668867    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-g8np6"
Oct 31 13:04:32.952014 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:04:32.951982    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-g8np6"]
Oct 31 13:04:32.953362 ip-10-0-130-190 kubenswrapper[5060]: W1031 13:04:32.953333    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27d9938b_04e6_4792_aaca_78da1f022533.slice/crio-0f3097aac36890e5019cb34453760ee26bbb6ab20da781283abf19b1ee0e44e9 WatchSource:0}: Error finding container 0f3097aac36890e5019cb34453760ee26bbb6ab20da781283abf19b1ee0e44e9: Status 404 returned error can't find the container with id 0f3097aac36890e5019cb34453760ee26bbb6ab20da781283abf19b1ee0e44e9
Oct 31 13:04:33.868726 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:04:33.868697    5060 generic.go:356] "Generic (PLEG): container finished" podID="27d9938b-04e6-4792-aaca-78da1f022533" containerID="91ad1262bda92289311478b2bb7df1101228d4c08df17ca287961480df79bed5" exitCode=0
Oct 31 13:04:33.869052 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:04:33.868778    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-g8np6" event={"ID":"27d9938b-04e6-4792-aaca-78da1f022533","Type":"ContainerDied","Data":"91ad1262bda92289311478b2bb7df1101228d4c08df17ca287961480df79bed5"}
Oct 31 13:04:33.869052 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:04:33.868828    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-g8np6" event={"ID":"27d9938b-04e6-4792-aaca-78da1f022533","Type":"ContainerStarted","Data":"0f3097aac36890e5019cb34453760ee26bbb6ab20da781283abf19b1ee0e44e9"}
Oct 31 13:04:35.016194 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:04:35.016167    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-g8np6"
Oct 31 13:04:35.058093 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:04:35.058051    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27d9938b-04e6-4792-aaca-78da1f022533-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27d9938b-04e6-4792-aaca-78da1f022533-maas-api-openshift-ca\") pod \"27d9938b-04e6-4792-aaca-78da1f022533\" (UID: \"27d9938b-04e6-4792-aaca-78da1f022533\") "
Oct 31 13:04:35.058266 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:04:35.058114    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/27d9938b-04e6-4792-aaca-78da1f022533-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27d9938b-04e6-4792-aaca-78da1f022533-maas-api-xks-ca\") pod \"27d9938b-04e6-4792-aaca-78da1f022533\" (UID: \"27d9938b-04e6-4792-aaca-78da1f022533\") "
Oct 31 13:04:35.058266 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:04:35.058203    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27d9938b-04e6-4792-aaca-78da1f022533-kube-api-access-d6chp\" (UniqueName: \"kubernetes.io/projected/27d9938b-04e6-4792-aaca-78da1f022533-kube-api-access-d6chp\") pod \"27d9938b-04e6-4792-aaca-78da1f022533\" (UID: \"27d9938b-04e6-4792-aaca-78da1f022533\") "
Oct 31 13:04:35.059387 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:04:35.059348    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27d9938b-04e6-4792-aaca-78da1f022533-maas-api-openshift-ca" pod "27d9938b-04e6-4792-aaca-78da1f022533" (UID: "27d9938b-04e6-4792-aaca-78da1f022533"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:04:35.062401 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:04:35.062367    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27d9938b-04e6-4792-aaca-78da1f022533-kube-api-access-d6chp" pod "27d9938b-04e6-4792-aaca-78da1f022533" (UID: "27d9938b-04e6-4792-aaca-78da1f022533"). InnerVolumeSpecName "kube-api-access-d6chp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:04:35.063289 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:04:35.063266    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27d9938b-04e6-4792-aaca-78da1f022533-maas-api-xks-ca" pod "27d9938b-04e6-4792-aaca-78da1f022533" (UID: "27d9938b-04e6-4792-aaca-78da1f022533"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:04:35.159610 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:04:35.159528    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6chp\" (UniqueName: \"kubernetes.io/projected/27d9938b-04e6-4792-aaca-78da1f022533-kube-api-access-d6chp\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:04:35.159610 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:04:35.159560    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27d9938b-04e6-4792-aaca-78da1f022533-maas-api-openshift-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:04:35.159610 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:04:35.159576    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27d9938b-04e6-4792-aaca-78da1f022533-maas-api-xks-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:04:35.883537 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:04:35.883477    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-g8np6"
Oct 31 13:04:35.883728 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:04:35.883529    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-g8np6" event={"ID":"27d9938b-04e6-4792-aaca-78da1f022533","Type":"ContainerDied","Data":"0f3097aac36890e5019cb34453760ee26bbb6ab20da781283abf19b1ee0e44e9"}
Oct 31 13:04:35.883728 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:04:35.883582    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0f3097aac36890e5019cb34453760ee26bbb6ab20da781283abf19b1ee0e44e9"
Oct 31 13:04:48.049515 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:04:48.048565    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b95e1a3-82c69"]
Oct 31 13:04:48.053513 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:04:48.051995    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b95e1a3-82c69"]
Oct 31 13:04:49.049019 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:04:49.048983    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61ba0787-c638-4376-86d9-4c1c90dd8b6d" path="/var/lib/kubelet/pods/61ba0787-c638-4376-86d9-4c1c90dd8b6d/volumes"
Oct 31 13:04:50.100040 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:04:50.100005    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-f986r"]
Oct 31 13:04:50.100483 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:04:50.100469    5060 cpu_manager.go:414] "containerMap: removing container" podUID="27d9938b-04e6-4792-aaca-78da1f022533" containerName="revoke-keys"
Oct 31 13:04:50.100560 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:04:50.100483    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27d9938b-04e6-4792-aaca-78da1f022533" containerName="revoke-keys"
Oct 31 13:04:50.100598 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:04:50.100579    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="27d9938b-04e6-4792-aaca-78da1f022533" containerName="revoke-keys"
Oct 31 13:04:50.103970 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:04:50.103952    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-f986r"
Oct 31 13:04:50.109701 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:04:50.109671    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 13:04:50.110714 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:04:50.110684    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 13:04:50.110846 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:04:50.110815    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 13:04:50.111038 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:04:50.111006    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7wsmb\""
Oct 31 13:04:50.115113 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:04:50.115090    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-f986r"]
Oct 31 13:04:50.195518 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:04:50.195463    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b93b36c-83bf-48c4-9492-ca4384f1827d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-f986r\" (UID: \"4b93b36c-83bf-48c4-9492-ca4384f1827d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-f986r"
Oct 31 13:04:50.195702 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:04:50.195587    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b93b36c-83bf-48c4-9492-ca4384f1827d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-f986r\" (UID: \"4b93b36c-83bf-48c4-9492-ca4384f1827d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-f986r"
Oct 31 13:04:50.195781 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:04:50.195713    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jxpn\" (UniqueName: \"kubernetes.io/projected/4b93b36c-83bf-48c4-9492-ca4384f1827d-kube-api-access-4jxpn\") pod \"maas-api-revoke-sub-0b7e8edb-f986r\" (UID: \"4b93b36c-83bf-48c4-9492-ca4384f1827d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-f986r"
Oct 31 13:04:50.297006 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:04:50.296969    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jxpn\" (UniqueName: \"kubernetes.io/projected/4b93b36c-83bf-48c4-9492-ca4384f1827d-kube-api-access-4jxpn\") pod \"maas-api-revoke-sub-0b7e8edb-f986r\" (UID: \"4b93b36c-83bf-48c4-9492-ca4384f1827d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-f986r"
Oct 31 13:04:50.297211 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:04:50.297047    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b93b36c-83bf-48c4-9492-ca4384f1827d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-f986r\" (UID: \"4b93b36c-83bf-48c4-9492-ca4384f1827d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-f986r"
Oct 31 13:04:50.297211 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:04:50.297107    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b93b36c-83bf-48c4-9492-ca4384f1827d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-f986r\" (UID: \"4b93b36c-83bf-48c4-9492-ca4384f1827d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-f986r"
Oct 31 13:04:50.297872 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:04:50.297843    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b93b36c-83bf-48c4-9492-ca4384f1827d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-f986r\" (UID: \"4b93b36c-83bf-48c4-9492-ca4384f1827d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-f986r"
Oct 31 13:04:50.299730 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:04:50.299705    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b93b36c-83bf-48c4-9492-ca4384f1827d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-f986r\" (UID: \"4b93b36c-83bf-48c4-9492-ca4384f1827d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-f986r"
Oct 31 13:04:50.309160 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:04:50.309139    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jxpn\" (UniqueName: \"kubernetes.io/projected/4b93b36c-83bf-48c4-9492-ca4384f1827d-kube-api-access-4jxpn\") pod \"maas-api-revoke-sub-0b7e8edb-f986r\" (UID: \"4b93b36c-83bf-48c4-9492-ca4384f1827d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-f986r"
Oct 31 13:04:50.422870 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:04:50.422792    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-f986r"
Oct 31 13:04:50.702988 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:04:50.702961    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-f986r"]
Oct 31 13:04:50.704115 ip-10-0-130-190 kubenswrapper[5060]: W1031 13:04:50.704085    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b93b36c_83bf_48c4_9492_ca4384f1827d.slice/crio-4cb3dceb43600aa05cdaad6a2e5c2afd6a8926467f6091d1b22627222b8147b0 WatchSource:0}: Error finding container 4cb3dceb43600aa05cdaad6a2e5c2afd6a8926467f6091d1b22627222b8147b0: Status 404 returned error can't find the container with id 4cb3dceb43600aa05cdaad6a2e5c2afd6a8926467f6091d1b22627222b8147b0
Oct 31 13:04:51.030431 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:04:51.030394    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-f986r" event={"ID":"4b93b36c-83bf-48c4-9492-ca4384f1827d","Type":"ContainerStarted","Data":"55eddffb550edecbbce333e70d29c3bffc8286b7d71dff70926cb026b360d695"}
Oct 31 13:04:51.030626 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:04:51.030446    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-f986r" event={"ID":"4b93b36c-83bf-48c4-9492-ca4384f1827d","Type":"ContainerStarted","Data":"4cb3dceb43600aa05cdaad6a2e5c2afd6a8926467f6091d1b22627222b8147b0"}
Oct 31 13:04:51.050741 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:04:51.050689    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-f986r" podStartSLOduration=1.050675035 podStartE2EDuration="1.050675035s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:04:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:04:51.048519803 +0000 UTC m=+2254.151587517" watchObservedRunningTime="2026-10-31 13:04:51.050675035 +0000 UTC m=+2254.153742746"
Oct 31 13:04:52.039009 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:04:52.038974    5060 generic.go:356] "Generic (PLEG): container finished" podID="4b93b36c-83bf-48c4-9492-ca4384f1827d" containerID="55eddffb550edecbbce333e70d29c3bffc8286b7d71dff70926cb026b360d695" exitCode=0
Oct 31 13:04:52.039379 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:04:52.039026    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-f986r" event={"ID":"4b93b36c-83bf-48c4-9492-ca4384f1827d","Type":"ContainerDied","Data":"55eddffb550edecbbce333e70d29c3bffc8286b7d71dff70926cb026b360d695"}
Oct 31 13:04:53.196086 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:04:53.196061    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-f986r"
Oct 31 13:04:53.224597 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:04:53.224568    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b93b36c-83bf-48c4-9492-ca4384f1827d-kube-api-access-4jxpn\" (UniqueName: \"kubernetes.io/projected/4b93b36c-83bf-48c4-9492-ca4384f1827d-kube-api-access-4jxpn\") pod \"4b93b36c-83bf-48c4-9492-ca4384f1827d\" (UID: \"4b93b36c-83bf-48c4-9492-ca4384f1827d\") "
Oct 31 13:04:53.224767 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:04:53.224694    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4b93b36c-83bf-48c4-9492-ca4384f1827d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b93b36c-83bf-48c4-9492-ca4384f1827d-maas-api-xks-ca\") pod \"4b93b36c-83bf-48c4-9492-ca4384f1827d\" (UID: \"4b93b36c-83bf-48c4-9492-ca4384f1827d\") "
Oct 31 13:04:53.224767 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:04:53.224734    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b93b36c-83bf-48c4-9492-ca4384f1827d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b93b36c-83bf-48c4-9492-ca4384f1827d-maas-api-openshift-ca\") pod \"4b93b36c-83bf-48c4-9492-ca4384f1827d\" (UID: \"4b93b36c-83bf-48c4-9492-ca4384f1827d\") "
Oct 31 13:04:53.225174 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:04:53.225138    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b93b36c-83bf-48c4-9492-ca4384f1827d-maas-api-openshift-ca" pod "4b93b36c-83bf-48c4-9492-ca4384f1827d" (UID: "4b93b36c-83bf-48c4-9492-ca4384f1827d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:04:53.226812 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:04:53.226787    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b93b36c-83bf-48c4-9492-ca4384f1827d-maas-api-xks-ca" pod "4b93b36c-83bf-48c4-9492-ca4384f1827d" (UID: "4b93b36c-83bf-48c4-9492-ca4384f1827d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:04:53.227121 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:04:53.227103    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b93b36c-83bf-48c4-9492-ca4384f1827d-kube-api-access-4jxpn" pod "4b93b36c-83bf-48c4-9492-ca4384f1827d" (UID: "4b93b36c-83bf-48c4-9492-ca4384f1827d"). InnerVolumeSpecName "kube-api-access-4jxpn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:04:53.326525 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:04:53.326421    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b93b36c-83bf-48c4-9492-ca4384f1827d-maas-api-xks-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:04:53.326525 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:04:53.326453    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b93b36c-83bf-48c4-9492-ca4384f1827d-maas-api-openshift-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:04:53.326525 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:04:53.326465    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jxpn\" (UniqueName: \"kubernetes.io/projected/4b93b36c-83bf-48c4-9492-ca4384f1827d-kube-api-access-4jxpn\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:04:54.061667 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:04:54.061633    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-f986r"
Oct 31 13:04:54.061823 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:04:54.061673    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-f986r" event={"ID":"4b93b36c-83bf-48c4-9492-ca4384f1827d","Type":"ContainerDied","Data":"4cb3dceb43600aa05cdaad6a2e5c2afd6a8926467f6091d1b22627222b8147b0"}
Oct 31 13:04:54.061823 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:04:54.061720    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4cb3dceb43600aa05cdaad6a2e5c2afd6a8926467f6091d1b22627222b8147b0"
Oct 31 13:04:59.922628 ip-10-0-130-190 kubenswrapper[5060]: E1031 13:04:59.922589    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7phb" containerName="extractor"
Oct 31 13:05:00.765815 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:05:00.765776    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-k67vn"]
Oct 31 13:05:00.766474 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:05:00.766459    5060 cpu_manager.go:414] "containerMap: removing container" podUID="4b93b36c-83bf-48c4-9492-ca4384f1827d" containerName="revoke-keys"
Oct 31 13:05:00.766541 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:05:00.766478    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b93b36c-83bf-48c4-9492-ca4384f1827d" containerName="revoke-keys"
Oct 31 13:05:00.766675 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:05:00.766663    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b93b36c-83bf-48c4-9492-ca4384f1827d" containerName="revoke-keys"
Oct 31 13:05:00.770672 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:05:00.770647    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-k67vn"
Oct 31 13:05:00.784675 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:05:00.784643    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-k67vn"]
Oct 31 13:05:00.785111 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:05:00.785084    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 13:05:00.785237 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:05:00.785134    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7wsmb\""
Oct 31 13:05:00.785237 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:05:00.785157    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 13:05:00.785237 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:05:00.785211    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 13:05:00.800079 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:05:00.800041    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/070f030c-c00e-46b6-8a08-f4d88b40ce4a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-k67vn\" (UID: \"070f030c-c00e-46b6-8a08-f4d88b40ce4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-k67vn"
Oct 31 13:05:00.800311 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:05:00.800290    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8q4sn\" (UniqueName: \"kubernetes.io/projected/070f030c-c00e-46b6-8a08-f4d88b40ce4a-kube-api-access-8q4sn\") pod \"maas-api-revoke-sub-3d4156f1-k67vn\" (UID: \"070f030c-c00e-46b6-8a08-f4d88b40ce4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-k67vn"
Oct 31 13:05:00.800510 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:05:00.800475    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/070f030c-c00e-46b6-8a08-f4d88b40ce4a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-k67vn\" (UID: \"070f030c-c00e-46b6-8a08-f4d88b40ce4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-k67vn"
Oct 31 13:05:00.902120 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:05:00.902078    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8q4sn\" (UniqueName: \"kubernetes.io/projected/070f030c-c00e-46b6-8a08-f4d88b40ce4a-kube-api-access-8q4sn\") pod \"maas-api-revoke-sub-3d4156f1-k67vn\" (UID: \"070f030c-c00e-46b6-8a08-f4d88b40ce4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-k67vn"
Oct 31 13:05:00.902330 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:05:00.902159    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/070f030c-c00e-46b6-8a08-f4d88b40ce4a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-k67vn\" (UID: \"070f030c-c00e-46b6-8a08-f4d88b40ce4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-k67vn"
Oct 31 13:05:00.902330 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:05:00.902235    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/070f030c-c00e-46b6-8a08-f4d88b40ce4a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-k67vn\" (UID: \"070f030c-c00e-46b6-8a08-f4d88b40ce4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-k67vn"
Oct 31 13:05:00.902890 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:05:00.902862    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/070f030c-c00e-46b6-8a08-f4d88b40ce4a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-k67vn\" (UID: \"070f030c-c00e-46b6-8a08-f4d88b40ce4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-k67vn"
Oct 31 13:05:00.904633 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:05:00.904612    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/070f030c-c00e-46b6-8a08-f4d88b40ce4a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-k67vn\" (UID: \"070f030c-c00e-46b6-8a08-f4d88b40ce4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-k67vn"
Oct 31 13:05:00.914976 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:05:00.914950    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8q4sn\" (UniqueName: \"kubernetes.io/projected/070f030c-c00e-46b6-8a08-f4d88b40ce4a-kube-api-access-8q4sn\") pod \"maas-api-revoke-sub-3d4156f1-k67vn\" (UID: \"070f030c-c00e-46b6-8a08-f4d88b40ce4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-k67vn"
Oct 31 13:05:01.089704 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:05:01.089671    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-k67vn"
Oct 31 13:05:01.374214 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:05:01.374181    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-k67vn"]
Oct 31 13:05:01.392740 ip-10-0-130-190 kubenswrapper[5060]: W1031 13:05:01.392572    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod070f030c_c00e_46b6_8a08_f4d88b40ce4a.slice/crio-11e03d31542776cdb5a10a54b48536b85bed01ea97d8d0f3b14b925afa60ff45 WatchSource:0}: Error finding container 11e03d31542776cdb5a10a54b48536b85bed01ea97d8d0f3b14b925afa60ff45: Status 404 returned error can't find the container with id 11e03d31542776cdb5a10a54b48536b85bed01ea97d8d0f3b14b925afa60ff45
Oct 31 13:05:02.115242 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:05:02.115204    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-k67vn" event={"ID":"070f030c-c00e-46b6-8a08-f4d88b40ce4a","Type":"ContainerStarted","Data":"01e291317dd4316741f7239d731d1de93b16ca4b213ad504bbe23527591ddf47"}
Oct 31 13:05:02.115674 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:05:02.115263    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-k67vn" event={"ID":"070f030c-c00e-46b6-8a08-f4d88b40ce4a","Type":"ContainerStarted","Data":"11e03d31542776cdb5a10a54b48536b85bed01ea97d8d0f3b14b925afa60ff45"}
Oct 31 13:05:02.138256 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:05:02.138206    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-k67vn" podStartSLOduration=2.13819318 podStartE2EDuration="2.13819318s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:05:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:05:02.134760411 +0000 UTC m=+2265.237828123" watchObservedRunningTime="2026-10-31 13:05:02.13819318 +0000 UTC m=+2265.241260893"
Oct 31 13:05:03.123752 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:05:03.123713    5060 generic.go:356] "Generic (PLEG): container finished" podID="070f030c-c00e-46b6-8a08-f4d88b40ce4a" containerID="01e291317dd4316741f7239d731d1de93b16ca4b213ad504bbe23527591ddf47" exitCode=0
Oct 31 13:05:03.124246 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:05:03.123797    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-k67vn" event={"ID":"070f030c-c00e-46b6-8a08-f4d88b40ce4a","Type":"ContainerDied","Data":"01e291317dd4316741f7239d731d1de93b16ca4b213ad504bbe23527591ddf47"}
Oct 31 13:05:04.272288 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:05:04.272258    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-k67vn"
Oct 31 13:05:04.331623 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:05:04.331588    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/070f030c-c00e-46b6-8a08-f4d88b40ce4a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/070f030c-c00e-46b6-8a08-f4d88b40ce4a-maas-api-xks-ca\") pod \"070f030c-c00e-46b6-8a08-f4d88b40ce4a\" (UID: \"070f030c-c00e-46b6-8a08-f4d88b40ce4a\") "
Oct 31 13:05:04.331807 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:05:04.331788    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/070f030c-c00e-46b6-8a08-f4d88b40ce4a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/070f030c-c00e-46b6-8a08-f4d88b40ce4a-maas-api-openshift-ca\") pod \"070f030c-c00e-46b6-8a08-f4d88b40ce4a\" (UID: \"070f030c-c00e-46b6-8a08-f4d88b40ce4a\") "
Oct 31 13:05:04.331878 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:05:04.331839    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/070f030c-c00e-46b6-8a08-f4d88b40ce4a-kube-api-access-8q4sn\" (UniqueName: \"kubernetes.io/projected/070f030c-c00e-46b6-8a08-f4d88b40ce4a-kube-api-access-8q4sn\") pod \"070f030c-c00e-46b6-8a08-f4d88b40ce4a\" (UID: \"070f030c-c00e-46b6-8a08-f4d88b40ce4a\") "
Oct 31 13:05:04.332124 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:05:04.332100    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/070f030c-c00e-46b6-8a08-f4d88b40ce4a-maas-api-openshift-ca" pod "070f030c-c00e-46b6-8a08-f4d88b40ce4a" (UID: "070f030c-c00e-46b6-8a08-f4d88b40ce4a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:05:04.333609 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:05:04.333586    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/070f030c-c00e-46b6-8a08-f4d88b40ce4a-maas-api-xks-ca" pod "070f030c-c00e-46b6-8a08-f4d88b40ce4a" (UID: "070f030c-c00e-46b6-8a08-f4d88b40ce4a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:05:04.333894 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:05:04.333873    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/070f030c-c00e-46b6-8a08-f4d88b40ce4a-kube-api-access-8q4sn" pod "070f030c-c00e-46b6-8a08-f4d88b40ce4a" (UID: "070f030c-c00e-46b6-8a08-f4d88b40ce4a"). InnerVolumeSpecName "kube-api-access-8q4sn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:05:04.433320 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:05:04.433242    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/070f030c-c00e-46b6-8a08-f4d88b40ce4a-maas-api-openshift-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:05:04.433320 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:05:04.433273    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8q4sn\" (UniqueName: \"kubernetes.io/projected/070f030c-c00e-46b6-8a08-f4d88b40ce4a-kube-api-access-8q4sn\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:05:04.433320 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:05:04.433285    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/070f030c-c00e-46b6-8a08-f4d88b40ce4a-maas-api-xks-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:05:05.137721 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:05:05.137689    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-k67vn"
Oct 31 13:05:05.137721 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:05:05.137711    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-k67vn" event={"ID":"070f030c-c00e-46b6-8a08-f4d88b40ce4a","Type":"ContainerDied","Data":"11e03d31542776cdb5a10a54b48536b85bed01ea97d8d0f3b14b925afa60ff45"}
Oct 31 13:05:05.137945 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:05:05.137761    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="11e03d31542776cdb5a10a54b48536b85bed01ea97d8d0f3b14b925afa60ff45"
Oct 31 13:05:21.791248 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:05:21.791212    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-k67vn"]
Oct 31 13:05:21.799804 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:05:21.799776    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-k67vn"]
Oct 31 13:05:21.825087 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:05:21.825054    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4s7st"]
Oct 31 13:05:21.825518 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:05:21.825505    5060 cpu_manager.go:414] "containerMap: removing container" podUID="070f030c-c00e-46b6-8a08-f4d88b40ce4a" containerName="revoke-keys"
Oct 31 13:05:21.825568 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:05:21.825522    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="070f030c-c00e-46b6-8a08-f4d88b40ce4a" containerName="revoke-keys"
Oct 31 13:05:21.825623 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:05:21.825614    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="070f030c-c00e-46b6-8a08-f4d88b40ce4a" containerName="revoke-keys"
Oct 31 13:05:21.828894 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:05:21.828878    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4s7st"
Oct 31 13:05:21.833260 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:05:21.833235    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 13:05:21.833682 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:05:21.833656    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7wsmb\""
Oct 31 13:05:21.834571 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:05:21.834552    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 13:05:21.834700 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:05:21.834624    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 13:05:21.845382 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:05:21.845346    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4s7st"]
Oct 31 13:05:21.876372 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:05:21.876318    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f715829-b32e-4e08-aabe-ea34bdbe7565-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-4s7st\" (UID: \"6f715829-b32e-4e08-aabe-ea34bdbe7565\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4s7st"
Oct 31 13:05:21.876588 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:05:21.876470    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f715829-b32e-4e08-aabe-ea34bdbe7565-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-4s7st\" (UID: \"6f715829-b32e-4e08-aabe-ea34bdbe7565\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4s7st"
Oct 31 13:05:21.876692 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:05:21.876662    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ww6gb\" (UniqueName: \"kubernetes.io/projected/6f715829-b32e-4e08-aabe-ea34bdbe7565-kube-api-access-ww6gb\") pod \"maas-api-revoke-sub-3d4156f1-4s7st\" (UID: \"6f715829-b32e-4e08-aabe-ea34bdbe7565\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4s7st"
Oct 31 13:05:21.977659 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:05:21.977623    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ww6gb\" (UniqueName: \"kubernetes.io/projected/6f715829-b32e-4e08-aabe-ea34bdbe7565-kube-api-access-ww6gb\") pod \"maas-api-revoke-sub-3d4156f1-4s7st\" (UID: \"6f715829-b32e-4e08-aabe-ea34bdbe7565\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4s7st"
Oct 31 13:05:21.977836 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:05:21.977672    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f715829-b32e-4e08-aabe-ea34bdbe7565-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-4s7st\" (UID: \"6f715829-b32e-4e08-aabe-ea34bdbe7565\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4s7st"
Oct 31 13:05:21.977836 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:05:21.977744    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f715829-b32e-4e08-aabe-ea34bdbe7565-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-4s7st\" (UID: \"6f715829-b32e-4e08-aabe-ea34bdbe7565\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4s7st"
Oct 31 13:05:21.978357 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:05:21.978332    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f715829-b32e-4e08-aabe-ea34bdbe7565-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-4s7st\" (UID: \"6f715829-b32e-4e08-aabe-ea34bdbe7565\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4s7st"
Oct 31 13:05:21.980091 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:05:21.980060    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f715829-b32e-4e08-aabe-ea34bdbe7565-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-4s7st\" (UID: \"6f715829-b32e-4e08-aabe-ea34bdbe7565\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4s7st"
Oct 31 13:05:21.990024 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:05:21.989993    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ww6gb\" (UniqueName: \"kubernetes.io/projected/6f715829-b32e-4e08-aabe-ea34bdbe7565-kube-api-access-ww6gb\") pod \"maas-api-revoke-sub-3d4156f1-4s7st\" (UID: \"6f715829-b32e-4e08-aabe-ea34bdbe7565\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4s7st"
Oct 31 13:05:22.155348 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:05:22.155258    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4s7st"
Oct 31 13:05:22.443803 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:05:22.443769    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4s7st"]
Oct 31 13:05:22.444591 ip-10-0-130-190 kubenswrapper[5060]: W1031 13:05:22.444564    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f715829_b32e_4e08_aabe_ea34bdbe7565.slice/crio-8fd28706642609063d207aa97760b62597da131086cc5356f6736591135ed940 WatchSource:0}: Error finding container 8fd28706642609063d207aa97760b62597da131086cc5356f6736591135ed940: Status 404 returned error can't find the container with id 8fd28706642609063d207aa97760b62597da131086cc5356f6736591135ed940
Oct 31 13:05:23.047472 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:05:23.047440    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="070f030c-c00e-46b6-8a08-f4d88b40ce4a" path="/var/lib/kubelet/pods/070f030c-c00e-46b6-8a08-f4d88b40ce4a/volumes"
Oct 31 13:05:23.350430 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:05:23.350346    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4s7st" event={"ID":"6f715829-b32e-4e08-aabe-ea34bdbe7565","Type":"ContainerStarted","Data":"e8a6c278e8b9093e9137aadf967a65573d9dcd1d3ef6a2e0454e92190bae90ae"}
Oct 31 13:05:23.350430 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:05:23.350403    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4s7st" event={"ID":"6f715829-b32e-4e08-aabe-ea34bdbe7565","Type":"ContainerStarted","Data":"8fd28706642609063d207aa97760b62597da131086cc5356f6736591135ed940"}
Oct 31 13:05:23.379795 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:05:23.379751    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4s7st" podStartSLOduration=2.379737474 podStartE2EDuration="2.379737474s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:05:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:05:23.376673495 +0000 UTC m=+2286.479741208" watchObservedRunningTime="2026-10-31 13:05:23.379737474 +0000 UTC m=+2286.482805185"
Oct 31 13:05:24.358688 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:05:24.358650    5060 generic.go:356] "Generic (PLEG): container finished" podID="6f715829-b32e-4e08-aabe-ea34bdbe7565" containerID="e8a6c278e8b9093e9137aadf967a65573d9dcd1d3ef6a2e0454e92190bae90ae" exitCode=0
Oct 31 13:05:24.359151 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:05:24.358721    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4s7st" event={"ID":"6f715829-b32e-4e08-aabe-ea34bdbe7565","Type":"ContainerDied","Data":"e8a6c278e8b9093e9137aadf967a65573d9dcd1d3ef6a2e0454e92190bae90ae"}
Oct 31 13:05:25.509924 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:05:25.509895    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4s7st"
Oct 31 13:05:25.611695 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:05:25.611652    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f715829-b32e-4e08-aabe-ea34bdbe7565-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f715829-b32e-4e08-aabe-ea34bdbe7565-maas-api-openshift-ca\") pod \"6f715829-b32e-4e08-aabe-ea34bdbe7565\" (UID: \"6f715829-b32e-4e08-aabe-ea34bdbe7565\") "
Oct 31 13:05:25.611695 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:05:25.611702    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6f715829-b32e-4e08-aabe-ea34bdbe7565-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f715829-b32e-4e08-aabe-ea34bdbe7565-maas-api-xks-ca\") pod \"6f715829-b32e-4e08-aabe-ea34bdbe7565\" (UID: \"6f715829-b32e-4e08-aabe-ea34bdbe7565\") "
Oct 31 13:05:25.611978 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:05:25.611756    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f715829-b32e-4e08-aabe-ea34bdbe7565-kube-api-access-ww6gb\" (UniqueName: \"kubernetes.io/projected/6f715829-b32e-4e08-aabe-ea34bdbe7565-kube-api-access-ww6gb\") pod \"6f715829-b32e-4e08-aabe-ea34bdbe7565\" (UID: \"6f715829-b32e-4e08-aabe-ea34bdbe7565\") "
Oct 31 13:05:25.612104 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:05:25.612077    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f715829-b32e-4e08-aabe-ea34bdbe7565-maas-api-openshift-ca" pod "6f715829-b32e-4e08-aabe-ea34bdbe7565" (UID: "6f715829-b32e-4e08-aabe-ea34bdbe7565"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:05:25.613909 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:05:25.613884    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f715829-b32e-4e08-aabe-ea34bdbe7565-kube-api-access-ww6gb" pod "6f715829-b32e-4e08-aabe-ea34bdbe7565" (UID: "6f715829-b32e-4e08-aabe-ea34bdbe7565"). InnerVolumeSpecName "kube-api-access-ww6gb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:05:25.613988 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:05:25.613882    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f715829-b32e-4e08-aabe-ea34bdbe7565-maas-api-xks-ca" pod "6f715829-b32e-4e08-aabe-ea34bdbe7565" (UID: "6f715829-b32e-4e08-aabe-ea34bdbe7565"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:05:25.712953 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:05:25.712918    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f715829-b32e-4e08-aabe-ea34bdbe7565-maas-api-openshift-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:05:25.712953 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:05:25.712952    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f715829-b32e-4e08-aabe-ea34bdbe7565-maas-api-xks-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:05:25.713169 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:05:25.712966    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ww6gb\" (UniqueName: \"kubernetes.io/projected/6f715829-b32e-4e08-aabe-ea34bdbe7565-kube-api-access-ww6gb\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:05:26.374559 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:05:26.374527    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4s7st"
Oct 31 13:05:26.374559 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:05:26.374539    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4s7st" event={"ID":"6f715829-b32e-4e08-aabe-ea34bdbe7565","Type":"ContainerDied","Data":"8fd28706642609063d207aa97760b62597da131086cc5356f6736591135ed940"}
Oct 31 13:05:26.374767 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:05:26.374593    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8fd28706642609063d207aa97760b62597da131086cc5356f6736591135ed940"
Oct 31 13:05:26.734253 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:05:26.734170    5060 scope.go:122] "RemoveContainer" containerID="72181aa4d5b599cd120e0f81fe2adb34b0c8640625ac2c6611d87d03b36ffaf0"
Oct 31 13:05:26.744745 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:05:26.744723    5060 scope.go:122] "RemoveContainer" containerID="f3f56582bd0372765e78061fb6b025d1bf3ecbc8d30890360526cae3b072edb4"
Oct 31 13:05:26.753816 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:05:26.753795    5060 scope.go:122] "RemoveContainer" containerID="bcc9dff60dd5843791c36e7563fc86cdd0ded9d00b9a5d96269771a71bcca9e3"
Oct 31 13:05:26.762669 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:05:26.762648    5060 scope.go:122] "RemoveContainer" containerID="628a065de5c8ef62178bcfe61d7da9a579e462ff23678c576911709d365a628e"
Oct 31 13:05:26.771538 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:05:26.771517    5060 scope.go:122] "RemoveContainer" containerID="e52178036598dbfb506c27779018f4fbda6e60d94ff6f8ba3f97fb2dfb0fadd7"
Oct 31 13:05:29.921344 ip-10-0-130-190 kubenswrapper[5060]: E1031 13:05:29.921258    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7phb" containerName="extractor"
Oct 31 13:05:33.028762 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:05:33.028074    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4918fc6d-zr525"]
Oct 31 13:05:33.035566 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:05:33.035527    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4918fc6d-zr525"]
Oct 31 13:05:33.048008 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:05:33.047979    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5b0a48d-4c1d-4fef-a172-8e608048f155" path="/var/lib/kubelet/pods/a5b0a48d-4c1d-4fef-a172-8e608048f155/volumes"
Oct 31 13:05:51.032246 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:05:51.032208    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef807d2f-h9wvz"]
Oct 31 13:05:51.036354 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:05:51.036325    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef807d2f-h9wvz"]
Oct 31 13:05:51.047587 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:05:51.047561    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48f68ea6-2cb0-4e83-8d61-db1532be4735" path="/var/lib/kubelet/pods/48f68ea6-2cb0-4e83-8d61-db1532be4735/volumes"
Oct 31 13:05:59.921537 ip-10-0-130-190 kubenswrapper[5060]: E1031 13:05:59.921470    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7phb" containerName="extractor"
Oct 31 13:06:15.462562 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:06:15.462525    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4s7st"]
Oct 31 13:06:15.466117 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:06:15.466085    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4s7st"]
Oct 31 13:06:17.054652 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:06:17.054616    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f715829-b32e-4e08-aabe-ea34bdbe7565" path="/var/lib/kubelet/pods/6f715829-b32e-4e08-aabe-ea34bdbe7565/volumes"
Oct 31 13:06:26.817290 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:06:26.817236    5060 scope.go:122] "RemoveContainer" containerID="0a6da146988fd4d3c56f5c0c02e9fbc13603f4f47d5d91edf450faad7a84a23f"
Oct 31 13:06:26.831344 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:06:26.831310    5060 scope.go:122] "RemoveContainer" containerID="9beb8a228a0b9d09f03bd6061e406c42cfc070ae42d4225e719e6ad4b88b85e7"
Oct 31 13:06:26.922400 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:06:26.922359    5060 scope.go:122] "RemoveContainer" containerID="3fc7124fac130459a49f92a1b45743e6b3427fecbf7479aba369b3902b3efa7f"
Oct 31 13:06:26.983728 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:06:26.983702    5060 scope.go:122] "RemoveContainer" containerID="dfeda8a20ae6abe67354c38176d3ce94af9c708d58adf1d82fb9b2a0e395487e"
Oct 31 13:06:27.030248 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:06:27.030222    5060 scope.go:122] "RemoveContainer" containerID="ebbd7fa4f5bf47ae102fa6c68be365e19defd1b0d65517ab20f9d1d0f19721ae"
Oct 31 13:06:28.307516 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:06:28.300589    5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-645cf464d7-rz7mt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 13:06:28.307516 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:06:28.301040    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-rz7mt" podUID="978e5cf9-a89e-4fc9-b4ba-fe0ee2de401b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:06:28.379514 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:06:28.378607    5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-645cf464d7-7psf5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 13:06:28.379514 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:06:28.378692    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-7psf5" podUID="37aec2da-59cb-41ed-9e6b-a443f6ff84c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:06:29.923037 ip-10-0-130-190 kubenswrapper[5060]: E1031 13:06:29.923000    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7phb" containerName="extractor"
Oct 31 13:06:38.032233 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:06:38.032185    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-t6spd"]
Oct 31 13:06:38.034821 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:06:38.034795    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-t6spd"]
Oct 31 13:06:39.048690 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:06:39.048653    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5c01d42-6758-451e-9937-2f5bf4b97264" path="/var/lib/kubelet/pods/d5c01d42-6758-451e-9937-2f5bf4b97264/volumes"
Oct 31 13:06:59.921771 ip-10-0-130-190 kubenswrapper[5060]: E1031 13:06:59.921691    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7phb" containerName="extractor"
Oct 31 13:07:17.957071 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:07:17.957039    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pcvvm_f032909d-bc39-4c9e-99b7-f3d800842119/console-operator/2.log"
Oct 31 13:07:17.960754 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:07:17.960726    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pcvvm_f032909d-bc39-4c9e-99b7-f3d800842119/console-operator/2.log"
Oct 31 13:07:24.017156 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:07:24.017116    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5296c54-64sf7"]
Oct 31 13:07:24.017640 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:07:24.017624    5060 cpu_manager.go:414] "containerMap: removing container" podUID="6f715829-b32e-4e08-aabe-ea34bdbe7565" containerName="revoke-keys"
Oct 31 13:07:24.017699 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:07:24.017641    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f715829-b32e-4e08-aabe-ea34bdbe7565" containerName="revoke-keys"
Oct 31 13:07:24.017769 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:07:24.017715    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f715829-b32e-4e08-aabe-ea34bdbe7565" containerName="revoke-keys"
Oct 31 13:07:24.021010 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:07:24.020994    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5296c54-64sf7"
Oct 31 13:07:24.029159 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:07:24.029130    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 13:07:24.030378 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:07:24.030344    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 13:07:24.030745 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:07:24.030706    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7wsmb\""
Oct 31 13:07:24.030939 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:07:24.030723    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 13:07:24.036720 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:07:24.036690    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5296c54-64sf7"]
Oct 31 13:07:24.081033 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:07:24.081001    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjwgc\" (UniqueName: \"kubernetes.io/projected/d7d27637-37d0-4447-aa4f-7fb34a01adf1-kube-api-access-xjwgc\") pod \"maas-api-revoke-sub-c5296c54-64sf7\" (UID: \"d7d27637-37d0-4447-aa4f-7fb34a01adf1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5296c54-64sf7"
Oct 31 13:07:24.081197 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:07:24.081110    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7d27637-37d0-4447-aa4f-7fb34a01adf1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c5296c54-64sf7\" (UID: \"d7d27637-37d0-4447-aa4f-7fb34a01adf1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5296c54-64sf7"
Oct 31 13:07:24.081197 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:07:24.081138    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7d27637-37d0-4447-aa4f-7fb34a01adf1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c5296c54-64sf7\" (UID: \"d7d27637-37d0-4447-aa4f-7fb34a01adf1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5296c54-64sf7"
Oct 31 13:07:24.181745 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:07:24.181710    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjwgc\" (UniqueName: \"kubernetes.io/projected/d7d27637-37d0-4447-aa4f-7fb34a01adf1-kube-api-access-xjwgc\") pod \"maas-api-revoke-sub-c5296c54-64sf7\" (UID: \"d7d27637-37d0-4447-aa4f-7fb34a01adf1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5296c54-64sf7"
Oct 31 13:07:24.181934 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:07:24.181800    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7d27637-37d0-4447-aa4f-7fb34a01adf1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c5296c54-64sf7\" (UID: \"d7d27637-37d0-4447-aa4f-7fb34a01adf1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5296c54-64sf7"
Oct 31 13:07:24.181934 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:07:24.181823    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7d27637-37d0-4447-aa4f-7fb34a01adf1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c5296c54-64sf7\" (UID: \"d7d27637-37d0-4447-aa4f-7fb34a01adf1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5296c54-64sf7"
Oct 31 13:07:24.182599 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:07:24.182575    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7d27637-37d0-4447-aa4f-7fb34a01adf1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c5296c54-64sf7\" (UID: \"d7d27637-37d0-4447-aa4f-7fb34a01adf1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5296c54-64sf7"
Oct 31 13:07:24.184435 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:07:24.184404    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7d27637-37d0-4447-aa4f-7fb34a01adf1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c5296c54-64sf7\" (UID: \"d7d27637-37d0-4447-aa4f-7fb34a01adf1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5296c54-64sf7"
Oct 31 13:07:24.195188 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:07:24.195159    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjwgc\" (UniqueName: \"kubernetes.io/projected/d7d27637-37d0-4447-aa4f-7fb34a01adf1-kube-api-access-xjwgc\") pod \"maas-api-revoke-sub-c5296c54-64sf7\" (UID: \"d7d27637-37d0-4447-aa4f-7fb34a01adf1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5296c54-64sf7"
Oct 31 13:07:24.342742 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:07:24.342709    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5296c54-64sf7"
Oct 31 13:07:24.627295 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:07:24.627267    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5296c54-64sf7"]
Oct 31 13:07:25.382191 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:07:25.382145    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5296c54-64sf7" event={"ID":"d7d27637-37d0-4447-aa4f-7fb34a01adf1","Type":"ContainerStarted","Data":"2ae5f85cf2ed9e2df6b3a421749de0ebc3afc04538c3444508a8800d7d7a638a"}
Oct 31 13:07:25.382915 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:07:25.382208    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5296c54-64sf7" event={"ID":"d7d27637-37d0-4447-aa4f-7fb34a01adf1","Type":"ContainerStarted","Data":"135e2d8fa1a7abcebb1e2828bfadbfc4324f14c6e7f813660fc0644ed1804bd6"}
Oct 31 13:07:25.400659 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:07:25.400610    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5296c54-64sf7" podStartSLOduration=2.400597235 podStartE2EDuration="2.400597235s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:07:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:07:25.399420828 +0000 UTC m=+2408.502488540" watchObservedRunningTime="2026-10-31 13:07:25.400597235 +0000 UTC m=+2408.503664948"
Oct 31 13:07:26.390022 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:07:26.389988    5060 generic.go:356] "Generic (PLEG): container finished" podID="d7d27637-37d0-4447-aa4f-7fb34a01adf1" containerID="2ae5f85cf2ed9e2df6b3a421749de0ebc3afc04538c3444508a8800d7d7a638a" exitCode=0
Oct 31 13:07:26.390414 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:07:26.390069    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5296c54-64sf7" event={"ID":"d7d27637-37d0-4447-aa4f-7fb34a01adf1","Type":"ContainerDied","Data":"2ae5f85cf2ed9e2df6b3a421749de0ebc3afc04538c3444508a8800d7d7a638a"}
Oct 31 13:07:27.231026 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:07:27.230993    5060 scope.go:122] "RemoveContainer" containerID="633c4a8874cd6c68b1bd9b37c36fa00c02bcf97cf4e27419b166b768dade1bab"
Oct 31 13:07:27.241237 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:07:27.241212    5060 scope.go:122] "RemoveContainer" containerID="a281ef5accb16b0b9e3d81b3447776184f312bb530b36af79df4589523999ce8"
Oct 31 13:07:27.250985 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:07:27.250965    5060 scope.go:122] "RemoveContainer" containerID="52adb4beb2b982405dba128837cf938918914beb34d14851c7e7c42cfef7217c"
Oct 31 13:07:27.537857 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:07:27.537831    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5296c54-64sf7"
Oct 31 13:07:27.615605 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:07:27.615572    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7d27637-37d0-4447-aa4f-7fb34a01adf1-kube-api-access-xjwgc\" (UniqueName: \"kubernetes.io/projected/d7d27637-37d0-4447-aa4f-7fb34a01adf1-kube-api-access-xjwgc\") pod \"d7d27637-37d0-4447-aa4f-7fb34a01adf1\" (UID: \"d7d27637-37d0-4447-aa4f-7fb34a01adf1\") "
Oct 31 13:07:27.615798 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:07:27.615661    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7d27637-37d0-4447-aa4f-7fb34a01adf1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7d27637-37d0-4447-aa4f-7fb34a01adf1-maas-api-openshift-ca\") pod \"d7d27637-37d0-4447-aa4f-7fb34a01adf1\" (UID: \"d7d27637-37d0-4447-aa4f-7fb34a01adf1\") "
Oct 31 13:07:27.615798 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:07:27.615732    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d7d27637-37d0-4447-aa4f-7fb34a01adf1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7d27637-37d0-4447-aa4f-7fb34a01adf1-maas-api-xks-ca\") pod \"d7d27637-37d0-4447-aa4f-7fb34a01adf1\" (UID: \"d7d27637-37d0-4447-aa4f-7fb34a01adf1\") "
Oct 31 13:07:27.616018 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:07:27.615991    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7d27637-37d0-4447-aa4f-7fb34a01adf1-maas-api-openshift-ca" pod "d7d27637-37d0-4447-aa4f-7fb34a01adf1" (UID: "d7d27637-37d0-4447-aa4f-7fb34a01adf1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:07:27.617749 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:07:27.617719    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7d27637-37d0-4447-aa4f-7fb34a01adf1-maas-api-xks-ca" pod "d7d27637-37d0-4447-aa4f-7fb34a01adf1" (UID: "d7d27637-37d0-4447-aa4f-7fb34a01adf1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:07:27.617826 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:07:27.617735    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7d27637-37d0-4447-aa4f-7fb34a01adf1-kube-api-access-xjwgc" pod "d7d27637-37d0-4447-aa4f-7fb34a01adf1" (UID: "d7d27637-37d0-4447-aa4f-7fb34a01adf1"). InnerVolumeSpecName "kube-api-access-xjwgc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:07:27.716472 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:07:27.716439    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7d27637-37d0-4447-aa4f-7fb34a01adf1-maas-api-xks-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:07:27.716472 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:07:27.716467    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjwgc\" (UniqueName: \"kubernetes.io/projected/d7d27637-37d0-4447-aa4f-7fb34a01adf1-kube-api-access-xjwgc\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:07:27.716472 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:07:27.716479    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7d27637-37d0-4447-aa4f-7fb34a01adf1-maas-api-openshift-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:07:28.405552 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:07:28.405520    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5296c54-64sf7"
Oct 31 13:07:28.405708 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:07:28.405520    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5296c54-64sf7" event={"ID":"d7d27637-37d0-4447-aa4f-7fb34a01adf1","Type":"ContainerDied","Data":"135e2d8fa1a7abcebb1e2828bfadbfc4324f14c6e7f813660fc0644ed1804bd6"}
Oct 31 13:07:28.405708 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:07:28.405649    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="135e2d8fa1a7abcebb1e2828bfadbfc4324f14c6e7f813660fc0644ed1804bd6"
Oct 31 13:07:29.920352 ip-10-0-130-190 kubenswrapper[5060]: E1031 13:07:29.920310    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7phb" containerName="extractor"
Oct 31 13:07:32.036059 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:07:32.036022    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-dq8tm"]
Oct 31 13:07:32.039792 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:07:32.039765    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-dq8tm"]
Oct 31 13:07:33.047506 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:07:33.047461    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9116105-64ca-4d5b-8620-a5f60a504e5c" path="/var/lib/kubelet/pods/e9116105-64ca-4d5b-8620-a5f60a504e5c/volumes"
Oct 31 13:07:34.991452 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:07:34.991398    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-aa9688-b5r68"]
Oct 31 13:07:34.991949 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:07:34.991934    5060 cpu_manager.go:414] "containerMap: removing container" podUID="d7d27637-37d0-4447-aa4f-7fb34a01adf1" containerName="revoke-keys"
Oct 31 13:07:34.991998 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:07:34.991950    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7d27637-37d0-4447-aa4f-7fb34a01adf1" containerName="revoke-keys"
Oct 31 13:07:34.992035 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:07:34.992029    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7d27637-37d0-4447-aa4f-7fb34a01adf1" containerName="revoke-keys"
Oct 31 13:07:34.995665 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:07:34.995645    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-aa9688-b5r68"
Oct 31 13:07:34.999635 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:07:34.999606    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 13:07:34.999850 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:07:34.999828    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 13:07:35.000768 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:07:35.000742    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7wsmb\""
Oct 31 13:07:35.000893 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:07:35.000745    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 13:07:35.002947 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:07:35.002924    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-aa9688-b5r68"]
Oct 31 13:07:35.091978 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:07:35.091940    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09d94163-9a77-4b0e-8165-bc2bfe6998f3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-aa9688-b5r68\" (UID: \"09d94163-9a77-4b0e-8165-bc2bfe6998f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-aa9688-b5r68"
Oct 31 13:07:35.092196 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:07:35.091998    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09d94163-9a77-4b0e-8165-bc2bfe6998f3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-aa9688-b5r68\" (UID: \"09d94163-9a77-4b0e-8165-bc2bfe6998f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-aa9688-b5r68"
Oct 31 13:07:35.092196 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:07:35.092090    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nn6gq\" (UniqueName: \"kubernetes.io/projected/09d94163-9a77-4b0e-8165-bc2bfe6998f3-kube-api-access-nn6gq\") pod \"maas-api-revoke-keys-e2e-worker-single-main-aa9688-b5r68\" (UID: \"09d94163-9a77-4b0e-8165-bc2bfe6998f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-aa9688-b5r68"
Oct 31 13:07:35.193388 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:07:35.193348    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09d94163-9a77-4b0e-8165-bc2bfe6998f3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-aa9688-b5r68\" (UID: \"09d94163-9a77-4b0e-8165-bc2bfe6998f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-aa9688-b5r68"
Oct 31 13:07:35.193601 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:07:35.193396    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nn6gq\" (UniqueName: \"kubernetes.io/projected/09d94163-9a77-4b0e-8165-bc2bfe6998f3-kube-api-access-nn6gq\") pod \"maas-api-revoke-keys-e2e-worker-single-main-aa9688-b5r68\" (UID: \"09d94163-9a77-4b0e-8165-bc2bfe6998f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-aa9688-b5r68"
Oct 31 13:07:35.193601 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:07:35.193475    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09d94163-9a77-4b0e-8165-bc2bfe6998f3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-aa9688-b5r68\" (UID: \"09d94163-9a77-4b0e-8165-bc2bfe6998f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-aa9688-b5r68"
Oct 31 13:07:35.194098 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:07:35.194075    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09d94163-9a77-4b0e-8165-bc2bfe6998f3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-aa9688-b5r68\" (UID: \"09d94163-9a77-4b0e-8165-bc2bfe6998f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-aa9688-b5r68"
Oct 31 13:07:35.195986 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:07:35.195960    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09d94163-9a77-4b0e-8165-bc2bfe6998f3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-aa9688-b5r68\" (UID: \"09d94163-9a77-4b0e-8165-bc2bfe6998f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-aa9688-b5r68"
Oct 31 13:07:35.206868 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:07:35.206842    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nn6gq\" (UniqueName: \"kubernetes.io/projected/09d94163-9a77-4b0e-8165-bc2bfe6998f3-kube-api-access-nn6gq\") pod \"maas-api-revoke-keys-e2e-worker-single-main-aa9688-b5r68\" (UID: \"09d94163-9a77-4b0e-8165-bc2bfe6998f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-aa9688-b5r68"
Oct 31 13:07:35.312817 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:07:35.312782    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-aa9688-b5r68"
Oct 31 13:07:35.604308 ip-10-0-130-190 kubenswrapper[5060]: W1031 13:07:35.604272    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09d94163_9a77_4b0e_8165_bc2bfe6998f3.slice/crio-845d221cb88709ceec389a101f8ea957c2c88521f782c0f1e1b04114c2913205 WatchSource:0}: Error finding container 845d221cb88709ceec389a101f8ea957c2c88521f782c0f1e1b04114c2913205: Status 404 returned error can't find the container with id 845d221cb88709ceec389a101f8ea957c2c88521f782c0f1e1b04114c2913205
Oct 31 13:07:35.604466 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:07:35.604433    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-aa9688-b5r68"]
Oct 31 13:07:36.465538 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:07:36.465485    5060 generic.go:356] "Generic (PLEG): container finished" podID="09d94163-9a77-4b0e-8165-bc2bfe6998f3" containerID="7180bd558d22c92ccb691f33bb587f3e9a19ca42447a559a48700de659dcc3b4" exitCode=0
Oct 31 13:07:36.465538 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:07:36.465521    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-aa9688-b5r68" event={"ID":"09d94163-9a77-4b0e-8165-bc2bfe6998f3","Type":"ContainerDied","Data":"7180bd558d22c92ccb691f33bb587f3e9a19ca42447a559a48700de659dcc3b4"}
Oct 31 13:07:36.466034 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:07:36.465573    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-aa9688-b5r68" event={"ID":"09d94163-9a77-4b0e-8165-bc2bfe6998f3","Type":"ContainerStarted","Data":"845d221cb88709ceec389a101f8ea957c2c88521f782c0f1e1b04114c2913205"}
Oct 31 13:07:37.632074 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:07:37.632046    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-aa9688-b5r68"
Oct 31 13:07:37.719224 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:07:37.719191    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09d94163-9a77-4b0e-8165-bc2bfe6998f3-kube-api-access-nn6gq\" (UniqueName: \"kubernetes.io/projected/09d94163-9a77-4b0e-8165-bc2bfe6998f3-kube-api-access-nn6gq\") pod \"09d94163-9a77-4b0e-8165-bc2bfe6998f3\" (UID: \"09d94163-9a77-4b0e-8165-bc2bfe6998f3\") "
Oct 31 13:07:37.719392 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:07:37.719287    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09d94163-9a77-4b0e-8165-bc2bfe6998f3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09d94163-9a77-4b0e-8165-bc2bfe6998f3-maas-api-openshift-ca\") pod \"09d94163-9a77-4b0e-8165-bc2bfe6998f3\" (UID: \"09d94163-9a77-4b0e-8165-bc2bfe6998f3\") "
Oct 31 13:07:37.719392 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:07:37.719341    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/09d94163-9a77-4b0e-8165-bc2bfe6998f3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09d94163-9a77-4b0e-8165-bc2bfe6998f3-maas-api-xks-ca\") pod \"09d94163-9a77-4b0e-8165-bc2bfe6998f3\" (UID: \"09d94163-9a77-4b0e-8165-bc2bfe6998f3\") "
Oct 31 13:07:37.719705 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:07:37.719681    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09d94163-9a77-4b0e-8165-bc2bfe6998f3-maas-api-openshift-ca" pod "09d94163-9a77-4b0e-8165-bc2bfe6998f3" (UID: "09d94163-9a77-4b0e-8165-bc2bfe6998f3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:07:37.721350 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:07:37.721321    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09d94163-9a77-4b0e-8165-bc2bfe6998f3-maas-api-xks-ca" pod "09d94163-9a77-4b0e-8165-bc2bfe6998f3" (UID: "09d94163-9a77-4b0e-8165-bc2bfe6998f3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:07:37.721476 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:07:37.721353    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09d94163-9a77-4b0e-8165-bc2bfe6998f3-kube-api-access-nn6gq" pod "09d94163-9a77-4b0e-8165-bc2bfe6998f3" (UID: "09d94163-9a77-4b0e-8165-bc2bfe6998f3"). InnerVolumeSpecName "kube-api-access-nn6gq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:07:37.820540 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:07:37.820486    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09d94163-9a77-4b0e-8165-bc2bfe6998f3-maas-api-openshift-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:07:37.820540 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:07:37.820536    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09d94163-9a77-4b0e-8165-bc2bfe6998f3-maas-api-xks-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:07:37.820540 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:07:37.820546    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nn6gq\" (UniqueName: \"kubernetes.io/projected/09d94163-9a77-4b0e-8165-bc2bfe6998f3-kube-api-access-nn6gq\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:07:38.479965 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:07:38.479923    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-aa9688-b5r68" event={"ID":"09d94163-9a77-4b0e-8165-bc2bfe6998f3","Type":"ContainerDied","Data":"845d221cb88709ceec389a101f8ea957c2c88521f782c0f1e1b04114c2913205"}
Oct 31 13:07:38.480199 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:07:38.479985    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="845d221cb88709ceec389a101f8ea957c2c88521f782c0f1e1b04114c2913205"
Oct 31 13:07:38.480199 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:07:38.479939    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-aa9688-b5r68"
Oct 31 13:07:46.302838 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:07:46.302801    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-aa9688-b5r68"]
Oct 31 13:07:46.309635 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:07:46.309607    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-aa9688-b5r68"]
Oct 31 13:07:47.049452 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:07:47.049414    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09d94163-9a77-4b0e-8165-bc2bfe6998f3" path="/var/lib/kubelet/pods/09d94163-9a77-4b0e-8165-bc2bfe6998f3/volumes"
Oct 31 13:07:51.030775 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:07:51.030739    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-csff5"]
Oct 31 13:07:51.034517 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:07:51.034478    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-csff5"]
Oct 31 13:07:51.046621 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:07:51.046595    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="114f1bed-041e-48f6-9bc9-12d816bdc5ad" path="/var/lib/kubelet/pods/114f1bed-041e-48f6-9bc9-12d816bdc5ad/volumes"
Oct 31 13:07:59.921376 ip-10-0-130-190 kubenswrapper[5060]: E1031 13:07:59.921340    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7phb" containerName="extractor"
Oct 31 13:08:02.037503 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:08:02.037455    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-lk95k"]
Oct 31 13:08:02.040187 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:08:02.040159    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-lk95k"]
Oct 31 13:08:03.046927 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:08:03.046892    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ba9cdab-bb2b-4aa0-9cef-180597f674a2" path="/var/lib/kubelet/pods/6ba9cdab-bb2b-4aa0-9cef-180597f674a2/volumes"
Oct 31 13:08:15.753288 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:08:15.753252    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7pf4p"]
Oct 31 13:08:15.753748 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:08:15.753484    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7pf4p" podUID="0f6d8e05-a27e-49b1-ba33-cfea30956f83" containerName="manager" containerID="cri-o://ccdaf64db1370f052353f5b067fc2aa20246e7e773bb32a9850de4065f1db8ff" gracePeriod=10
Oct 31 13:08:16.012890 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:08:16.012831    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7pf4p"
Oct 31 13:08:16.178950 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:08:16.178911    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f6d8e05-a27e-49b1-ba33-cfea30956f83-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0f6d8e05-a27e-49b1-ba33-cfea30956f83-extensions-socket-volume\") pod \"0f6d8e05-a27e-49b1-ba33-cfea30956f83\" (UID: \"0f6d8e05-a27e-49b1-ba33-cfea30956f83\") "
Oct 31 13:08:16.179155 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:08:16.178969    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f6d8e05-a27e-49b1-ba33-cfea30956f83-kube-api-access-47b6x\" (UniqueName: \"kubernetes.io/projected/0f6d8e05-a27e-49b1-ba33-cfea30956f83-kube-api-access-47b6x\") pod \"0f6d8e05-a27e-49b1-ba33-cfea30956f83\" (UID: \"0f6d8e05-a27e-49b1-ba33-cfea30956f83\") "
Oct 31 13:08:16.179291 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:08:16.179267    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f6d8e05-a27e-49b1-ba33-cfea30956f83-extensions-socket-volume" pod "0f6d8e05-a27e-49b1-ba33-cfea30956f83" (UID: "0f6d8e05-a27e-49b1-ba33-cfea30956f83"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:08:16.181008 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:08:16.180972    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f6d8e05-a27e-49b1-ba33-cfea30956f83-kube-api-access-47b6x" pod "0f6d8e05-a27e-49b1-ba33-cfea30956f83" (UID: "0f6d8e05-a27e-49b1-ba33-cfea30956f83"). InnerVolumeSpecName "kube-api-access-47b6x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:08:16.280355 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:08:16.280318    5060 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0f6d8e05-a27e-49b1-ba33-cfea30956f83-extensions-socket-volume\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:08:16.280355 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:08:16.280350    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47b6x\" (UniqueName: \"kubernetes.io/projected/0f6d8e05-a27e-49b1-ba33-cfea30956f83-kube-api-access-47b6x\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:08:16.764470 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:08:16.764381    5060 generic.go:356] "Generic (PLEG): container finished" podID="0f6d8e05-a27e-49b1-ba33-cfea30956f83" containerID="ccdaf64db1370f052353f5b067fc2aa20246e7e773bb32a9850de4065f1db8ff" exitCode=0
Oct 31 13:08:16.764470 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:08:16.764430    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7pf4p"
Oct 31 13:08:16.764973 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:08:16.764462    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7pf4p" event={"ID":"0f6d8e05-a27e-49b1-ba33-cfea30956f83","Type":"ContainerDied","Data":"ccdaf64db1370f052353f5b067fc2aa20246e7e773bb32a9850de4065f1db8ff"}
Oct 31 13:08:16.764973 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:08:16.764541    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7pf4p" event={"ID":"0f6d8e05-a27e-49b1-ba33-cfea30956f83","Type":"ContainerDied","Data":"c89d7fccb9320216f02f56118726d48d40dc7932db2b55817e3deb8310a81edd"}
Oct 31 13:08:16.764973 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:08:16.764567    5060 scope.go:122] "RemoveContainer" containerID="ccdaf64db1370f052353f5b067fc2aa20246e7e773bb32a9850de4065f1db8ff"
Oct 31 13:08:16.777034 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:08:16.777014    5060 scope.go:122] "RemoveContainer" containerID="ccdaf64db1370f052353f5b067fc2aa20246e7e773bb32a9850de4065f1db8ff"
Oct 31 13:08:16.777348 ip-10-0-130-190 kubenswrapper[5060]: E1031 13:08:16.777327    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ccdaf64db1370f052353f5b067fc2aa20246e7e773bb32a9850de4065f1db8ff\": container with ID starting with ccdaf64db1370f052353f5b067fc2aa20246e7e773bb32a9850de4065f1db8ff not found: ID does not exist" containerID="ccdaf64db1370f052353f5b067fc2aa20246e7e773bb32a9850de4065f1db8ff"
Oct 31 13:08:16.777431 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:08:16.777362    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ccdaf64db1370f052353f5b067fc2aa20246e7e773bb32a9850de4065f1db8ff"} err="failed to get container status \"ccdaf64db1370f052353f5b067fc2aa20246e7e773bb32a9850de4065f1db8ff\": rpc error: code = NotFound desc = could not find container \"ccdaf64db1370f052353f5b067fc2aa20246e7e773bb32a9850de4065f1db8ff\": container with ID starting with ccdaf64db1370f052353f5b067fc2aa20246e7e773bb32a9850de4065f1db8ff not found: ID does not exist"
Oct 31 13:08:16.794276 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:08:16.794251    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7pf4p"]
Oct 31 13:08:16.801923 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:08:16.801897    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7pf4p"]
Oct 31 13:08:17.047865 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:08:17.047835    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f6d8e05-a27e-49b1-ba33-cfea30956f83" path="/var/lib/kubelet/pods/0f6d8e05-a27e-49b1-ba33-cfea30956f83/volumes"
Oct 31 13:08:27.285049 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:08:27.285016    5060 scope.go:122] "RemoveContainer" containerID="2dba03e2aa46a9afb3149277531c5664585b2bd6764c97ce2da43ebe7b1d634f"
Oct 31 13:08:28.026567 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:08:28.026470    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vzdmm"]
Oct 31 13:08:28.030017 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:08:28.029990    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vzdmm"]
Oct 31 13:08:29.047039 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:08:29.047005    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68253505-aee8-43f7-bd54-aefce9f53e8c" path="/var/lib/kubelet/pods/68253505-aee8-43f7-bd54-aefce9f53e8c/volumes"
Oct 31 13:08:29.921178 ip-10-0-130-190 kubenswrapper[5060]: E1031 13:08:29.921139    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7phb" containerName="extractor"
Oct 31 13:08:39.023193 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:08:39.023152    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-n5mwc"]
Oct 31 13:08:39.028423 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:08:39.028394    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-n5mwc"]
Oct 31 13:08:39.047465 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:08:39.047435    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ab0af02-0da7-4d8f-9e72-53f16ee23c18" path="/var/lib/kubelet/pods/3ab0af02-0da7-4d8f-9e72-53f16ee23c18/volumes"
Oct 31 13:08:59.025507 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:08:59.025448    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-lkrg5"]
Oct 31 13:08:59.028442 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:08:59.028419    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-lkrg5"]
Oct 31 13:08:59.046308 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:08:59.046279    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c283cc5-c508-43bb-a626-f0eedaac259e" path="/var/lib/kubelet/pods/5c283cc5-c508-43bb-a626-f0eedaac259e/volumes"
Oct 31 13:08:59.920906 ip-10-0-130-190 kubenswrapper[5060]: E1031 13:08:59.920871    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7phb" containerName="extractor"
Oct 31 13:09:16.800143 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:09:16.800106    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lqk5j"]
Oct 31 13:09:16.800664 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:09:16.800647    5060 cpu_manager.go:414] "containerMap: removing container" podUID="0f6d8e05-a27e-49b1-ba33-cfea30956f83" containerName="manager"
Oct 31 13:09:16.800717 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:09:16.800665    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f6d8e05-a27e-49b1-ba33-cfea30956f83" containerName="manager"
Oct 31 13:09:16.800717 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:09:16.800683    5060 cpu_manager.go:414] "containerMap: removing container" podUID="09d94163-9a77-4b0e-8165-bc2bfe6998f3" containerName="revoke-keys"
Oct 31 13:09:16.800717 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:09:16.800689    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09d94163-9a77-4b0e-8165-bc2bfe6998f3" containerName="revoke-keys"
Oct 31 13:09:16.800816 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:09:16.800777    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f6d8e05-a27e-49b1-ba33-cfea30956f83" containerName="manager"
Oct 31 13:09:16.800816 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:09:16.800788    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="09d94163-9a77-4b0e-8165-bc2bfe6998f3" containerName="revoke-keys"
Oct 31 13:09:16.804019 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:09:16.804001    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lqk5j"
Oct 31 13:09:16.808826 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:09:16.808802    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-l2tnf\""
Oct 31 13:09:16.817506 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:09:16.817468    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lqk5j"]
Oct 31 13:09:16.849275 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:09:16.849245    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqtm6\" (UniqueName: \"kubernetes.io/projected/d289b4b1-3490-492e-95a7-f170cddbfe63-kube-api-access-jqtm6\") pod \"kuadrant-operator-controller-manager-678d87766f-lqk5j\" (UID: \"d289b4b1-3490-492e-95a7-f170cddbfe63\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lqk5j"
Oct 31 13:09:16.849427 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:09:16.849356    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d289b4b1-3490-492e-95a7-f170cddbfe63-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-lqk5j\" (UID: \"d289b4b1-3490-492e-95a7-f170cddbfe63\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lqk5j"
Oct 31 13:09:16.950218 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:09:16.950182    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d289b4b1-3490-492e-95a7-f170cddbfe63-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-lqk5j\" (UID: \"d289b4b1-3490-492e-95a7-f170cddbfe63\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lqk5j"
Oct 31 13:09:16.950382 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:09:16.950329    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqtm6\" (UniqueName: \"kubernetes.io/projected/d289b4b1-3490-492e-95a7-f170cddbfe63-kube-api-access-jqtm6\") pod \"kuadrant-operator-controller-manager-678d87766f-lqk5j\" (UID: \"d289b4b1-3490-492e-95a7-f170cddbfe63\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lqk5j"
Oct 31 13:09:16.950631 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:09:16.950601    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d289b4b1-3490-492e-95a7-f170cddbfe63-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-lqk5j\" (UID: \"d289b4b1-3490-492e-95a7-f170cddbfe63\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lqk5j"
Oct 31 13:09:16.962470 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:09:16.962442    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqtm6\" (UniqueName: \"kubernetes.io/projected/d289b4b1-3490-492e-95a7-f170cddbfe63-kube-api-access-jqtm6\") pod \"kuadrant-operator-controller-manager-678d87766f-lqk5j\" (UID: \"d289b4b1-3490-492e-95a7-f170cddbfe63\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lqk5j"
Oct 31 13:09:17.061614 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:09:17.061539    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-g48rp"]
Oct 31 13:09:17.069202 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:09:17.069173    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-g48rp"]
Oct 31 13:09:17.118484 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:09:17.118457    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lqk5j"
Oct 31 13:09:17.416014 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:09:17.415989    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lqk5j"]
Oct 31 13:09:17.419119 ip-10-0-130-190 kubenswrapper[5060]: W1031 13:09:17.419089    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd289b4b1_3490_492e_95a7_f170cddbfe63.slice/crio-cc3d809a65ac90db3aa94c42f104d734249cee8e4888d971ac1a2461716d46cf WatchSource:0}: Error finding container cc3d809a65ac90db3aa94c42f104d734249cee8e4888d971ac1a2461716d46cf: Status 404 returned error can't find the container with id cc3d809a65ac90db3aa94c42f104d734249cee8e4888d971ac1a2461716d46cf
Oct 31 13:09:18.153223 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:09:18.153182    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lqk5j" event={"ID":"d289b4b1-3490-492e-95a7-f170cddbfe63","Type":"ContainerStarted","Data":"b30db3103054032bc686ef6834b6e22f2932352effd96040a9763de3b3b2a461"}
Oct 31 13:09:18.153651 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:09:18.153246    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lqk5j" event={"ID":"d289b4b1-3490-492e-95a7-f170cddbfe63","Type":"ContainerStarted","Data":"cc3d809a65ac90db3aa94c42f104d734249cee8e4888d971ac1a2461716d46cf"}
Oct 31 13:09:18.153651 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:09:18.153421    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lqk5j"
Oct 31 13:09:18.179224 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:09:18.179167    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lqk5j" podStartSLOduration=2.179153537 podStartE2EDuration="2.179153537s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:09:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:09:18.177786814 +0000 UTC m=+2521.280854531" watchObservedRunningTime="2026-10-31 13:09:18.179153537 +0000 UTC m=+2521.282221248"
Oct 31 13:09:19.046820 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:09:19.046790    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a33e935-ef9b-4ded-a616-f0996f626f20" path="/var/lib/kubelet/pods/7a33e935-ef9b-4ded-a616-f0996f626f20/volumes"
Oct 31 13:09:27.328622 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:09:27.328588    5060 scope.go:122] "RemoveContainer" containerID="bd95b88425e537f69341837457b26afdfd57004325881bd311ec1db16c3bb116"
Oct 31 13:09:27.338375 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:09:27.338353    5060 scope.go:122] "RemoveContainer" containerID="6d049363621f24156515c9980e6a206c21951a3bf5998b93cbc8686ed24f28f4"
Oct 31 13:09:27.346816 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:09:27.346791    5060 scope.go:122] "RemoveContainer" containerID="1ebfd62e4daf98266087abb4a85e97522fd52c0fbdc2e5c458fc3f0912086667"
Oct 31 13:09:27.357827 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:09:27.357804    5060 scope.go:122] "RemoveContainer" containerID="147f2c65b917848d42aa5eae80331ec59e3074e1162b8bfb98790a4fca649c4d"
Oct 31 13:09:29.162796 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:09:29.162761    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lqk5j"
Oct 31 13:09:29.920900 ip-10-0-130-190 kubenswrapper[5060]: E1031 13:09:29.920857    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7phb" containerName="extractor"
Oct 31 13:09:36.023118 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:09:36.023078    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-g8np6"]
Oct 31 13:09:36.025728 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:09:36.025705    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-g8np6"]
Oct 31 13:09:37.051279 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:09:37.051234    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27d9938b-04e6-4792-aaca-78da1f022533" path="/var/lib/kubelet/pods/27d9938b-04e6-4792-aaca-78da1f022533/volumes"
Oct 31 13:09:54.027163 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:09:54.027116    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-f986r"]
Oct 31 13:09:54.031325 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:09:54.031290    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-f986r"]
Oct 31 13:09:55.071514 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:09:55.070838    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b93b36c-83bf-48c4-9492-ca4384f1827d" path="/var/lib/kubelet/pods/4b93b36c-83bf-48c4-9492-ca4384f1827d/volumes"
Oct 31 13:09:59.920361 ip-10-0-130-190 kubenswrapper[5060]: E1031 13:09:59.920275    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7phb" containerName="extractor"
Oct 31 13:10:27.395752 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:10:27.395712    5060 scope.go:122] "RemoveContainer" containerID="163c2802d750ecc725f38129ddba4e3836b0d86d67d90eb2295b2db14c4e9078"
Oct 31 13:10:27.405234 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:10:27.405193    5060 scope.go:122] "RemoveContainer" containerID="be5ce9257ee010965feefd2475b7e08e127f6bf232441960051f640d9cc232dd"
Oct 31 13:10:27.413901 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:10:27.413878    5060 scope.go:122] "RemoveContainer" containerID="2ecafe6bc658820b6b0848fc74b124c014b28dff1a301b36f5e534ee5604f48c"
Oct 31 13:10:29.924600 ip-10-0-130-190 kubenswrapper[5060]: E1031 13:10:29.924556    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7phb" containerName="extractor"
Oct 31 13:10:59.921049 ip-10-0-130-190 kubenswrapper[5060]: E1031 13:10:59.921012    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7phb" containerName="extractor"
Oct 31 13:11:01.295573 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:11:01.295538    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9dc80aa2-2l2l6"]
Oct 31 13:11:01.299307 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:11:01.299284    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dc80aa2-2l2l6"
Oct 31 13:11:01.315974 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:11:01.315947    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 13:11:01.316117 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:11:01.316098    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7wsmb\""
Oct 31 13:11:01.316260 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:11:01.316243    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 13:11:01.316811 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:11:01.316793    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 13:11:01.329080 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:11:01.329052    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9dc80aa2-2l2l6"]
Oct 31 13:11:01.425612 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:11:01.425578    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2vhx\" (UniqueName: \"kubernetes.io/projected/baa81c5d-59fb-4d8d-a3f7-4b253b9f6402-kube-api-access-w2vhx\") pod \"maas-api-revoke-sub-9dc80aa2-2l2l6\" (UID: \"baa81c5d-59fb-4d8d-a3f7-4b253b9f6402\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dc80aa2-2l2l6"
Oct 31 13:11:01.425796 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:11:01.425624    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/baa81c5d-59fb-4d8d-a3f7-4b253b9f6402-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9dc80aa2-2l2l6\" (UID: \"baa81c5d-59fb-4d8d-a3f7-4b253b9f6402\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dc80aa2-2l2l6"
Oct 31 13:11:01.425796 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:11:01.425756    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/baa81c5d-59fb-4d8d-a3f7-4b253b9f6402-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9dc80aa2-2l2l6\" (UID: \"baa81c5d-59fb-4d8d-a3f7-4b253b9f6402\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dc80aa2-2l2l6"
Oct 31 13:11:01.526506 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:11:01.526448    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2vhx\" (UniqueName: \"kubernetes.io/projected/baa81c5d-59fb-4d8d-a3f7-4b253b9f6402-kube-api-access-w2vhx\") pod \"maas-api-revoke-sub-9dc80aa2-2l2l6\" (UID: \"baa81c5d-59fb-4d8d-a3f7-4b253b9f6402\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dc80aa2-2l2l6"
Oct 31 13:11:01.526699 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:11:01.526609    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/baa81c5d-59fb-4d8d-a3f7-4b253b9f6402-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9dc80aa2-2l2l6\" (UID: \"baa81c5d-59fb-4d8d-a3f7-4b253b9f6402\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dc80aa2-2l2l6"
Oct 31 13:11:01.526775 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:11:01.526699    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/baa81c5d-59fb-4d8d-a3f7-4b253b9f6402-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9dc80aa2-2l2l6\" (UID: \"baa81c5d-59fb-4d8d-a3f7-4b253b9f6402\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dc80aa2-2l2l6"
Oct 31 13:11:01.527286 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:11:01.527258    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/baa81c5d-59fb-4d8d-a3f7-4b253b9f6402-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9dc80aa2-2l2l6\" (UID: \"baa81c5d-59fb-4d8d-a3f7-4b253b9f6402\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dc80aa2-2l2l6"
Oct 31 13:11:01.528982 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:11:01.528962    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/baa81c5d-59fb-4d8d-a3f7-4b253b9f6402-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9dc80aa2-2l2l6\" (UID: \"baa81c5d-59fb-4d8d-a3f7-4b253b9f6402\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dc80aa2-2l2l6"
Oct 31 13:11:01.539953 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:11:01.539927    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2vhx\" (UniqueName: \"kubernetes.io/projected/baa81c5d-59fb-4d8d-a3f7-4b253b9f6402-kube-api-access-w2vhx\") pod \"maas-api-revoke-sub-9dc80aa2-2l2l6\" (UID: \"baa81c5d-59fb-4d8d-a3f7-4b253b9f6402\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dc80aa2-2l2l6"
Oct 31 13:11:01.614075 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:11:01.614000    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dc80aa2-2l2l6"
Oct 31 13:11:01.896108 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:11:01.896077    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9dc80aa2-2l2l6"]
Oct 31 13:11:01.898443 ip-10-0-130-190 kubenswrapper[5060]: W1031 13:11:01.898410    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbaa81c5d_59fb_4d8d_a3f7_4b253b9f6402.slice/crio-ab13f85060f57bba13b728cc8be025c16e5839faf3e73b67048e8b2c406ac5a3 WatchSource:0}: Error finding container ab13f85060f57bba13b728cc8be025c16e5839faf3e73b67048e8b2c406ac5a3: Status 404 returned error can't find the container with id ab13f85060f57bba13b728cc8be025c16e5839faf3e73b67048e8b2c406ac5a3
Oct 31 13:11:01.966785 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:11:01.966747    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dc80aa2-2l2l6" event={"ID":"baa81c5d-59fb-4d8d-a3f7-4b253b9f6402","Type":"ContainerStarted","Data":"ab13f85060f57bba13b728cc8be025c16e5839faf3e73b67048e8b2c406ac5a3"}
Oct 31 13:11:02.976103 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:11:02.976065    5060 generic.go:356] "Generic (PLEG): container finished" podID="baa81c5d-59fb-4d8d-a3f7-4b253b9f6402" containerID="ced879045f08d62566df46059a4388bed04e6bcd7ac8c3aa785ae615ff8413cf" exitCode=0
Oct 31 13:11:02.976627 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:11:02.976160    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dc80aa2-2l2l6" event={"ID":"baa81c5d-59fb-4d8d-a3f7-4b253b9f6402","Type":"ContainerDied","Data":"ced879045f08d62566df46059a4388bed04e6bcd7ac8c3aa785ae615ff8413cf"}
Oct 31 13:11:04.132630 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:11:04.132601    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dc80aa2-2l2l6"
Oct 31 13:11:04.256712 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:11:04.256622    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/baa81c5d-59fb-4d8d-a3f7-4b253b9f6402-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/baa81c5d-59fb-4d8d-a3f7-4b253b9f6402-maas-api-openshift-ca\") pod \"baa81c5d-59fb-4d8d-a3f7-4b253b9f6402\" (UID: \"baa81c5d-59fb-4d8d-a3f7-4b253b9f6402\") "
Oct 31 13:11:04.256894 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:11:04.256800    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/baa81c5d-59fb-4d8d-a3f7-4b253b9f6402-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/baa81c5d-59fb-4d8d-a3f7-4b253b9f6402-maas-api-xks-ca\") pod \"baa81c5d-59fb-4d8d-a3f7-4b253b9f6402\" (UID: \"baa81c5d-59fb-4d8d-a3f7-4b253b9f6402\") "
Oct 31 13:11:04.256971 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:11:04.256952    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/baa81c5d-59fb-4d8d-a3f7-4b253b9f6402-maas-api-openshift-ca" pod "baa81c5d-59fb-4d8d-a3f7-4b253b9f6402" (UID: "baa81c5d-59fb-4d8d-a3f7-4b253b9f6402"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:11:04.257033 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:11:04.257011    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/baa81c5d-59fb-4d8d-a3f7-4b253b9f6402-kube-api-access-w2vhx\" (UniqueName: \"kubernetes.io/projected/baa81c5d-59fb-4d8d-a3f7-4b253b9f6402-kube-api-access-w2vhx\") pod \"baa81c5d-59fb-4d8d-a3f7-4b253b9f6402\" (UID: \"baa81c5d-59fb-4d8d-a3f7-4b253b9f6402\") "
Oct 31 13:11:04.257326 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:11:04.257303    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/baa81c5d-59fb-4d8d-a3f7-4b253b9f6402-maas-api-openshift-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:11:04.259023 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:11:04.258990    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/baa81c5d-59fb-4d8d-a3f7-4b253b9f6402-kube-api-access-w2vhx" pod "baa81c5d-59fb-4d8d-a3f7-4b253b9f6402" (UID: "baa81c5d-59fb-4d8d-a3f7-4b253b9f6402"). InnerVolumeSpecName "kube-api-access-w2vhx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:11:04.259134 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:11:04.259052    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/baa81c5d-59fb-4d8d-a3f7-4b253b9f6402-maas-api-xks-ca" pod "baa81c5d-59fb-4d8d-a3f7-4b253b9f6402" (UID: "baa81c5d-59fb-4d8d-a3f7-4b253b9f6402"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:11:04.358109 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:11:04.358070    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2vhx\" (UniqueName: \"kubernetes.io/projected/baa81c5d-59fb-4d8d-a3f7-4b253b9f6402-kube-api-access-w2vhx\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:11:04.358109 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:11:04.358109    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/baa81c5d-59fb-4d8d-a3f7-4b253b9f6402-maas-api-xks-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:11:04.992230 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:11:04.992196    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dc80aa2-2l2l6" event={"ID":"baa81c5d-59fb-4d8d-a3f7-4b253b9f6402","Type":"ContainerDied","Data":"ab13f85060f57bba13b728cc8be025c16e5839faf3e73b67048e8b2c406ac5a3"}
Oct 31 13:11:04.992411 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:11:04.992251    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ab13f85060f57bba13b728cc8be025c16e5839faf3e73b67048e8b2c406ac5a3"
Oct 31 13:11:04.992411 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:11:04.992211    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dc80aa2-2l2l6"
Oct 31 13:11:12.090403 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:11:12.090366    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-169e4e-q96np"]
Oct 31 13:11:12.091102 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:11:12.091079    5060 cpu_manager.go:414] "containerMap: removing container" podUID="baa81c5d-59fb-4d8d-a3f7-4b253b9f6402" containerName="revoke-keys"
Oct 31 13:11:12.091203 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:11:12.091103    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="baa81c5d-59fb-4d8d-a3f7-4b253b9f6402" containerName="revoke-keys"
Oct 31 13:11:12.091280 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:11:12.091228    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="baa81c5d-59fb-4d8d-a3f7-4b253b9f6402" containerName="revoke-keys"
Oct 31 13:11:12.095172 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:11:12.095148    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-169e4e-q96np"
Oct 31 13:11:12.104957 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:11:12.104931    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 13:11:12.105154 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:11:12.104977    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 13:11:12.105287 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:11:12.105097    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 13:11:12.105376 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:11:12.105167    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7wsmb\""
Oct 31 13:11:12.105794 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:11:12.105771    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-169e4e-q96np"]
Oct 31 13:11:12.237733 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:11:12.237697    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acacb435-0488-4ea8-941f-aa1b51851ae1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-169e4e-q96np\" (UID: \"acacb435-0488-4ea8-941f-aa1b51851ae1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-169e4e-q96np"
Oct 31 13:11:12.237733 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:11:12.237733    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnxxl\" (UniqueName: \"kubernetes.io/projected/acacb435-0488-4ea8-941f-aa1b51851ae1-kube-api-access-vnxxl\") pod \"maas-api-revoke-keys-e2e-worker-single-main-169e4e-q96np\" (UID: \"acacb435-0488-4ea8-941f-aa1b51851ae1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-169e4e-q96np"
Oct 31 13:11:12.237955 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:11:12.237825    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acacb435-0488-4ea8-941f-aa1b51851ae1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-169e4e-q96np\" (UID: \"acacb435-0488-4ea8-941f-aa1b51851ae1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-169e4e-q96np"
Oct 31 13:11:12.338731 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:11:12.338698    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acacb435-0488-4ea8-941f-aa1b51851ae1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-169e4e-q96np\" (UID: \"acacb435-0488-4ea8-941f-aa1b51851ae1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-169e4e-q96np"
Oct 31 13:11:12.338944 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:11:12.338748    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnxxl\" (UniqueName: \"kubernetes.io/projected/acacb435-0488-4ea8-941f-aa1b51851ae1-kube-api-access-vnxxl\") pod \"maas-api-revoke-keys-e2e-worker-single-main-169e4e-q96np\" (UID: \"acacb435-0488-4ea8-941f-aa1b51851ae1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-169e4e-q96np"
Oct 31 13:11:12.338944 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:11:12.338806    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acacb435-0488-4ea8-941f-aa1b51851ae1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-169e4e-q96np\" (UID: \"acacb435-0488-4ea8-941f-aa1b51851ae1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-169e4e-q96np"
Oct 31 13:11:12.339472 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:11:12.339443    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acacb435-0488-4ea8-941f-aa1b51851ae1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-169e4e-q96np\" (UID: \"acacb435-0488-4ea8-941f-aa1b51851ae1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-169e4e-q96np"
Oct 31 13:11:12.341330 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:11:12.341265    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acacb435-0488-4ea8-941f-aa1b51851ae1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-169e4e-q96np\" (UID: \"acacb435-0488-4ea8-941f-aa1b51851ae1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-169e4e-q96np"
Oct 31 13:11:12.351641 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:11:12.351616    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnxxl\" (UniqueName: \"kubernetes.io/projected/acacb435-0488-4ea8-941f-aa1b51851ae1-kube-api-access-vnxxl\") pod \"maas-api-revoke-keys-e2e-worker-single-main-169e4e-q96np\" (UID: \"acacb435-0488-4ea8-941f-aa1b51851ae1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-169e4e-q96np"
Oct 31 13:11:12.420357 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:11:12.420322    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-169e4e-q96np"
Oct 31 13:11:12.711834 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:11:12.711809    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-169e4e-q96np"]
Oct 31 13:11:12.714254 ip-10-0-130-190 kubenswrapper[5060]: W1031 13:11:12.714213    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacacb435_0488_4ea8_941f_aa1b51851ae1.slice/crio-1903ada914e2c5e43f220f5be5b77ea7238ecf978945042c44809b1d34a75455 WatchSource:0}: Error finding container 1903ada914e2c5e43f220f5be5b77ea7238ecf978945042c44809b1d34a75455: Status 404 returned error can't find the container with id 1903ada914e2c5e43f220f5be5b77ea7238ecf978945042c44809b1d34a75455
Oct 31 13:11:13.048592 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:11:13.048541    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-169e4e-q96np" event={"ID":"acacb435-0488-4ea8-941f-aa1b51851ae1","Type":"ContainerStarted","Data":"1903ada914e2c5e43f220f5be5b77ea7238ecf978945042c44809b1d34a75455"}
Oct 31 13:11:14.053353 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:11:14.053315    5060 generic.go:356] "Generic (PLEG): container finished" podID="acacb435-0488-4ea8-941f-aa1b51851ae1" containerID="65e4780061b7669ad7ab84ae704bb8624170580daf7e74f419396dc46bb4a6b0" exitCode=0
Oct 31 13:11:14.053789 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:11:14.053363    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-169e4e-q96np" event={"ID":"acacb435-0488-4ea8-941f-aa1b51851ae1","Type":"ContainerDied","Data":"65e4780061b7669ad7ab84ae704bb8624170580daf7e74f419396dc46bb4a6b0"}
Oct 31 13:11:15.212304 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:11:15.212270    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-169e4e-q96np"
Oct 31 13:11:15.372177 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:11:15.372098    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/acacb435-0488-4ea8-941f-aa1b51851ae1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acacb435-0488-4ea8-941f-aa1b51851ae1-maas-api-openshift-ca\") pod \"acacb435-0488-4ea8-941f-aa1b51851ae1\" (UID: \"acacb435-0488-4ea8-941f-aa1b51851ae1\") "
Oct 31 13:11:15.372360 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:11:15.372190    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/acacb435-0488-4ea8-941f-aa1b51851ae1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acacb435-0488-4ea8-941f-aa1b51851ae1-maas-api-xks-ca\") pod \"acacb435-0488-4ea8-941f-aa1b51851ae1\" (UID: \"acacb435-0488-4ea8-941f-aa1b51851ae1\") "
Oct 31 13:11:15.372360 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:11:15.372229    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/acacb435-0488-4ea8-941f-aa1b51851ae1-kube-api-access-vnxxl\" (UniqueName: \"kubernetes.io/projected/acacb435-0488-4ea8-941f-aa1b51851ae1-kube-api-access-vnxxl\") pod \"acacb435-0488-4ea8-941f-aa1b51851ae1\" (UID: \"acacb435-0488-4ea8-941f-aa1b51851ae1\") "
Oct 31 13:11:15.372561 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:11:15.372528    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acacb435-0488-4ea8-941f-aa1b51851ae1-maas-api-openshift-ca" pod "acacb435-0488-4ea8-941f-aa1b51851ae1" (UID: "acacb435-0488-4ea8-941f-aa1b51851ae1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:11:15.374588 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:11:15.374552    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acacb435-0488-4ea8-941f-aa1b51851ae1-kube-api-access-vnxxl" pod "acacb435-0488-4ea8-941f-aa1b51851ae1" (UID: "acacb435-0488-4ea8-941f-aa1b51851ae1"). InnerVolumeSpecName "kube-api-access-vnxxl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:11:15.374834 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:11:15.374808    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/acacb435-0488-4ea8-941f-aa1b51851ae1-maas-api-xks-ca" pod "acacb435-0488-4ea8-941f-aa1b51851ae1" (UID: "acacb435-0488-4ea8-941f-aa1b51851ae1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:11:15.473926 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:11:15.473814    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acacb435-0488-4ea8-941f-aa1b51851ae1-maas-api-xks-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:11:15.473926 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:11:15.473846    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vnxxl\" (UniqueName: \"kubernetes.io/projected/acacb435-0488-4ea8-941f-aa1b51851ae1-kube-api-access-vnxxl\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:11:15.473926 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:11:15.473858    5060 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acacb435-0488-4ea8-941f-aa1b51851ae1-maas-api-openshift-ca\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:11:16.069569 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:11:16.069533    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-169e4e-q96np"
Oct 31 13:11:16.069569 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:11:16.069559    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-169e4e-q96np" event={"ID":"acacb435-0488-4ea8-941f-aa1b51851ae1","Type":"ContainerDied","Data":"1903ada914e2c5e43f220f5be5b77ea7238ecf978945042c44809b1d34a75455"}
Oct 31 13:11:16.069811 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:11:16.069617    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1903ada914e2c5e43f220f5be5b77ea7238ecf978945042c44809b1d34a75455"
Oct 31 13:11:23.441017 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:11:23.440981    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-169e4e-q96np"]
Oct 31 13:11:23.444256 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:11:23.444227    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-169e4e-q96np"]
Oct 31 13:11:25.049427 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:11:25.049391    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="acacb435-0488-4ea8-941f-aa1b51851ae1" path="/var/lib/kubelet/pods/acacb435-0488-4ea8-941f-aa1b51851ae1/volumes"
Oct 31 13:11:27.446633 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:11:27.446598    5060 scope.go:122] "RemoveContainer" containerID="e8a6c278e8b9093e9137aadf967a65573d9dcd1d3ef6a2e0454e92190bae90ae"
Oct 31 13:11:27.460136 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:11:27.460106    5060 scope.go:122] "RemoveContainer" containerID="01e291317dd4316741f7239d731d1de93b16ca4b213ad504bbe23527591ddf47"
Oct 31 13:11:27.469174 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:11:27.469148    5060 scope.go:122] "RemoveContainer" containerID="55eddffb550edecbbce333e70d29c3bffc8286b7d71dff70926cb026b360d695"
Oct 31 13:11:27.477552 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:11:27.477531    5060 scope.go:122] "RemoveContainer" containerID="91ad1262bda92289311478b2bb7df1101228d4c08df17ca287961480df79bed5"
Oct 31 13:11:29.928939 ip-10-0-130-190 kubenswrapper[5060]: E1031 13:11:29.928857    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7phb" containerName="extractor"
Oct 31 13:11:59.923797 ip-10-0-130-190 kubenswrapper[5060]: E1031 13:11:59.923755    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7phb" containerName="extractor"
Oct 31 13:12:12.244671 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:12:12.244618    5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-645cf464d7-mwc4r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.14:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 13:12:12.245204 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:12:12.244695    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-mwc4r" podUID="51fa30eb-9214-4402-b911-28d4a2ff4a5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.14:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:12:18.012745 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:12:18.012644    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pcvvm_f032909d-bc39-4c9e-99b7-f3d800842119/console-operator/2.log"
Oct 31 13:12:18.018865 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:12:18.018842    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pcvvm_f032909d-bc39-4c9e-99b7-f3d800842119/console-operator/2.log"
Oct 31 13:12:27.228520 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:12:27.227485    5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-645cf464d7-rzth9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.13:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 13:12:27.228520 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:12:27.227768    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-rzth9" podUID="1d2738c9-9abf-47cf-a6d8-b83789055353" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.13:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:12:27.269514 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:12:27.268601    5060 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-645cf464d7-mwc4r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.14:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 13:12:27.269514 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:12:27.268665    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-mwc4r" podUID="51fa30eb-9214-4402-b911-28d4a2ff4a5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.14:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:12:28.024782 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:12:28.024721    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5296c54-64sf7"]
Oct 31 13:12:28.027789 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:12:28.027760    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5296c54-64sf7"]
Oct 31 13:12:29.047465 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:12:29.047430    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7d27637-37d0-4447-aa4f-7fb34a01adf1" path="/var/lib/kubelet/pods/d7d27637-37d0-4447-aa4f-7fb34a01adf1/volumes"
Oct 31 13:12:29.920539 ip-10-0-130-190 kubenswrapper[5060]: E1031 13:12:29.920472    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7phb" containerName="extractor"
Oct 31 13:12:59.920515 ip-10-0-130-190 kubenswrapper[5060]: E1031 13:12:59.920419    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7phb" containerName="extractor"
Oct 31 13:13:04.517231 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:13:04.517195    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-bbf78971-openshift-default-787bb4945c-8tj7h"]
Oct 31 13:13:04.517704 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:13:04.517688    5060 cpu_manager.go:414] "containerMap: removing container" podUID="acacb435-0488-4ea8-941f-aa1b51851ae1" containerName="revoke-keys"
Oct 31 13:13:04.517751 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:13:04.517706    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="acacb435-0488-4ea8-941f-aa1b51851ae1" containerName="revoke-keys"
Oct 31 13:13:04.517816 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:13:04.517806    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="acacb435-0488-4ea8-941f-aa1b51851ae1" containerName="revoke-keys"
Oct 31 13:13:04.521136 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:13:04.521115    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-bbf78971-openshift-default-787bb4945c-8tj7h"
Oct 31 13:13:04.528867 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:13:04.528843    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-bbf78971-openshift-default-dockercfg-qxxw4\""
Oct 31 13:13:04.547505 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:13:04.547465    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-bbf78971-openshift-default-787bb4945c-8tj7h"]
Oct 31 13:13:04.645754 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:13:04.645714    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e00a8c30-1ae5-4e96-8e53-6b13c670ae1d-istio-podinfo\") pod \"e2e-ait-bbf78971-openshift-default-787bb4945c-8tj7h\" (UID: \"e00a8c30-1ae5-4e96-8e53-6b13c670ae1d\") " pod="openshift-ingress/e2e-ait-bbf78971-openshift-default-787bb4945c-8tj7h"
Oct 31 13:13:04.645972 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:13:04.645780    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e00a8c30-1ae5-4e96-8e53-6b13c670ae1d-istio-token\") pod \"e2e-ait-bbf78971-openshift-default-787bb4945c-8tj7h\" (UID: \"e00a8c30-1ae5-4e96-8e53-6b13c670ae1d\") " pod="openshift-ingress/e2e-ait-bbf78971-openshift-default-787bb4945c-8tj7h"
Oct 31 13:13:04.645972 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:13:04.645823    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e00a8c30-1ae5-4e96-8e53-6b13c670ae1d-istio-envoy\") pod \"e2e-ait-bbf78971-openshift-default-787bb4945c-8tj7h\" (UID: \"e00a8c30-1ae5-4e96-8e53-6b13c670ae1d\") " pod="openshift-ingress/e2e-ait-bbf78971-openshift-default-787bb4945c-8tj7h"
Oct 31 13:13:04.645972 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:13:04.645867    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scmm5\" (UniqueName: \"kubernetes.io/projected/e00a8c30-1ae5-4e96-8e53-6b13c670ae1d-kube-api-access-scmm5\") pod \"e2e-ait-bbf78971-openshift-default-787bb4945c-8tj7h\" (UID: \"e00a8c30-1ae5-4e96-8e53-6b13c670ae1d\") " pod="openshift-ingress/e2e-ait-bbf78971-openshift-default-787bb4945c-8tj7h"
Oct 31 13:13:04.645972 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:13:04.645909    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e00a8c30-1ae5-4e96-8e53-6b13c670ae1d-workload-socket\") pod \"e2e-ait-bbf78971-openshift-default-787bb4945c-8tj7h\" (UID: \"e00a8c30-1ae5-4e96-8e53-6b13c670ae1d\") " pod="openshift-ingress/e2e-ait-bbf78971-openshift-default-787bb4945c-8tj7h"
Oct 31 13:13:04.646216 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:13:04.645978    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e00a8c30-1ae5-4e96-8e53-6b13c670ae1d-credential-socket\") pod \"e2e-ait-bbf78971-openshift-default-787bb4945c-8tj7h\" (UID: \"e00a8c30-1ae5-4e96-8e53-6b13c670ae1d\") " pod="openshift-ingress/e2e-ait-bbf78971-openshift-default-787bb4945c-8tj7h"
Oct 31 13:13:04.646216 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:13:04.646024    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e00a8c30-1ae5-4e96-8e53-6b13c670ae1d-istiod-ca-cert\") pod \"e2e-ait-bbf78971-openshift-default-787bb4945c-8tj7h\" (UID: \"e00a8c30-1ae5-4e96-8e53-6b13c670ae1d\") " pod="openshift-ingress/e2e-ait-bbf78971-openshift-default-787bb4945c-8tj7h"
Oct 31 13:13:04.646216 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:13:04.646050    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e00a8c30-1ae5-4e96-8e53-6b13c670ae1d-istio-data\") pod \"e2e-ait-bbf78971-openshift-default-787bb4945c-8tj7h\" (UID: \"e00a8c30-1ae5-4e96-8e53-6b13c670ae1d\") " pod="openshift-ingress/e2e-ait-bbf78971-openshift-default-787bb4945c-8tj7h"
Oct 31 13:13:04.646216 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:13:04.646078    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e00a8c30-1ae5-4e96-8e53-6b13c670ae1d-workload-certs\") pod \"e2e-ait-bbf78971-openshift-default-787bb4945c-8tj7h\" (UID: \"e00a8c30-1ae5-4e96-8e53-6b13c670ae1d\") " pod="openshift-ingress/e2e-ait-bbf78971-openshift-default-787bb4945c-8tj7h"
Oct 31 13:13:04.747445 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:13:04.747402    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e00a8c30-1ae5-4e96-8e53-6b13c670ae1d-credential-socket\") pod \"e2e-ait-bbf78971-openshift-default-787bb4945c-8tj7h\" (UID: \"e00a8c30-1ae5-4e96-8e53-6b13c670ae1d\") " pod="openshift-ingress/e2e-ait-bbf78971-openshift-default-787bb4945c-8tj7h"
Oct 31 13:13:04.747445 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:13:04.747452    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e00a8c30-1ae5-4e96-8e53-6b13c670ae1d-istiod-ca-cert\") pod \"e2e-ait-bbf78971-openshift-default-787bb4945c-8tj7h\" (UID: \"e00a8c30-1ae5-4e96-8e53-6b13c670ae1d\") " pod="openshift-ingress/e2e-ait-bbf78971-openshift-default-787bb4945c-8tj7h"
Oct 31 13:13:04.747748 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:13:04.747540    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e00a8c30-1ae5-4e96-8e53-6b13c670ae1d-istio-data\") pod \"e2e-ait-bbf78971-openshift-default-787bb4945c-8tj7h\" (UID: \"e00a8c30-1ae5-4e96-8e53-6b13c670ae1d\") " pod="openshift-ingress/e2e-ait-bbf78971-openshift-default-787bb4945c-8tj7h"
Oct 31 13:13:04.747748 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:13:04.747679    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e00a8c30-1ae5-4e96-8e53-6b13c670ae1d-workload-certs\") pod \"e2e-ait-bbf78971-openshift-default-787bb4945c-8tj7h\" (UID: \"e00a8c30-1ae5-4e96-8e53-6b13c670ae1d\") " pod="openshift-ingress/e2e-ait-bbf78971-openshift-default-787bb4945c-8tj7h"
Oct 31 13:13:04.747748 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:13:04.747731    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e00a8c30-1ae5-4e96-8e53-6b13c670ae1d-istio-podinfo\") pod \"e2e-ait-bbf78971-openshift-default-787bb4945c-8tj7h\" (UID: \"e00a8c30-1ae5-4e96-8e53-6b13c670ae1d\") " pod="openshift-ingress/e2e-ait-bbf78971-openshift-default-787bb4945c-8tj7h"
Oct 31 13:13:04.747903 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:13:04.747782    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e00a8c30-1ae5-4e96-8e53-6b13c670ae1d-istio-token\") pod \"e2e-ait-bbf78971-openshift-default-787bb4945c-8tj7h\" (UID: \"e00a8c30-1ae5-4e96-8e53-6b13c670ae1d\") " pod="openshift-ingress/e2e-ait-bbf78971-openshift-default-787bb4945c-8tj7h"
Oct 31 13:13:04.747903 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:13:04.747826    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e00a8c30-1ae5-4e96-8e53-6b13c670ae1d-istio-envoy\") pod \"e2e-ait-bbf78971-openshift-default-787bb4945c-8tj7h\" (UID: \"e00a8c30-1ae5-4e96-8e53-6b13c670ae1d\") " pod="openshift-ingress/e2e-ait-bbf78971-openshift-default-787bb4945c-8tj7h"
Oct 31 13:13:04.747903 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:13:04.747882    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scmm5\" (UniqueName: \"kubernetes.io/projected/e00a8c30-1ae5-4e96-8e53-6b13c670ae1d-kube-api-access-scmm5\") pod \"e2e-ait-bbf78971-openshift-default-787bb4945c-8tj7h\" (UID: \"e00a8c30-1ae5-4e96-8e53-6b13c670ae1d\") " pod="openshift-ingress/e2e-ait-bbf78971-openshift-default-787bb4945c-8tj7h"
Oct 31 13:13:04.748083 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:13:04.747928    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e00a8c30-1ae5-4e96-8e53-6b13c670ae1d-workload-socket\") pod \"e2e-ait-bbf78971-openshift-default-787bb4945c-8tj7h\" (UID: \"e00a8c30-1ae5-4e96-8e53-6b13c670ae1d\") " pod="openshift-ingress/e2e-ait-bbf78971-openshift-default-787bb4945c-8tj7h"
Oct 31 13:13:04.748083 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:13:04.747974    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e00a8c30-1ae5-4e96-8e53-6b13c670ae1d-istio-data\") pod \"e2e-ait-bbf78971-openshift-default-787bb4945c-8tj7h\" (UID: \"e00a8c30-1ae5-4e96-8e53-6b13c670ae1d\") " pod="openshift-ingress/e2e-ait-bbf78971-openshift-default-787bb4945c-8tj7h"
Oct 31 13:13:04.748083 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:13:04.748001    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e00a8c30-1ae5-4e96-8e53-6b13c670ae1d-credential-socket\") pod \"e2e-ait-bbf78971-openshift-default-787bb4945c-8tj7h\" (UID: \"e00a8c30-1ae5-4e96-8e53-6b13c670ae1d\") " pod="openshift-ingress/e2e-ait-bbf78971-openshift-default-787bb4945c-8tj7h"
Oct 31 13:13:04.748237 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:13:04.748104    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e00a8c30-1ae5-4e96-8e53-6b13c670ae1d-workload-certs\") pod \"e2e-ait-bbf78971-openshift-default-787bb4945c-8tj7h\" (UID: \"e00a8c30-1ae5-4e96-8e53-6b13c670ae1d\") " pod="openshift-ingress/e2e-ait-bbf78971-openshift-default-787bb4945c-8tj7h"
Oct 31 13:13:04.748294 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:13:04.748274    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e00a8c30-1ae5-4e96-8e53-6b13c670ae1d-istiod-ca-cert\") pod \"e2e-ait-bbf78971-openshift-default-787bb4945c-8tj7h\" (UID: \"e00a8c30-1ae5-4e96-8e53-6b13c670ae1d\") " pod="openshift-ingress/e2e-ait-bbf78971-openshift-default-787bb4945c-8tj7h"
Oct 31 13:13:04.748361 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:13:04.748343    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e00a8c30-1ae5-4e96-8e53-6b13c670ae1d-workload-socket\") pod \"e2e-ait-bbf78971-openshift-default-787bb4945c-8tj7h\" (UID: \"e00a8c30-1ae5-4e96-8e53-6b13c670ae1d\") " pod="openshift-ingress/e2e-ait-bbf78971-openshift-default-787bb4945c-8tj7h"
Oct 31 13:13:04.750183 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:13:04.750159    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e00a8c30-1ae5-4e96-8e53-6b13c670ae1d-istio-envoy\") pod \"e2e-ait-bbf78971-openshift-default-787bb4945c-8tj7h\" (UID: \"e00a8c30-1ae5-4e96-8e53-6b13c670ae1d\") " pod="openshift-ingress/e2e-ait-bbf78971-openshift-default-787bb4945c-8tj7h"
Oct 31 13:13:04.750429 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:13:04.750412    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e00a8c30-1ae5-4e96-8e53-6b13c670ae1d-istio-podinfo\") pod \"e2e-ait-bbf78971-openshift-default-787bb4945c-8tj7h\" (UID: \"e00a8c30-1ae5-4e96-8e53-6b13c670ae1d\") " pod="openshift-ingress/e2e-ait-bbf78971-openshift-default-787bb4945c-8tj7h"
Oct 31 13:13:04.761287 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:13:04.761255    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e00a8c30-1ae5-4e96-8e53-6b13c670ae1d-istio-token\") pod \"e2e-ait-bbf78971-openshift-default-787bb4945c-8tj7h\" (UID: \"e00a8c30-1ae5-4e96-8e53-6b13c670ae1d\") " pod="openshift-ingress/e2e-ait-bbf78971-openshift-default-787bb4945c-8tj7h"
Oct 31 13:13:04.762265 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:13:04.762235    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-scmm5\" (UniqueName: \"kubernetes.io/projected/e00a8c30-1ae5-4e96-8e53-6b13c670ae1d-kube-api-access-scmm5\") pod \"e2e-ait-bbf78971-openshift-default-787bb4945c-8tj7h\" (UID: \"e00a8c30-1ae5-4e96-8e53-6b13c670ae1d\") " pod="openshift-ingress/e2e-ait-bbf78971-openshift-default-787bb4945c-8tj7h"
Oct 31 13:13:04.835163 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:13:04.835129    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-bbf78971-openshift-default-787bb4945c-8tj7h"
Oct 31 13:13:05.132987 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:13:05.132950    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-bbf78971-openshift-default-787bb4945c-8tj7h"]
Oct 31 13:13:05.134563 ip-10-0-130-190 kubenswrapper[5060]: W1031 13:13:05.134538    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode00a8c30_1ae5_4e96_8e53_6b13c670ae1d.slice/crio-b3ff26440aa227e56c9033fe4ba1fa7a009e4908b8c639a1c6d4498ad67934bf WatchSource:0}: Error finding container b3ff26440aa227e56c9033fe4ba1fa7a009e4908b8c639a1c6d4498ad67934bf: Status 404 returned error can't find the container with id b3ff26440aa227e56c9033fe4ba1fa7a009e4908b8c639a1c6d4498ad67934bf
Oct 31 13:13:05.137086 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:13:05.137053    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 13:13:05.137174 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:13:05.137125    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 13:13:05.137174 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:13:05.137154    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 13:13:05.387548 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:13:05.387447    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-7c54444cdb-qxjzm"]
Oct 31 13:13:05.392154 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:13:05.392132    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7c54444cdb-qxjzm"
Oct 31 13:13:05.406698 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:13:05.406655    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 13:13:05.406880 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:13:05.406764    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 13:13:05.413345 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:13:05.412754    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Oct 31 13:13:05.413345 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:13:05.412876    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 31 13:13:05.413345 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:13:05.412986    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-6jkzx\""
Oct 31 13:13:05.414439 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:13:05.414410    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7c54444cdb-qxjzm"]
Oct 31 13:13:05.559205 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:13:05.559171    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d8b24af-3e72-495d-8c18-f0427d81e9a2-metrics-certs\") pod \"maas-api-7c54444cdb-qxjzm\" (UID: \"6d8b24af-3e72-495d-8c18-f0427d81e9a2\") " pod="odh-ai-gateway-infra/maas-api-7c54444cdb-qxjzm"
Oct 31 13:13:05.559714 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:13:05.559232    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d8b24af-3e72-495d-8c18-f0427d81e9a2-openshift-service-ca\") pod \"maas-api-7c54444cdb-qxjzm\" (UID: \"6d8b24af-3e72-495d-8c18-f0427d81e9a2\") " pod="odh-ai-gateway-infra/maas-api-7c54444cdb-qxjzm"
Oct 31 13:13:05.559714 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:13:05.559313    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5whj\" (UniqueName: \"kubernetes.io/projected/6d8b24af-3e72-495d-8c18-f0427d81e9a2-kube-api-access-p5whj\") pod \"maas-api-7c54444cdb-qxjzm\" (UID: \"6d8b24af-3e72-495d-8c18-f0427d81e9a2\") " pod="odh-ai-gateway-infra/maas-api-7c54444cdb-qxjzm"
Oct 31 13:13:05.559714 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:13:05.559346    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d8b24af-3e72-495d-8c18-f0427d81e9a2-maas-api-tls\") pod \"maas-api-7c54444cdb-qxjzm\" (UID: \"6d8b24af-3e72-495d-8c18-f0427d81e9a2\") " pod="odh-ai-gateway-infra/maas-api-7c54444cdb-qxjzm"
Oct 31 13:13:05.660938 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:13:05.660835    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d8b24af-3e72-495d-8c18-f0427d81e9a2-metrics-certs\") pod \"maas-api-7c54444cdb-qxjzm\" (UID: \"6d8b24af-3e72-495d-8c18-f0427d81e9a2\") " pod="odh-ai-gateway-infra/maas-api-7c54444cdb-qxjzm"
Oct 31 13:13:05.660938 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:13:05.660931    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d8b24af-3e72-495d-8c18-f0427d81e9a2-openshift-service-ca\") pod \"maas-api-7c54444cdb-qxjzm\" (UID: \"6d8b24af-3e72-495d-8c18-f0427d81e9a2\") " pod="odh-ai-gateway-infra/maas-api-7c54444cdb-qxjzm"
Oct 31 13:13:05.661190 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:13:05.660962    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5whj\" (UniqueName: \"kubernetes.io/projected/6d8b24af-3e72-495d-8c18-f0427d81e9a2-kube-api-access-p5whj\") pod \"maas-api-7c54444cdb-qxjzm\" (UID: \"6d8b24af-3e72-495d-8c18-f0427d81e9a2\") " pod="odh-ai-gateway-infra/maas-api-7c54444cdb-qxjzm"
Oct 31 13:13:05.661190 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:13:05.660993    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d8b24af-3e72-495d-8c18-f0427d81e9a2-maas-api-tls\") pod \"maas-api-7c54444cdb-qxjzm\" (UID: \"6d8b24af-3e72-495d-8c18-f0427d81e9a2\") " pod="odh-ai-gateway-infra/maas-api-7c54444cdb-qxjzm"
Oct 31 13:13:05.661803 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:13:05.661771    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d8b24af-3e72-495d-8c18-f0427d81e9a2-openshift-service-ca\") pod \"maas-api-7c54444cdb-qxjzm\" (UID: \"6d8b24af-3e72-495d-8c18-f0427d81e9a2\") " pod="odh-ai-gateway-infra/maas-api-7c54444cdb-qxjzm"
Oct 31 13:13:05.663441 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:13:05.663413    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d8b24af-3e72-495d-8c18-f0427d81e9a2-metrics-certs\") pod \"maas-api-7c54444cdb-qxjzm\" (UID: \"6d8b24af-3e72-495d-8c18-f0427d81e9a2\") " pod="odh-ai-gateway-infra/maas-api-7c54444cdb-qxjzm"
Oct 31 13:13:05.663638 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:13:05.663612    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d8b24af-3e72-495d-8c18-f0427d81e9a2-maas-api-tls\") pod \"maas-api-7c54444cdb-qxjzm\" (UID: \"6d8b24af-3e72-495d-8c18-f0427d81e9a2\") " pod="odh-ai-gateway-infra/maas-api-7c54444cdb-qxjzm"
Oct 31 13:13:05.674195 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:13:05.674162    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5whj\" (UniqueName: \"kubernetes.io/projected/6d8b24af-3e72-495d-8c18-f0427d81e9a2-kube-api-access-p5whj\") pod \"maas-api-7c54444cdb-qxjzm\" (UID: \"6d8b24af-3e72-495d-8c18-f0427d81e9a2\") " pod="odh-ai-gateway-infra/maas-api-7c54444cdb-qxjzm"
Oct 31 13:13:05.718439 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:13:05.718406    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7c54444cdb-qxjzm"
Oct 31 13:13:06.000845 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:13:06.000819    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7c54444cdb-qxjzm"]
Oct 31 13:13:06.002787 ip-10-0-130-190 kubenswrapper[5060]: W1031 13:13:06.002756    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d8b24af_3e72_495d_8c18_f0427d81e9a2.slice/crio-160b146e7f171624dc9a6fa74f2550cc9050c0a29a29e1aea08c75c99fa50850 WatchSource:0}: Error finding container 160b146e7f171624dc9a6fa74f2550cc9050c0a29a29e1aea08c75c99fa50850: Status 404 returned error can't find the container with id 160b146e7f171624dc9a6fa74f2550cc9050c0a29a29e1aea08c75c99fa50850
Oct 31 13:13:06.004665 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:13:06.004644    5060 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 13:13:06.070214 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:13:06.070178    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7c54444cdb-qxjzm" event={"ID":"6d8b24af-3e72-495d-8c18-f0427d81e9a2","Type":"ContainerStarted","Data":"160b146e7f171624dc9a6fa74f2550cc9050c0a29a29e1aea08c75c99fa50850"}
Oct 31 13:13:06.072076 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:13:06.072042    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-bbf78971-openshift-default-787bb4945c-8tj7h" event={"ID":"e00a8c30-1ae5-4e96-8e53-6b13c670ae1d","Type":"ContainerStarted","Data":"7a9824beebec3908f948429c83877116f0580794a5fd1a8cc2f211cb6e647e60"}
Oct 31 13:13:06.072242 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:13:06.072096    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-bbf78971-openshift-default-787bb4945c-8tj7h" event={"ID":"e00a8c30-1ae5-4e96-8e53-6b13c670ae1d","Type":"ContainerStarted","Data":"b3ff26440aa227e56c9033fe4ba1fa7a009e4908b8c639a1c6d4498ad67934bf"}
Oct 31 13:13:06.098057 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:13:06.098005    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-bbf78971-openshift-default-787bb4945c-8tj7h" podStartSLOduration=2.097990853 podStartE2EDuration="2.097990853s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:13:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:13:06.095843775 +0000 UTC m=+2749.198911487" watchObservedRunningTime="2026-10-31 13:13:06.097990853 +0000 UTC m=+2749.201058567"
Oct 31 13:13:06.835576 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:13:06.835536    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-bbf78971-openshift-default-787bb4945c-8tj7h"
Oct 31 13:13:06.840419 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:13:06.840391    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-bbf78971-openshift-default-787bb4945c-8tj7h"
Oct 31 13:13:07.082611 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:13:07.082566    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7c54444cdb-qxjzm" event={"ID":"6d8b24af-3e72-495d-8c18-f0427d81e9a2","Type":"ContainerStarted","Data":"ad320d7811ae95a4d0d485df9d4b40df80758b5ebbbc3655f953d789ccdde7e1"}
Oct 31 13:13:07.082828 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:13:07.082646    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-bbf78971-openshift-default-787bb4945c-8tj7h"
Oct 31 13:13:07.082882 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:13:07.082866    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-7c54444cdb-qxjzm"
Oct 31 13:13:07.083856 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:13:07.083829    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-bbf78971-openshift-default-787bb4945c-8tj7h"
Oct 31 13:13:07.105149 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:13:07.105053    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-7c54444cdb-qxjzm" podStartSLOduration=1.716261948 podStartE2EDuration="2.105035911s" totalImagesPullingTime="388.773963ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:13:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:13:07.104099343 +0000 UTC m=+2750.207167054" watchObservedRunningTime="2026-10-31 13:13:07.105035911 +0000 UTC m=+2750.208103623"
Oct 31 13:13:13.101957 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:13:13.101927    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-7c54444cdb-qxjzm"
Oct 31 13:13:21.175110 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:13:21.175077    5060 generic.go:356] "Generic (PLEG): container finished" podID="e00a8c30-1ae5-4e96-8e53-6b13c670ae1d" containerID="7a9824beebec3908f948429c83877116f0580794a5fd1a8cc2f211cb6e647e60" exitCode=0
Oct 31 13:13:21.175628 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:13:21.175151    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-bbf78971-openshift-default-787bb4945c-8tj7h" event={"ID":"e00a8c30-1ae5-4e96-8e53-6b13c670ae1d","Type":"ContainerDied","Data":"7a9824beebec3908f948429c83877116f0580794a5fd1a8cc2f211cb6e647e60"}
Oct 31 13:13:21.175628 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:13:21.175557    5060 scope.go:122] "RemoveContainer" containerID="7a9824beebec3908f948429c83877116f0580794a5fd1a8cc2f211cb6e647e60"
Oct 31 13:13:21.176912 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:13:21.176874    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 13:13:21.177014 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:13:21.176965    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 13:13:21.177058 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:13:21.177016    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 13:13:22.083135 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:13:22.083102    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-bbf78971-openshift-default-787bb4945c-8tj7h"
Oct 31 13:13:22.187915 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:13:22.187879    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-bbf78971-openshift-default-787bb4945c-8tj7h" event={"ID":"e00a8c30-1ae5-4e96-8e53-6b13c670ae1d","Type":"ContainerStarted","Data":"f9cc30703d30fc521dd547f099118d9bc77f4036c59929ce0ac471b864e6b9a5"}
Oct 31 13:13:22.836023 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:13:22.835987    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-bbf78971-openshift-default-787bb4945c-8tj7h"
Oct 31 13:13:22.837026 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:13:22.837001    5060 patch_prober.go:29] interesting pod/e2e-ait-bbf78971-openshift-default-787bb4945c-8tj7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.32:15021/healthz/ready\": dial tcp 10.132.1.32:15021: connect: connection refused" start-of-body=
Oct 31 13:13:22.837135 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:13:22.837057    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-bbf78971-openshift-default-787bb4945c-8tj7h" podUID="e00a8c30-1ae5-4e96-8e53-6b13c670ae1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.32:15021/healthz/ready\": dial tcp 10.132.1.32:15021: connect: connection refused"
Oct 31 13:13:23.835615 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:13:23.835577    5060 patch_prober.go:29] interesting pod/e2e-ait-bbf78971-openshift-default-787bb4945c-8tj7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.32:15021/healthz/ready\": dial tcp 10.132.1.32:15021: connect: connection refused" start-of-body=
Oct 31 13:13:23.836099 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:13:23.835643    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-bbf78971-openshift-default-787bb4945c-8tj7h" podUID="e00a8c30-1ae5-4e96-8e53-6b13c670ae1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.32:15021/healthz/ready\": dial tcp 10.132.1.32:15021: connect: connection refused"
Oct 31 13:13:24.836256 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:13:24.836215    5060 patch_prober.go:29] interesting pod/e2e-ait-bbf78971-openshift-default-787bb4945c-8tj7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.32:15021/healthz/ready\": dial tcp 10.132.1.32:15021: connect: connection refused" start-of-body=
Oct 31 13:13:24.836692 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:13:24.836288    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-bbf78971-openshift-default-787bb4945c-8tj7h" podUID="e00a8c30-1ae5-4e96-8e53-6b13c670ae1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.32:15021/healthz/ready\": dial tcp 10.132.1.32:15021: connect: connection refused"
Oct 31 13:13:25.835956 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:13:25.835919    5060 patch_prober.go:29] interesting pod/e2e-ait-bbf78971-openshift-default-787bb4945c-8tj7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.32:15021/healthz/ready\": dial tcp 10.132.1.32:15021: connect: connection refused" start-of-body=
Oct 31 13:13:25.836142 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:13:25.835974    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-bbf78971-openshift-default-787bb4945c-8tj7h" podUID="e00a8c30-1ae5-4e96-8e53-6b13c670ae1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.32:15021/healthz/ready\": dial tcp 10.132.1.32:15021: connect: connection refused"
Oct 31 13:13:26.835964 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:13:26.835928    5060 patch_prober.go:29] interesting pod/e2e-ait-bbf78971-openshift-default-787bb4945c-8tj7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.32:15021/healthz/ready\": dial tcp 10.132.1.32:15021: connect: connection refused" start-of-body=
Oct 31 13:13:26.836379 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:13:26.835987    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-bbf78971-openshift-default-787bb4945c-8tj7h" podUID="e00a8c30-1ae5-4e96-8e53-6b13c670ae1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.32:15021/healthz/ready\": dial tcp 10.132.1.32:15021: connect: connection refused"
Oct 31 13:13:27.526252 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:13:27.526220    5060 scope.go:122] "RemoveContainer" containerID="2ae5f85cf2ed9e2df6b3a421749de0ebc3afc04538c3444508a8800d7d7a638a"
Oct 31 13:13:27.839651 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:13:27.839576    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-bbf78971-openshift-default-787bb4945c-8tj7h"
Oct 31 13:13:27.840063 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:13:27.839976    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-bbf78971-openshift-default-787bb4945c-8tj7h"
Oct 31 13:13:27.840630 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:13:27.840612    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-bbf78971-openshift-default-787bb4945c-8tj7h"
Oct 31 13:13:29.922134 ip-10-0-130-190 kubenswrapper[5060]: E1031 13:13:29.922102    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7phb" containerName="extractor"
Oct 31 13:13:39.617390 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:13:39.617336    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-bbf78971-openshift-default-787bb4945c-8tj7h"]
Oct 31 13:13:39.617938 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:13:39.617637    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-bbf78971-openshift-default-787bb4945c-8tj7h" podUID="e00a8c30-1ae5-4e96-8e53-6b13c670ae1d" containerName="istio-proxy" containerID="cri-o://f9cc30703d30fc521dd547f099118d9bc77f4036c59929ce0ac471b864e6b9a5" gracePeriod=30
Oct 31 13:13:42.842676 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:13:42.842638    5060 patch_prober.go:29] interesting pod/e2e-ait-bbf78971-openshift-default-787bb4945c-8tj7h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused
Oct 31 13:13:42.843089 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:13:42.842729    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-bbf78971-openshift-default-787bb4945c-8tj7h" podUID="e00a8c30-1ae5-4e96-8e53-6b13c670ae1d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 13:13:44.875615 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:13:44.875587    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-bbf78971-openshift-default-787bb4945c-8tj7h"
Oct 31 13:13:45.030079 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:13:45.030047    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e00a8c30-1ae5-4e96-8e53-6b13c670ae1d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e00a8c30-1ae5-4e96-8e53-6b13c670ae1d-workload-socket\") pod \"e00a8c30-1ae5-4e96-8e53-6b13c670ae1d\" (UID: \"e00a8c30-1ae5-4e96-8e53-6b13c670ae1d\") "
Oct 31 13:13:45.030272 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:13:45.030108    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e00a8c30-1ae5-4e96-8e53-6b13c670ae1d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e00a8c30-1ae5-4e96-8e53-6b13c670ae1d-istio-podinfo\") pod \"e00a8c30-1ae5-4e96-8e53-6b13c670ae1d\" (UID: \"e00a8c30-1ae5-4e96-8e53-6b13c670ae1d\") "
Oct 31 13:13:45.030272 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:13:45.030136    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e00a8c30-1ae5-4e96-8e53-6b13c670ae1d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e00a8c30-1ae5-4e96-8e53-6b13c670ae1d-workload-certs\") pod \"e00a8c30-1ae5-4e96-8e53-6b13c670ae1d\" (UID: \"e00a8c30-1ae5-4e96-8e53-6b13c670ae1d\") "
Oct 31 13:13:45.030272 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:13:45.030182    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e00a8c30-1ae5-4e96-8e53-6b13c670ae1d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e00a8c30-1ae5-4e96-8e53-6b13c670ae1d-istio-data\") pod \"e00a8c30-1ae5-4e96-8e53-6b13c670ae1d\" (UID: \"e00a8c30-1ae5-4e96-8e53-6b13c670ae1d\") "
Oct 31 13:13:45.030459 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:13:45.030296    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e00a8c30-1ae5-4e96-8e53-6b13c670ae1d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e00a8c30-1ae5-4e96-8e53-6b13c670ae1d-istiod-ca-cert\") pod \"e00a8c30-1ae5-4e96-8e53-6b13c670ae1d\" (UID: \"e00a8c30-1ae5-4e96-8e53-6b13c670ae1d\") "
Oct 31 13:13:45.030459 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:13:45.030375    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e00a8c30-1ae5-4e96-8e53-6b13c670ae1d-workload-socket" pod "e00a8c30-1ae5-4e96-8e53-6b13c670ae1d" (UID: "e00a8c30-1ae5-4e96-8e53-6b13c670ae1d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:13:45.030459 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:13:45.030404    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e00a8c30-1ae5-4e96-8e53-6b13c670ae1d-kube-api-access-scmm5\" (UniqueName: \"kubernetes.io/projected/e00a8c30-1ae5-4e96-8e53-6b13c670ae1d-kube-api-access-scmm5\") pod \"e00a8c30-1ae5-4e96-8e53-6b13c670ae1d\" (UID: \"e00a8c30-1ae5-4e96-8e53-6b13c670ae1d\") "
Oct 31 13:13:45.030459 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:13:45.030435    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e00a8c30-1ae5-4e96-8e53-6b13c670ae1d-workload-certs" pod "e00a8c30-1ae5-4e96-8e53-6b13c670ae1d" (UID: "e00a8c30-1ae5-4e96-8e53-6b13c670ae1d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:13:45.030459 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:13:45.030441    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e00a8c30-1ae5-4e96-8e53-6b13c670ae1d-istio-token\" (UniqueName: \"kubernetes.io/projected/e00a8c30-1ae5-4e96-8e53-6b13c670ae1d-istio-token\") pod \"e00a8c30-1ae5-4e96-8e53-6b13c670ae1d\" (UID: \"e00a8c30-1ae5-4e96-8e53-6b13c670ae1d\") "
Oct 31 13:13:45.030799 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:13:45.030582    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e00a8c30-1ae5-4e96-8e53-6b13c670ae1d-istio-data" pod "e00a8c30-1ae5-4e96-8e53-6b13c670ae1d" (UID: "e00a8c30-1ae5-4e96-8e53-6b13c670ae1d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:13:45.030799 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:13:45.030589    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e00a8c30-1ae5-4e96-8e53-6b13c670ae1d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e00a8c30-1ae5-4e96-8e53-6b13c670ae1d-istio-envoy\") pod \"e00a8c30-1ae5-4e96-8e53-6b13c670ae1d\" (UID: \"e00a8c30-1ae5-4e96-8e53-6b13c670ae1d\") "
Oct 31 13:13:45.030799 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:13:45.030678    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e00a8c30-1ae5-4e96-8e53-6b13c670ae1d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e00a8c30-1ae5-4e96-8e53-6b13c670ae1d-credential-socket\") pod \"e00a8c30-1ae5-4e96-8e53-6b13c670ae1d\" (UID: \"e00a8c30-1ae5-4e96-8e53-6b13c670ae1d\") "
Oct 31 13:13:45.030799 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:13:45.030739    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e00a8c30-1ae5-4e96-8e53-6b13c670ae1d-istiod-ca-cert" pod "e00a8c30-1ae5-4e96-8e53-6b13c670ae1d" (UID: "e00a8c30-1ae5-4e96-8e53-6b13c670ae1d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:13:45.031020 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:13:45.030980    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e00a8c30-1ae5-4e96-8e53-6b13c670ae1d-credential-socket" pod "e00a8c30-1ae5-4e96-8e53-6b13c670ae1d" (UID: "e00a8c30-1ae5-4e96-8e53-6b13c670ae1d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:13:45.031084 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:13:45.031051    5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e00a8c30-1ae5-4e96-8e53-6b13c670ae1d-credential-socket\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:13:45.031084 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:13:45.031070    5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e00a8c30-1ae5-4e96-8e53-6b13c670ae1d-workload-socket\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:13:45.031193 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:13:45.031091    5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e00a8c30-1ae5-4e96-8e53-6b13c670ae1d-workload-certs\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:13:45.031193 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:13:45.031110    5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e00a8c30-1ae5-4e96-8e53-6b13c670ae1d-istio-data\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:13:45.031193 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:13:45.031123    5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e00a8c30-1ae5-4e96-8e53-6b13c670ae1d-istiod-ca-cert\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:13:45.032740 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:13:45.032708    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e00a8c30-1ae5-4e96-8e53-6b13c670ae1d-istio-podinfo" pod "e00a8c30-1ae5-4e96-8e53-6b13c670ae1d" (UID: "e00a8c30-1ae5-4e96-8e53-6b13c670ae1d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 13:13:45.032866 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:13:45.032760    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e00a8c30-1ae5-4e96-8e53-6b13c670ae1d-istio-token" pod "e00a8c30-1ae5-4e96-8e53-6b13c670ae1d" (UID: "e00a8c30-1ae5-4e96-8e53-6b13c670ae1d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:13:45.032866 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:13:45.032857    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e00a8c30-1ae5-4e96-8e53-6b13c670ae1d-istio-envoy" pod "e00a8c30-1ae5-4e96-8e53-6b13c670ae1d" (UID: "e00a8c30-1ae5-4e96-8e53-6b13c670ae1d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:13:45.032991 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:13:45.032902    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e00a8c30-1ae5-4e96-8e53-6b13c670ae1d-kube-api-access-scmm5" pod "e00a8c30-1ae5-4e96-8e53-6b13c670ae1d" (UID: "e00a8c30-1ae5-4e96-8e53-6b13c670ae1d"). InnerVolumeSpecName "kube-api-access-scmm5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:13:45.132237 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:13:45.132205    5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e00a8c30-1ae5-4e96-8e53-6b13c670ae1d-istio-podinfo\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:13:45.132237 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:13:45.132236    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-scmm5\" (UniqueName: \"kubernetes.io/projected/e00a8c30-1ae5-4e96-8e53-6b13c670ae1d-kube-api-access-scmm5\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:13:45.132237 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:13:45.132252    5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e00a8c30-1ae5-4e96-8e53-6b13c670ae1d-istio-token\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:13:45.132588 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:13:45.132266    5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e00a8c30-1ae5-4e96-8e53-6b13c670ae1d-istio-envoy\") on node \"ip-10-0-130-190.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:13:45.355960 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:13:45.355875    5060 generic.go:356] "Generic (PLEG): container finished" podID="e00a8c30-1ae5-4e96-8e53-6b13c670ae1d" containerID="f9cc30703d30fc521dd547f099118d9bc77f4036c59929ce0ac471b864e6b9a5" exitCode=0
Oct 31 13:13:45.355960 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:13:45.355928    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-bbf78971-openshift-default-787bb4945c-8tj7h"
Oct 31 13:13:45.356175 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:13:45.355962    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-bbf78971-openshift-default-787bb4945c-8tj7h" event={"ID":"e00a8c30-1ae5-4e96-8e53-6b13c670ae1d","Type":"ContainerDied","Data":"f9cc30703d30fc521dd547f099118d9bc77f4036c59929ce0ac471b864e6b9a5"}
Oct 31 13:13:45.356175 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:13:45.356020    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-bbf78971-openshift-default-787bb4945c-8tj7h" event={"ID":"e00a8c30-1ae5-4e96-8e53-6b13c670ae1d","Type":"ContainerDied","Data":"b3ff26440aa227e56c9033fe4ba1fa7a009e4908b8c639a1c6d4498ad67934bf"}
Oct 31 13:13:45.356175 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:13:45.356044    5060 scope.go:122] "RemoveContainer" containerID="f9cc30703d30fc521dd547f099118d9bc77f4036c59929ce0ac471b864e6b9a5"
Oct 31 13:13:45.366125 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:13:45.366104    5060 scope.go:122] "RemoveContainer" containerID="7a9824beebec3908f948429c83877116f0580794a5fd1a8cc2f211cb6e647e60"
Oct 31 13:13:45.374885 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:13:45.374868    5060 scope.go:122] "RemoveContainer" containerID="f9cc30703d30fc521dd547f099118d9bc77f4036c59929ce0ac471b864e6b9a5"
Oct 31 13:13:45.375163 ip-10-0-130-190 kubenswrapper[5060]: E1031 13:13:45.375143    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9cc30703d30fc521dd547f099118d9bc77f4036c59929ce0ac471b864e6b9a5\": container with ID starting with f9cc30703d30fc521dd547f099118d9bc77f4036c59929ce0ac471b864e6b9a5 not found: ID does not exist" containerID="f9cc30703d30fc521dd547f099118d9bc77f4036c59929ce0ac471b864e6b9a5"
Oct 31 13:13:45.375213 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:13:45.375174    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9cc30703d30fc521dd547f099118d9bc77f4036c59929ce0ac471b864e6b9a5"} err="failed to get container status \"f9cc30703d30fc521dd547f099118d9bc77f4036c59929ce0ac471b864e6b9a5\": rpc error: code = NotFound desc = could not find container \"f9cc30703d30fc521dd547f099118d9bc77f4036c59929ce0ac471b864e6b9a5\": container with ID starting with f9cc30703d30fc521dd547f099118d9bc77f4036c59929ce0ac471b864e6b9a5 not found: ID does not exist"
Oct 31 13:13:45.375213 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:13:45.375190    5060 scope.go:122] "RemoveContainer" containerID="7a9824beebec3908f948429c83877116f0580794a5fd1a8cc2f211cb6e647e60"
Oct 31 13:13:45.375440 ip-10-0-130-190 kubenswrapper[5060]: E1031 13:13:45.375422    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a9824beebec3908f948429c83877116f0580794a5fd1a8cc2f211cb6e647e60\": container with ID starting with 7a9824beebec3908f948429c83877116f0580794a5fd1a8cc2f211cb6e647e60 not found: ID does not exist" containerID="7a9824beebec3908f948429c83877116f0580794a5fd1a8cc2f211cb6e647e60"
Oct 31 13:13:45.375504 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:13:45.375445    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a9824beebec3908f948429c83877116f0580794a5fd1a8cc2f211cb6e647e60"} err="failed to get container status \"7a9824beebec3908f948429c83877116f0580794a5fd1a8cc2f211cb6e647e60\": rpc error: code = NotFound desc = could not find container \"7a9824beebec3908f948429c83877116f0580794a5fd1a8cc2f211cb6e647e60\": container with ID starting with 7a9824beebec3908f948429c83877116f0580794a5fd1a8cc2f211cb6e647e60 not found: ID does not exist"
Oct 31 13:13:45.385368 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:13:45.385346    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-bbf78971-openshift-default-787bb4945c-8tj7h"]
Oct 31 13:13:45.410749 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:13:45.410718    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-bbf78971-openshift-default-787bb4945c-8tj7h"]
Oct 31 13:13:47.047788 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:13:47.047751    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e00a8c30-1ae5-4e96-8e53-6b13c670ae1d" path="/var/lib/kubelet/pods/e00a8c30-1ae5-4e96-8e53-6b13c670ae1d/volumes"
Oct 31 13:13:59.923468 ip-10-0-130-190 kubenswrapper[5060]: E1031 13:13:59.923421    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7phb" containerName="extractor"
Oct 31 13:14:27.559961 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:14:27.559931    5060 scope.go:122] "RemoveContainer" containerID="7180bd558d22c92ccb691f33bb587f3e9a19ca42447a559a48700de659dcc3b4"
Oct 31 13:14:29.921605 ip-10-0-130-190 kubenswrapper[5060]: E1031 13:14:29.921566    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7phb" containerName="extractor"
Oct 31 13:14:59.920903 ip-10-0-130-190 kubenswrapper[5060]: E1031 13:14:59.920858    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7phb" containerName="extractor"
Oct 31 13:15:29.921585 ip-10-0-130-190 kubenswrapper[5060]: E1031 13:15:29.921547    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7phb" containerName="extractor"
Oct 31 13:15:59.922409 ip-10-0-130-190 kubenswrapper[5060]: E1031 13:15:59.922325    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7phb" containerName="extractor"
Oct 31 13:16:05.025170 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:16:05.025133    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9dc80aa2-2l2l6"]
Oct 31 13:16:05.028799 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:16:05.028774    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9dc80aa2-2l2l6"]
Oct 31 13:16:05.047078 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:16:05.047048    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="baa81c5d-59fb-4d8d-a3f7-4b253b9f6402" path="/var/lib/kubelet/pods/baa81c5d-59fb-4d8d-a3f7-4b253b9f6402/volumes"
Oct 31 13:16:29.921533 ip-10-0-130-190 kubenswrapper[5060]: E1031 13:16:29.921467    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7phb" containerName="extractor"
Oct 31 13:16:38.094093 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:16:38.094055    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5ccbfdc857-zp7zh_9a585b0c-1708-4bb9-8896-3e10e29824d3/authorino/0.log"
Oct 31 13:16:48.238422 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:16:48.238391    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-56dd777d6d-cnzrb_4535c997-310c-4fb4-bd16-d268151053ab/manager/0.log"
Oct 31 13:16:49.065085 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:16:49.065048    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-64d9577675-jdt6k_66ebd441-f618-4dab-9bdd-d119f4111152/manager/0.log"
Oct 31 13:16:50.535923 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:16:50.535894    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-7c54444cdb-qxjzm_6d8b24af-3e72-495d-8c18-f0427d81e9a2/maas-api/0.log"
Oct 31 13:16:53.210216 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:16:53.210183    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5ccbfdc857-zp7zh_9a585b0c-1708-4bb9-8896-3e10e29824d3/authorino/0.log"
Oct 31 13:16:53.429828 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:16:53.429795    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-jbwh5_2eda55f0-194c-462f-b1d0-dc52a2b4a669/manager/0.log"
Oct 31 13:16:53.605954 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:16:53.605905    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-crpzx_b674e577-791f-43ef-9ad8-81938802d946/manager/0.log"
Oct 31 13:16:53.978017 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:16:53.977935    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-lqk5j_d289b4b1-3490-492e-95a7-f170cddbfe63/manager/0.log"
Oct 31 13:16:54.367204 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:16:54.367168    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-vhq87_e3209cd3-5f31-4d69-95d8-7ec4cf348ddf/manager/0.log"
Oct 31 13:16:55.817283 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:16:55.817246    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-645cf464d7-7psf5_37aec2da-59cb-41ed-9e6b-a443f6ff84c7/istio-proxy/0.log"
Oct 31 13:16:57.053113 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:16:57.053081    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-645cf464d7-mwc4r_51fa30eb-9214-4402-b911-28d4a2ff4a5e/istio-proxy/0.log"
Oct 31 13:16:57.243640 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:16:57.243607    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-645cf464d7-rz7mt_978e5cf9-a89e-4fc9-b4ba-fe0ee2de401b/istio-proxy/0.log"
Oct 31 13:16:57.419044 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:16:57.418969    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-645cf464d7-rzth9_1d2738c9-9abf-47cf-a6d8-b83789055353/istio-proxy/0.log"
Oct 31 13:16:57.734137 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:16:57.734023    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-vtzns_d3ee1ad7-1684-4a48-b7fb-b35b3a45b86d/payload-pre-processing/0.log"
Oct 31 13:16:57.910224 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:16:57.910195    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-v452b_6e5b132f-d4fc-48de-beab-1c638e61d6a4/payload-processing/0.log"
Oct 31 13:16:58.092553 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:16:58.092520    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-659589798c-776bx_556dda7b-d8e6-4558-b5f3-994b01d6a866/router/0.log"
Oct 31 13:16:59.191900 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:16:59.191871    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-vmzbr_7049f597-e139-4f0e-afbd-ec6ea0effab9/main/0.log"
Oct 31 13:16:59.552809 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:16:59.552782    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-rm7cl_8292eaa8-0a02-46be-8654-3c4136ff604f/main/0.log"
Oct 31 13:16:59.901469 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:16:59.901389    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-rbn8q_dea929f7-61c3-4921-9bdb-106f763e067e/main/0.log"
Oct 31 13:16:59.920315 ip-10-0-130-190 kubenswrapper[5060]: E1031 13:16:59.920283    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7phb" containerName="extractor"
Oct 31 13:17:00.250381 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:17:00.250303    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-x2qzv_c247ed2a-aa90-424b-9b7e-4f2936d67e3f/main/0.log"
Oct 31 13:17:01.107878 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:17:01.107846    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5ccbfdc857-zp7zh_9a585b0c-1708-4bb9-8896-3e10e29824d3/authorino/0.log"
Oct 31 13:17:09.872687 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:17:09.872654    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-56dd777d6d-cnzrb_4535c997-310c-4fb4-bd16-d268151053ab/manager/0.log"
Oct 31 13:17:10.674689 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:17:10.674658    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-64d9577675-jdt6k_66ebd441-f618-4dab-9bdd-d119f4111152/manager/0.log"
Oct 31 13:17:11.937253 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:17:11.937222    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-7c54444cdb-qxjzm_6d8b24af-3e72-495d-8c18-f0427d81e9a2/maas-api/0.log"
Oct 31 13:17:14.070544 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:17:14.070480    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5ccbfdc857-zp7zh_9a585b0c-1708-4bb9-8896-3e10e29824d3/authorino/0.log"
Oct 31 13:17:14.296824 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:17:14.296796    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-jbwh5_2eda55f0-194c-462f-b1d0-dc52a2b4a669/manager/0.log"
Oct 31 13:17:14.481415 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:17:14.481336    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-crpzx_b674e577-791f-43ef-9ad8-81938802d946/manager/0.log"
Oct 31 13:17:14.821632 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:17:14.821605    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-lqk5j_d289b4b1-3490-492e-95a7-f170cddbfe63/manager/0.log"
Oct 31 13:17:15.203723 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:17:15.203648    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-vhq87_e3209cd3-5f31-4d69-95d8-7ec4cf348ddf/manager/0.log"
Oct 31 13:17:16.429286 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:17:16.429246    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-645cf464d7-7psf5_37aec2da-59cb-41ed-9e6b-a443f6ff84c7/istio-proxy/0.log"
Oct 31 13:17:17.645185 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:17:17.645154    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-645cf464d7-mwc4r_51fa30eb-9214-4402-b911-28d4a2ff4a5e/istio-proxy/0.log"
Oct 31 13:17:17.826879 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:17:17.826853    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-645cf464d7-rz7mt_978e5cf9-a89e-4fc9-b4ba-fe0ee2de401b/istio-proxy/0.log"
Oct 31 13:17:17.988749 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:17:17.988670    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-645cf464d7-rzth9_1d2738c9-9abf-47cf-a6d8-b83789055353/istio-proxy/0.log"
Oct 31 13:17:18.056600 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:17:18.056566    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pcvvm_f032909d-bc39-4c9e-99b7-f3d800842119/console-operator/2.log"
Oct 31 13:17:18.063671 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:17:18.063649    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pcvvm_f032909d-bc39-4c9e-99b7-f3d800842119/console-operator/2.log"
Oct 31 13:17:18.345190 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:17:18.345158    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-vtzns_d3ee1ad7-1684-4a48-b7fb-b35b3a45b86d/payload-pre-processing/0.log"
Oct 31 13:17:18.516916 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:17:18.516886    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-v452b_6e5b132f-d4fc-48de-beab-1c638e61d6a4/payload-processing/0.log"
Oct 31 13:17:18.690689 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:17:18.690609    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-659589798c-776bx_556dda7b-d8e6-4558-b5f3-994b01d6a866/router/0.log"
Oct 31 13:17:19.398318 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:17:19.398285    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-vmzbr_7049f597-e139-4f0e-afbd-ec6ea0effab9/main/0.log"
Oct 31 13:17:19.737015 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:17:19.736924    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-rm7cl_8292eaa8-0a02-46be-8654-3c4136ff604f/main/0.log"
Oct 31 13:17:20.075953 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:17:20.075917    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-rbn8q_dea929f7-61c3-4921-9bdb-106f763e067e/main/0.log"
Oct 31 13:17:20.414256 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:17:20.414186    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-x2qzv_c247ed2a-aa90-424b-9b7e-4f2936d67e3f/main/0.log"
Oct 31 13:17:27.609010 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:17:27.608972    5060 scope.go:122] "RemoveContainer" containerID="65e4780061b7669ad7ab84ae704bb8624170580daf7e74f419396dc46bb4a6b0"
Oct 31 13:17:27.618805 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:17:27.618780    5060 scope.go:122] "RemoveContainer" containerID="ced879045f08d62566df46059a4388bed04e6bcd7ac8c3aa785ae615ff8413cf"
Oct 31 13:17:29.929439 ip-10-0-130-190 kubenswrapper[5060]: E1031 13:17:29.929397    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7phb" containerName="extractor"
Oct 31 13:17:56.056563 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:17:56.056523    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-64d9577675-jdt6k_66ebd441-f618-4dab-9bdd-d119f4111152/manager/0.log"
Oct 31 13:17:56.298881 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:17:56.298851    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-64d9577675-jdt6k_66ebd441-f618-4dab-9bdd-d119f4111152/manager/0.log"
Oct 31 13:17:56.974586 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:17:56.974556    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-v452b_6e5b132f-d4fc-48de-beab-1c638e61d6a4/payload-processing/0.log"
Oct 31 13:17:57.135418 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:17:57.135387    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-v452b_6e5b132f-d4fc-48de-beab-1c638e61d6a4/payload-processing/0.log"
Oct 31 13:17:59.919926 ip-10-0-130-190 kubenswrapper[5060]: E1031 13:17:59.919892    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7phb" containerName="extractor"
Oct 31 13:18:29.920857 ip-10-0-130-190 kubenswrapper[5060]: E1031 13:18:29.920813    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7phb" containerName="extractor"
Oct 31 13:18:39.279372 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:18:39.279336    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-64d9577675-jdt6k_66ebd441-f618-4dab-9bdd-d119f4111152/manager/0.log"
Oct 31 13:18:39.505903 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:18:39.505860    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-64d9577675-jdt6k_66ebd441-f618-4dab-9bdd-d119f4111152/manager/0.log"
Oct 31 13:18:40.150968 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:18:40.150937    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-v452b_6e5b132f-d4fc-48de-beab-1c638e61d6a4/payload-processing/0.log"
Oct 31 13:18:40.323678 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:18:40.323644    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-v452b_6e5b132f-d4fc-48de-beab-1c638e61d6a4/payload-processing/0.log"
Oct 31 13:18:43.963508 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:18:43.963467    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-2bmzr_a2b450be-ca79-4edf-a1ea-377531a49519/global-pull-secret-syncer/0.log"
Oct 31 13:18:44.142932 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:18:44.142904    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-k64bz_613125d4-fbb4-4b43-9c47-24e039bd4697/connection-checker/0.log"
Oct 31 13:18:44.170452 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:18:44.170427    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-rrc8k_ff424288-19ca-4db0-9f16-fed9fb58404a/connection-checker/0.log"
Oct 31 13:18:44.198924 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:18:44.198893    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-vvzpg_07714f20-a70e-49e8-8e51-d03a8e00005a/connection-checker/0.log"
Oct 31 13:18:44.297023 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:18:44.296983    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-8jlvq_b5014b57-9b83-4fc9-b0ed-55793b2b3a98/konnectivity-agent/0.log"
Oct 31 13:18:44.377797 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:18:44.377750    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-130-190.us-east-2.compute.internal_7fb532212935ad861267bbe741c1366b/haproxy/0.log"
Oct 31 13:18:51.541860 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:18:51.541815    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5ccbfdc857-zp7zh_9a585b0c-1708-4bb9-8896-3e10e29824d3/authorino/0.log"
Oct 31 13:18:51.597210 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:18:51.597180    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-jbwh5_2eda55f0-194c-462f-b1d0-dc52a2b4a669/manager/0.log"
Oct 31 13:18:51.630231 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:18:51.630199    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-crpzx_b674e577-791f-43ef-9ad8-81938802d946/manager/0.log"
Oct 31 13:18:51.718065 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:18:51.718028    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-lqk5j_d289b4b1-3490-492e-95a7-f170cddbfe63/manager/0.log"
Oct 31 13:18:51.825897 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:18:51.825810    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-vhq87_e3209cd3-5f31-4d69-95d8-7ec4cf348ddf/manager/0.log"
Oct 31 13:18:53.350893 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:18:53.350859    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-gt9rn_dda0c26f-8225-4239-bc27-866b66cbb919/cluster-monitoring-operator/0.log"
Oct 31 13:18:53.743339 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:18:53.743261    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gv4qj_77b0599f-2680-4374-be49-1c2dd34e2618/node-exporter/0.log"
Oct 31 13:18:53.770101 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:18:53.770074    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gv4qj_77b0599f-2680-4374-be49-1c2dd34e2618/kube-rbac-proxy/0.log"
Oct 31 13:18:53.796925 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:18:53.796900    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gv4qj_77b0599f-2680-4374-be49-1c2dd34e2618/init-textfile/0.log"
Oct 31 13:18:54.018344 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:18:54.018266    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-q58mx/perf-node-gather-daemonset-b24sc"]
Oct 31 13:18:54.018900 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:18:54.018872    5060 cpu_manager.go:414] "containerMap: removing container" podUID="e00a8c30-1ae5-4e96-8e53-6b13c670ae1d" containerName="istio-proxy"
Oct 31 13:18:54.018900 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:18:54.018897    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e00a8c30-1ae5-4e96-8e53-6b13c670ae1d" containerName="istio-proxy"
Oct 31 13:18:54.019143 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:18:54.019032    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="e00a8c30-1ae5-4e96-8e53-6b13c670ae1d" containerName="istio-proxy"
Oct 31 13:18:54.019143 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:18:54.019047    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="e00a8c30-1ae5-4e96-8e53-6b13c670ae1d" containerName="istio-proxy"
Oct 31 13:18:54.022263 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:18:54.022244    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-q58mx/perf-node-gather-daemonset-b24sc"
Oct 31 13:18:54.033847 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:18:54.033816    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-q58mx\"/\"openshift-service-ca.crt\""
Oct 31 13:18:54.033964 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:18:54.033935    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-q58mx\"/\"kube-root-ca.crt\""
Oct 31 13:18:54.035075 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:18:54.035048    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-q58mx/perf-node-gather-daemonset-b24sc"]
Oct 31 13:18:54.035177 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:18:54.035088    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-q58mx\"/\"default-dockercfg-7dg4w\""
Oct 31 13:18:54.056996 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:18:54.056964    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hs84b\" (UniqueName: \"kubernetes.io/projected/04455541-235d-4ea4-9b7b-f260185c7b8b-kube-api-access-hs84b\") pod \"perf-node-gather-daemonset-b24sc\" (UID: \"04455541-235d-4ea4-9b7b-f260185c7b8b\") " pod="openshift-must-gather-q58mx/perf-node-gather-daemonset-b24sc"
Oct 31 13:18:54.057170 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:18:54.057006    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/04455541-235d-4ea4-9b7b-f260185c7b8b-proc\") pod \"perf-node-gather-daemonset-b24sc\" (UID: \"04455541-235d-4ea4-9b7b-f260185c7b8b\") " pod="openshift-must-gather-q58mx/perf-node-gather-daemonset-b24sc"
Oct 31 13:18:54.057170 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:18:54.057024    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/04455541-235d-4ea4-9b7b-f260185c7b8b-sys\") pod \"perf-node-gather-daemonset-b24sc\" (UID: \"04455541-235d-4ea4-9b7b-f260185c7b8b\") " pod="openshift-must-gather-q58mx/perf-node-gather-daemonset-b24sc"
Oct 31 13:18:54.057316 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:18:54.057167    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/04455541-235d-4ea4-9b7b-f260185c7b8b-lib-modules\") pod \"perf-node-gather-daemonset-b24sc\" (UID: \"04455541-235d-4ea4-9b7b-f260185c7b8b\") " pod="openshift-must-gather-q58mx/perf-node-gather-daemonset-b24sc"
Oct 31 13:18:54.057316 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:18:54.057210    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/04455541-235d-4ea4-9b7b-f260185c7b8b-podres\") pod \"perf-node-gather-daemonset-b24sc\" (UID: \"04455541-235d-4ea4-9b7b-f260185c7b8b\") " pod="openshift-must-gather-q58mx/perf-node-gather-daemonset-b24sc"
Oct 31 13:18:54.158209 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:18:54.158174    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/04455541-235d-4ea4-9b7b-f260185c7b8b-podres\") pod \"perf-node-gather-daemonset-b24sc\" (UID: \"04455541-235d-4ea4-9b7b-f260185c7b8b\") " pod="openshift-must-gather-q58mx/perf-node-gather-daemonset-b24sc"
Oct 31 13:18:54.158380 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:18:54.158326    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/04455541-235d-4ea4-9b7b-f260185c7b8b-podres\") pod \"perf-node-gather-daemonset-b24sc\" (UID: \"04455541-235d-4ea4-9b7b-f260185c7b8b\") " pod="openshift-must-gather-q58mx/perf-node-gather-daemonset-b24sc"
Oct 31 13:18:54.158380 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:18:54.158339    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hs84b\" (UniqueName: \"kubernetes.io/projected/04455541-235d-4ea4-9b7b-f260185c7b8b-kube-api-access-hs84b\") pod \"perf-node-gather-daemonset-b24sc\" (UID: \"04455541-235d-4ea4-9b7b-f260185c7b8b\") " pod="openshift-must-gather-q58mx/perf-node-gather-daemonset-b24sc"
Oct 31 13:18:54.158460 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:18:54.158415    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/04455541-235d-4ea4-9b7b-f260185c7b8b-proc\") pod \"perf-node-gather-daemonset-b24sc\" (UID: \"04455541-235d-4ea4-9b7b-f260185c7b8b\") " pod="openshift-must-gather-q58mx/perf-node-gather-daemonset-b24sc"
Oct 31 13:18:54.158460 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:18:54.158443    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/04455541-235d-4ea4-9b7b-f260185c7b8b-sys\") pod \"perf-node-gather-daemonset-b24sc\" (UID: \"04455541-235d-4ea4-9b7b-f260185c7b8b\") " pod="openshift-must-gather-q58mx/perf-node-gather-daemonset-b24sc"
Oct 31 13:18:54.158578 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:18:54.158551    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/04455541-235d-4ea4-9b7b-f260185c7b8b-lib-modules\") pod \"perf-node-gather-daemonset-b24sc\" (UID: \"04455541-235d-4ea4-9b7b-f260185c7b8b\") " pod="openshift-must-gather-q58mx/perf-node-gather-daemonset-b24sc"
Oct 31 13:18:54.158628 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:18:54.158579    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/04455541-235d-4ea4-9b7b-f260185c7b8b-proc\") pod \"perf-node-gather-daemonset-b24sc\" (UID: \"04455541-235d-4ea4-9b7b-f260185c7b8b\") " pod="openshift-must-gather-q58mx/perf-node-gather-daemonset-b24sc"
Oct 31 13:18:54.158685 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:18:54.158656    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/04455541-235d-4ea4-9b7b-f260185c7b8b-sys\") pod \"perf-node-gather-daemonset-b24sc\" (UID: \"04455541-235d-4ea4-9b7b-f260185c7b8b\") " pod="openshift-must-gather-q58mx/perf-node-gather-daemonset-b24sc"
Oct 31 13:18:54.158740 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:18:54.158690    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/04455541-235d-4ea4-9b7b-f260185c7b8b-lib-modules\") pod \"perf-node-gather-daemonset-b24sc\" (UID: \"04455541-235d-4ea4-9b7b-f260185c7b8b\") " pod="openshift-must-gather-q58mx/perf-node-gather-daemonset-b24sc"
Oct 31 13:18:54.172998 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:18:54.172968    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hs84b\" (UniqueName: \"kubernetes.io/projected/04455541-235d-4ea4-9b7b-f260185c7b8b-kube-api-access-hs84b\") pod \"perf-node-gather-daemonset-b24sc\" (UID: \"04455541-235d-4ea4-9b7b-f260185c7b8b\") " pod="openshift-must-gather-q58mx/perf-node-gather-daemonset-b24sc"
Oct 31 13:18:54.335707 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:18:54.335670    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-q58mx/perf-node-gather-daemonset-b24sc"
Oct 31 13:18:54.613854 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:18:54.613826    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-q58mx/perf-node-gather-daemonset-b24sc"]
Oct 31 13:18:54.615873 ip-10-0-130-190 kubenswrapper[5060]: W1031 13:18:54.615841    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod04455541_235d_4ea4_9b7b_f260185c7b8b.slice/crio-ff2b3cc38df3b31d137417072115b0c767e1ac270c5df76cec7a32911030d77e WatchSource:0}: Error finding container ff2b3cc38df3b31d137417072115b0c767e1ac270c5df76cec7a32911030d77e: Status 404 returned error can't find the container with id ff2b3cc38df3b31d137417072115b0c767e1ac270c5df76cec7a32911030d77e
Oct 31 13:18:55.461633 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:18:55.461596    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-q58mx/perf-node-gather-daemonset-b24sc" event={"ID":"04455541-235d-4ea4-9b7b-f260185c7b8b","Type":"ContainerStarted","Data":"4295bc06e75f92ed553fb5de9ae75b4def0dcc02d4070d72c57faf08cc52f53f"}
Oct 31 13:18:55.461846 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:18:55.461651    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-q58mx/perf-node-gather-daemonset-b24sc"
Oct 31 13:18:55.461846 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:18:55.461668    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-q58mx/perf-node-gather-daemonset-b24sc" event={"ID":"04455541-235d-4ea4-9b7b-f260185c7b8b","Type":"ContainerStarted","Data":"ff2b3cc38df3b31d137417072115b0c767e1ac270c5df76cec7a32911030d77e"}
Oct 31 13:18:55.483286 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:18:55.483227    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-q58mx/perf-node-gather-daemonset-b24sc" podStartSLOduration=2.483210873 podStartE2EDuration="2.483210873s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 13:18:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:18:55.482962599 +0000 UTC m=+3098.586030311" watchObservedRunningTime="2026-10-31 13:18:55.483210873 +0000 UTC m=+3098.586278585"
Oct 31 13:18:55.678979 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:18:55.678948    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-86c478bb88-2bfvp_50ad7816-358d-461e-a9bd-fadce6246e8f/networking-console-plugin/0.log"
Oct 31 13:18:56.249126 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:18:56.249096    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pcvvm_f032909d-bc39-4c9e-99b7-f3d800842119/console-operator/2.log"
Oct 31 13:18:56.253350 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:18:56.253330    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pcvvm_f032909d-bc39-4c9e-99b7-f3d800842119/console-operator/3.log"
Oct 31 13:18:57.365871 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:18:57.365837    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-58c9f5f46d-hf9hr_b28ee2bb-0387-4d2d-8ca7-3fef91619c6b/volume-data-source-validator/0.log"
Oct 31 13:18:58.655685 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:18:58.655603    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-r6fvt_5c8865f9-fdf6-49b9-8b4f-69dbc7787084/dns/0.log"
Oct 31 13:18:58.682122 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:18:58.682093    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-r6fvt_5c8865f9-fdf6-49b9-8b4f-69dbc7787084/kube-rbac-proxy/0.log"
Oct 31 13:18:58.739576 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:18:58.739546    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-v9ks4_6bbe13aa-df4b-40b0-8595-68fe1166a082/dns-node-resolver/0.log"
Oct 31 13:18:59.513344 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:18:59.513313    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-wdkzp_6c661ce8-9f7b-4509-bd17-2380bcd1c425/node-ca/0.log"
Oct 31 13:18:59.920812 ip-10-0-130-190 kubenswrapper[5060]: E1031 13:18:59.920775    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7phb" containerName="extractor"
Oct 31 13:19:01.403770 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:19:01.403740    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-645cf464d7-7psf5_37aec2da-59cb-41ed-9e6b-a443f6ff84c7/istio-proxy/0.log"
Oct 31 13:19:01.478747 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:19:01.478722    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-q58mx/perf-node-gather-daemonset-b24sc"
Oct 31 13:19:01.642528 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:19:01.642485    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-645cf464d7-mwc4r_51fa30eb-9214-4402-b911-28d4a2ff4a5e/istio-proxy/0.log"
Oct 31 13:19:01.682054 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:19:01.681986    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-645cf464d7-rz7mt_978e5cf9-a89e-4fc9-b4ba-fe0ee2de401b/istio-proxy/0.log"
Oct 31 13:19:01.715727 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:19:01.715701    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-645cf464d7-rzth9_1d2738c9-9abf-47cf-a6d8-b83789055353/istio-proxy/0.log"
Oct 31 13:19:01.775902 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:19:01.775873    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-vtzns_d3ee1ad7-1684-4a48-b7fb-b35b3a45b86d/payload-pre-processing/0.log"
Oct 31 13:19:01.814105 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:19:01.814077    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-v452b_6e5b132f-d4fc-48de-beab-1c638e61d6a4/payload-processing/0.log"
Oct 31 13:19:01.850656 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:19:01.850629    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-659589798c-776bx_556dda7b-d8e6-4558-b5f3-994b01d6a866/router/0.log"
Oct 31 13:19:04.197834 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:19:04.197804    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-rl55s_891b3c47-ff7c-4a7a-a09b-411361a843f7/serve-healthcheck-canary/0.log"
Oct 31 13:19:04.756631 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:19:04.756597    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-7c4986d449-z879s_ddb04b2e-2724-44e9-948a-877a90d8a071/insights-operator/0.log"
Oct 31 13:19:04.757401 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:19:04.757383    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-7c4986d449-z879s_ddb04b2e-2724-44e9-948a-877a90d8a071/insights-operator/1.log"
Oct 31 13:19:04.990138 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:19:04.990110    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-p7phb_9557a3f0-ae0f-41f9-a9b5-15aeda410a38/kube-rbac-proxy/0.log"
Oct 31 13:19:05.017808 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:19:05.017744    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-p7phb_9557a3f0-ae0f-41f9-a9b5-15aeda410a38/exporter/0.log"
Oct 31 13:19:05.046905 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:19:05.046873    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-p7phb_9557a3f0-ae0f-41f9-a9b5-15aeda410a38/extractor/0.log"
Oct 31 13:19:05.086227 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:19:05.086200    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_periodic-gathering-rrx9r-zvwfn_9f7f000c-c637-4b51-b4ad-805ea3205f29/insights-gathering/0.log"
Oct 31 13:19:05.105472 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:19:05.105451    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-645cf464d7-7psf5_37aec2da-59cb-41ed-9e6b-a443f6ff84c7/istio-proxy/0.log"
Oct 31 13:19:05.280647 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:19:05.280618    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-645cf464d7-mwc4r_51fa30eb-9214-4402-b911-28d4a2ff4a5e/istio-proxy/0.log"
Oct 31 13:19:05.307440 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:19:05.307408    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-645cf464d7-rz7mt_978e5cf9-a89e-4fc9-b4ba-fe0ee2de401b/istio-proxy/0.log"
Oct 31 13:19:05.325069 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:19:05.325047    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-645cf464d7-rzth9_1d2738c9-9abf-47cf-a6d8-b83789055353/istio-proxy/0.log"
Oct 31 13:19:05.367715 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:19:05.367687    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-vtzns_d3ee1ad7-1684-4a48-b7fb-b35b3a45b86d/payload-pre-processing/0.log"
Oct 31 13:19:05.383431 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:19:05.383410    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-v452b_6e5b132f-d4fc-48de-beab-1c638e61d6a4/payload-processing/0.log"
Oct 31 13:19:05.410286 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:19:05.410263    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-659589798c-776bx_556dda7b-d8e6-4558-b5f3-994b01d6a866/router/0.log"
Oct 31 13:19:07.857360 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:19:07.857325    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-5967ccc7d4-tbkp7_9a2f809e-4e2d-4aa2-baf1-c9a4b24e9f96/manager/0.log"
Oct 31 13:19:07.898744 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:19:07.898715    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-5967ccc7d4-tbkp7_9a2f809e-4e2d-4aa2-baf1-c9a4b24e9f96/copy-manifests/0.log"
Oct 31 13:19:09.049903 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:19:09.049875    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-645cf464d7-7psf5_37aec2da-59cb-41ed-9e6b-a443f6ff84c7/istio-proxy/0.log"
Oct 31 13:19:09.200888 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:19:09.200857    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-jobset-operator_jobset-controller-manager-6698b9df8-qxvfh_f6a410bf-ca1f-4204-ac07-31fa2cb086a4/manager/0.log"
Oct 31 13:19:09.204096 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:19:09.204072    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-645cf464d7-mwc4r_51fa30eb-9214-4402-b911-28d4a2ff4a5e/istio-proxy/0.log"
Oct 31 13:19:09.221562 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:19:09.221539    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-645cf464d7-rz7mt_978e5cf9-a89e-4fc9-b4ba-fe0ee2de401b/istio-proxy/0.log"
Oct 31 13:19:09.235804 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:19:09.235781    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-jobset-operator_jobset-operator-674cd59dc5-h45r8_ef57947a-5e0a-43df-aebb-641564a41208/jobset-operator/0.log"
Oct 31 13:19:09.249140 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:19:09.249120    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-645cf464d7-rzth9_1d2738c9-9abf-47cf-a6d8-b83789055353/istio-proxy/0.log"
Oct 31 13:19:09.283965 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:19:09.283938    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-vtzns_d3ee1ad7-1684-4a48-b7fb-b35b3a45b86d/payload-pre-processing/0.log"
Oct 31 13:19:09.301433 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:19:09.301362    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-v452b_6e5b132f-d4fc-48de-beab-1c638e61d6a4/payload-processing/0.log"
Oct 31 13:19:09.328936 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:19:09.328912    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-659589798c-776bx_556dda7b-d8e6-4558-b5f3-994b01d6a866/router/0.log"
Oct 31 13:19:09.903890 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:19:09.903860    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-56dd777d6d-cnzrb_4535c997-310c-4fb4-bd16-d268151053ab/manager/0.log"
Oct 31 13:19:10.672993 ip-10-0-130-190 kubenswrapper[5060]: I1031 13:19:10.672959    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-64d9577675-jdt6k_66ebd441-f618-4dab-9bdd-d119f4111152/manager/0.log"